8 Replies Latest reply on Feb 21, 2019 2:22 AM by Hamish marson

    Get job logs for failed  targets

    Hamish marson
      Share:|

      Is it possible to get the job logs from the blcli (i.e. SOAP API interface) for a particular run of a job target?

       

      The granularity for reading the completion status of a target for a job isn't sufficient for us to be able to tell what went wrong without a human logging into the bladelogic GUI and finding the job then opening the logs and finding the failed target... Which takes a while and isn't really suitable where we're automating processes.

       

      Is it possible to read the bladelogic logs for this info via the blcli? e.g. where a log reads..

       

      (Sorry, the website has removed my image snapshot... but the error in the bladelogic job logs says 'Failed connecting to server <fqdn_of_server:4750>, connection refused)

       

      I want to be able to get the error text so I can save it off and display it in a report that's produced elsewhere at the end of the job. I can't get it during the job execution and log it elsewhere  because the runScript never gets started. It's not the fact that an error occurs that's the problem. It's finding out what the cause of the error was

        • 1. Re: Get job logs for failed  targets
          Bill Robinson
          Is it possible to get the job logs from the blcli (i.e. SOAP API interface) for a particular run of a job target?

          JobRun - getLogItemsByJobRunId - Documentation for BMC Server Automation Command Line Interface 8.9 - BMC Documentation

          ?

          • 2. Re: Get job logs for failed  targets
            Hamish marson

            That call is somewhat helpful... But it returns a single block of text that unlike the GUI just has all the errors and output mixed together. For example if one of the runScripts (Or attempted runScripts) exits with RC=1 I see a line with

             

            Type: Info Date: Tue Jan 15 14:37:20 GMT 2019 Message: Exit Code 1

             

            But I don't know which of the targets this error goes with. If there were 3 hosts failing there's 3 failure lines... Sometimes they're different and no good way to tell them apart. It's clear from the logs so far that the messages are from multiple threads in parallel. Which if I only have 1 target is OK... But when I have 100 targets and multiple different reasons for failure it becomes non-deterministic to say target X failed with this message and target Y failed with this message.

             

            The GUI manages to separate out logs from the job (Execution starting mainly is the important one) and logs from the targets themselves (Which identify whether the job started OK, or failed because RSCD was down (connection refused), or failed because the host was down (Timeout usually), or something else.

            Is there a call to separate out the logs like that? By job and target?

             

            H

            • 3. Re: Get job logs for failed  targets
              Bill Robinson

              what type of job is this ?

              • 4. Re: Get job logs for failed  targets
                Hamish marson

                It's an NSH Script job using the option to run a separate runScript for each target (i.e. lots of targets in parallel).

                • 5. Re: Get job logs for failed  targets
                  Bill Robinson

                  can you attach (adv editor) an example of what you find confusing ?  the way the nsh script job logs work is there is generally the job level messages at the top, then the per-target messages commingled, usually the line starts w/ the target name, then the job level stuff at the end.  grepping out the lines that start w/ the hostname of the target should do it.  there's also a feature added in 8.9.04 that only shows the per-target entries.

                   

                  i think there's also some variants of the blcli command that i linked to (maybe unrelased) that should give the per-target log output only.

                  • 6. Re: Get job logs for failed  targets
                    Hamish marson

                    The 'confusing' part is that there is no consistent mention of the target name. e.g.

                     

                    Type: Info Date: Wed Feb 13 23:16:33 GMT 2019 Message: Exit Code 1

                    Type: Info Date: Wed Feb 13 23:18:32 GMT 2019 Message: tellstatus BSA: GATHER: START

                     

                    Which particular target were those log lines for? I have absolutely no idea... The first is the exit code from one of the targets. The second is output from another target execution (Via echo) from the NSHScript. While I could add the target name into the second line (i.e. the echo). But the first that tells me the actual exit code of the execution, I can't do anything about.

                     

                    Starting of a target, yes. I could parse that out... e.g.

                     

                    Type: Info Date: Wed Feb 13 23:16:33 GMT 2019 Message: Executing work item NSH Script Job:BL1234; Server:targetnamehere.com;  on application server: bladelogicrds

                     

                    This entry clearly says a script was started for 'targetnamehere.com'

                     

                    So, inconsistent. And I see absolutely no evidence of the lines 'usually' starting with the target name.

                     

                    These logs are from 8.9.01 (An RDS - As in Rapid Development System from BMC, Because I use an RDS for development. However  just to make it more difficult, I'm told that RDS isn't being supported with later versions and that I can't simply upgrade the RDS to a later release such as 8.9.03 or 8.9.04).

                    • 7. Re: Get job logs for failed  targets
                      Bill Robinson

                      yeah - the getLogItems one doesn't pre-pend the hostname - i thought it did.

                       

                      if you use one of the Utility export commands you can get a file that does have the hostname pre-pended:

                      Job Name,"Sleep1 - Run at Feb 20, 2019 3:55:01 PM"

                      Start Time,Feb 20, 2019 3:55:01 PM

                      End Time,Feb 20, 2019 3:55:07 PM

                      Status,Completed Successfully

                       

                       

                      Participant,Type,Date,Message

                      "Run at Feb 20, 2019 3:55:01 PM",Info,"Feb 20, 2019 3:55:01 PM","JobRun Log Settings: JobRun Limit = 1000000, Per Target Log Limit = 1000, Per Target Log Level = AllInfo"

                      "Run at Feb 20, 2019 3:55:01 PM",Info,"Feb 20, 2019 3:55:01 PM","Started running the job 'Sleep1' with priority 'NORMAL' on application server 'blapp894.example.com'(2,000,000)"

                      "Run at Feb 20, 2019 3:55:01 PM",Info,"Feb 20, 2019 3:55:02 PM",Executing work item NSH Script Job:Sleep1; Server:blrepeater894.example.com;  on application server: blapp894.example.com

                      blrepeater894.example.com,Info,"Feb 20, 2019 3:55:03 PM",this is 1...

                      blrepeater894.example.com,Info,"Feb 20, 2019 3:55:03 PM",this is 2...

                      blrepeater894.example.com,Info,"Feb 20, 2019 3:55:03 PM",this is 3...

                      blrepeater894.example.com,Info,"Feb 20, 2019 3:55:03 PM",this is 4...

                       

                      if you only want the failed targets you could do something w/ some unreleased commands:

                      blcli_execute NSHScriptJob getDBKeyByGroupAndName "/Workspace/NSH Script Jobs" "Sleep1"

                      blcli_storeenv jobKey

                      blcli_execute JobRun findLastRunKeyByJobKey ${jobKey}

                      blcli_storeenv jobRunKey

                      blcli_execute JobRun jobRunKeyToJobRunId ${jobRunKey}

                      blcli_storeenv jobRunId

                      blcli_execute JobRun getServersStatusByJobRun ${jobRunId}

                      blcli_execute Utility mapPrint

                      blcli_execute Utility setTargetObject

                      blcli_storeenv statusMap

                      while read i

                              do

                              targetKey="$(awk '{print $1}' <<< "${i}")"

                              targetStatus="$(awk '{print $3}' <<< "${i}")"

                              if [[ ${targetStatus} -eq 2 ]]

                                      then

                                      blcli_execute Server findByDBKey ${targetKey}

                                      blcli_execute Server getDeviceId

                                      blcli_storeenv serverId

                                      blcli_execute LogItem getLogItemsByDevice ${jobKey} ${jobRunId} ${serverId}

                                      blcli_execute Utility listPrint

                                      blcli_storeenv logItems

                                      echo "${logItems}"

                              fi

                      done <<< "$(awk 'NF' <<< "${statusMap}")"

                      1 of 1 people found this helpful
                      • 8. Re: Get job logs for failed  targets
                        Hamish marson

                        Thanks, I'll look into them.