BBSA Windows Patch Troubleshooting: How to analyze Trace.txt generated by Analysis Job

Version 1
    Share:|

    Version 1.1
    Follow this link for the latest version:
    https://kb.bmc.com/infocenter/index?page=content&id=KA366485

     

    This article illustrates some information that can be learned from Trace.txt log (and additional results.xml) generated by Windows Patch Analysis Job or live browsing Hotfixes object.

     

    The information below is applicable to Trace.txt generated by Shavlik engine 6.0 which is used in BladeLogic versions 8.2.GA and below. Most of this information may also apply to Shavlik engine 8.0 which is used in BladeLogic 8.2 SP1 and higher.

     

    --------------
    BMC BladeLogic Server Automation Suite
    BMC BladeLogic Patch Management

     


    The following check points are discussed:

     

    1. How are the files generated and where
    2. Was the CAB or XML metadata file passed to the scan
    3. Was whitelist or Include Filter used
    4. Reviewing the logic stack of the patch or 'Reason'
    5. Did the scan complete gracefully or crash
    6. Error messages and troubleshooting

     


    1. How are the files generated and where

     

    Set the target Agent to debug, and you shell see the exact command that is run to perform the scan.
    Executable used for the scan: [RSCD_DIR]/sbin/BLPatchCheck2.exe
    Location of the logs:
         C:\Trace.txt and C:\Trace.txt.old (preserved from last two scans)
         [RSCD_DIR]/tmp/xxx/shavlik_results.xml (deleted after scan)

     

    # executed during live browse Hotfixes, and it only scans for security patches:
    BLPatchCheck2.exe 0 hfnetchk6b.xml shavlik_results.xml

     

    # -pt option is used during Patch Analysis to specify patch types:
    BLPatchCheck2.exe 0 -pt 13 hfnetchk6b.xml shavlik_results.xml
         (1)  Security patches (default)
         (4)  Security tools
         (5)  Security Patches, Security Tools
         (8)  Non-Security patches
         (9)  Security and non-security patches
         (12) Security Tools, Non-Security Patches
         (13) Security, non-security and tools
    In the Trace.txt you would also see the following corresponding line:
    ... MultiMachineScanner.cpp:625 After License check - Scanning for patchtypes [13]

     

    # -qf option is used to specify the whitelist:
    BLPatchCheck2.exe 0 -qf whitelist.txt hfnetchk6b.xml shavlik_results.xml

     

    # -xqf option is used to specify the blacklist:
    BLPatchCheck2.exe 0 -xqf blacklist.txt hfnetchk6b.xml shavlik_results.xml

     


    2. Was the CAB or XML metadata file passed to the scan

     

    The BLPatchCheck2.exe command in rscd.log will reference either hfnetchk6b.cab or hfnetchk6b.xml. The CAB file is the same XML only packaged. The information can also be obtained from the Trace.txt file in the beginning:

     

    # the following is logged if the CAB file was passed to the scan (notice: extraction in process):
    ... CabExtracter.cpp:138 Extracting files from hfnetchk6b.cab
    ... CabExtracter.cpp:63 Extracting hfnetchk6b.xml to \hfnetchk6b.xml
    ... CabExtracter.cpp:49 Extracted \hfnetchk6b.xml from hfnetchk6b.cab
    ... ShavlikXMLDocumentSource.cpp:218 Using XML name of **\hfnetchk6b.xml**

     

    # the following is logged if the XML file was passed to the scan (notice: no extraction):
    ... ShavlikXMLDocumentSource.cpp:218 Using XML name of **hfnetchk6b.xml**

     

    In the end, the actual analysis scan is performed against the XML file either way.

     


    3. Was whitelist or Include Filter used

     

    If the rscd.log in debug is available, then –qf option will be shown with BLPatchCheck2.exe command.
    In the Trace.txt you should see the whitelist logged as such:
    ... MultiMachineScanner.cpp:564 Qs to scan for
    ... MultiMachineScanner.cpp:571  Scanning for [Q2633880]
    ... MultiMachineScanner.cpp:571  Scanning for [Q2633870]
    ... MultiMachineScanner.cpp:571  Scanning for [Q2633874]
    ... MultiMachineScanner.cpp:571  Scanning for [Q2633879]
    ... MultiMachineScanner.cpp:571  Scanning for [Q2633873]

     

    Scanning for [Qxxxxxx] messages will not be present in the Trace.txt if the whitelist was not used.

    Sometimes you may see that the Q numbers repeat in the "Scanning for [Qxxxxxx]" column. This is most likely because you had a Bulletin added to the Include Filter instead of a Q number. The Bulletin may contain multiple hotfixes of the same KB or Q numbers for different Windows platforms or Product versions. To construct the whitelist, all KB numbers are extracted from the Bulletin.

     


    4. Reviewing the logic stack of the patch or 'Reason'

     

    Example 1:
    Here is a snippet from scanning MS12-016 bulletin. The KB numbers are not usually listed during this check, so search the Trace.txt files using Bulletin IDs.

     

    [target_hostname] ... PatchTest.cpp:354 TESTING MS12-016
    [target_hostname] ... PatchTest.cpp:403
         Bulletin ID = MS12-016
         Filecount = 2
         Regcount = 0
    [target_hostname] ... PatchTest.cpp:783 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL ErrNum=0
    [target_hostname] ... PatchTest.cpp:785 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL (2.0.50727.4927)  C 2 2.0.50727.5703 (AC 5)
    [target_hostname] ... PatchTest.cpp:783 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL ErrNum=0
    [target_hostname] ... PatchTest.cpp:785 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL (2.0.50727.4927)  C 2 2.0.50727.4968 (AC 5)
    [target_hostname] ... PatchTest.cpp:798 Is *A* FileChange
    [target_hostname] ... PatchTest.cpp:427 A file was tested
    [target_hostname] ... PatchTest.cpp:429 Did not pass file tests

     

    Key notes:
    "Did not pass file tests" – suggests that the file version found on the target was less than expected, so the patch is considered Missing.
    (2.0.50727.4927) – the file version in parentheses is the version found on the target
    2.0.50727.5703 – the file version that appears next is where we expect to be

     

    In this particular case we see two comparisons to SYSTEM.DLL file of two separate versions. Sometime you may see even more comparisons under the same Bulletin, and they may not be against the same file. This happens because there could be multiple hotfixes under the same Bulletin and they may fall under different branches (GDR and LDR), where each variation may be looking for specific file/version. The results.xml file (or the Patch Analysis results from the BladeLogic Console) would have the correct one listed (note: in rare cases, due to Shavlik defect the incorrect one would be listed – this is to be determined when cross-checking the specifications on vendor site):
    Reason="File version is less than expected. [C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL 2.0.50727.4927 < 2.0.50727.5703]"

     

    Example 2:
    Here is a snippet from analyzing for MS12-017 bulletin.

     

    [target_hostname] ... PatchTest.cpp:354 TESTING MS12-017
    [target_hostname] ... PatchTest.cpp:403
         Bulletin ID = MS12-017
         Filecount = 2
         Regcount = 0
    [target_hostname] ... PatchTest.cpp:783 File C:\windows\system32\dns.exe ErrNum=1
    [target_hostname] ... PatchTest.cpp:783 File C:\windows\system32\dns.exe ErrNum=1
    [target_hostname] ... PatchTest.cpp:427 A file was tested
    [target_hostname] ... PatchTest.cpp:1031 Reg passed

     

    Key notes:
    "Reg passed" – suggests that the condition passed, and the patch is not considered missing. As you may see, we do not list any file version comparison, which brings to the next key note:
    "File C:\windows\system32\dns.exe ErrNum=1" – suggests that the file is missing from the system. If the file is missing, then it means that a certain service is not running; in this case DNS is most likely not activated. If the file does not exist on the system, then Shavlik considers the vulnerability not present, and therefore the patch is not considered missing. If the patch is not 'missing', and it is not explicitly 'installed', then such patch would be reported as 'EffectivelyInstalled'. 

     

    Example 3:
    Sometimes the registry keys are referenced when performing the checks, so the reason is not only limited to files. Here is a snippet for MS12-A04 bulletin.

     

    ... PatchTest.cpp:528 Testing 'MS12-A04'.
    ... PatchTest.cpp:1418 Key 'SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{06b2b7ed-809a-44e6-8538-ca0f5b74ecc4}.sdb'.
    ... PatchTest.cpp:1572 Reg failed reason - The registry key 'SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{06b2b7ed-809a-44e6-8538-ca0f5b74ecc4}.sdb\DisplayName' does not exist. It is required for this patch to be considered installed.

     

    Key notes:
    "Reg failed reason - The registry key 'xxxxx' does not exist. It is required for this patch to be considered installed." – suggests that the patch is missing. If this patch were installed, then this particular registry entry would be present.

     


    5. Did the scan complete gracefully or crash

     

    If the scan completes and BLPatchCheck2 exits gracefully, the following messages are logged to Trace.txt towards the end:

    [target_hostname] ... MultiMachineScanner.cpp:955     Generating output for = [target_hostname]
    [target_hostname] ... MultiMachineScanner.cpp:956     Scanned Server Count = 0, MAX count = 1

     

    If the lines are not present, then the Patch Analysis Job most likely failed. Often, in the Trace.txt file there will be another error message towards the end of the log.

     


    6. Error messages and troubleshooting

     

    Check if the error message from Trace.txt is documented in KCS.
    When validating a conditions stack (reason), cross reference against the vendor site that the correct file and version is checked, and if necessary which branch is the applicable one (GDR or LDR).