Skip navigation

Diagnostic Solutions

May 2013 Previous month Next month
Share This:

As a long-time member of Customer Support, I spend a lot of time thinking about how to isolate and resolve issues more quickly.  Once I find the root cause, I ask myself,  “How could I have found it more quickly?”  There are differences between BMC products and the issues that occur, but also similarities in the investigative process. I have found some practices that work well for me and in this blog, I hope to share those tools, techniques and resources that have been useful across several BMC products.

 

If you are not familiar with it, the Maintenance Tool is installed with many BMC products to simplify diagnostic and maintenance activities, including:

  • BMC Cloud Lifecycle Management
  • BMC Atrium CMDB Suite
  • BMC Atrium Orchestrator
  • Most of the BMC Remedy products and applications, including BMC Remedy AR System Server, BMC Remedy IT Service Management Suite
  • BMC ProactiveNet Performance Management.

 

The exact name, location, and features are specific for the product and version which installed the maintenance tool but it is typically in a subdirectory of the product with a file name such as:

  • <product name>MaintenanceTool.cmd (Windows)
  • <product name>MaintenanceTool.sh  (UNIX)

 

The most commonly used feature of the Maintenance Tool is Zip Logs, used to collect diagnostics to a file which can be provided to Customer Support.  This output file collects the most commonly needed product, environment, and diagnostic information. In the most recent product versions, the Maintenance Tool also includes features to review these collected diagnostics and perform elementary analysis.  These are the features I will describe in this post.

 

If the Browse to Log button appears on the Logs page, the features described in this post should be available in the Maintenance Tool.  You can use any Maintenance Tool that has these features to view a Zip Logs output file, even if the zip logs file was created from an earlier version of the product or from a different BMC product.

 

browseToLog2.png

 

To open files:

  1. Click Browse to Log
  2. In the navigation tree, open a directory or Zip file to view the included files.
  3. Double-click a file to open it in the viewing pane.
    The file opens in the best viewer for that file.

 

Below are some files included and the viewer which opens to view them:

 

FileLocationViewerContent
ProductRegistry.xmlWindows directory (Windows)
etc directory (UNIX)
Paged XML viewer

Product versions.
Product install locations.

Product and feature installation history.

OperatingSystemsData.xml

User temporary directory on the origin systemPaged XML viewerEnvironment details at the time Zip Logs was performed.  Includes environment variables and may include System Memory, Partitions, Registry Keys, status of Ports, or other system details.
<product>InstalledConfiguration.xmlUser temporary directory on the origin systemPaged XML viewerEnvironment information and installation options during the last successful installation.
<product>InstallingConfiguration.xmlUser temporary directory on the origin systemPaged XML viewerEnvironment information and installation options during the last attempted installation.

<product>_install_log.txt

<product>_uninstall_log.txt

User temporary directory on the origin systemLog viewerInstall logs and uninstall logs.
<product>_configuration_log.txtUser temporary directory on the origin systemLog viewer

Activity in the Maintenance Tool, including

Zip Logs and Health Check (in some products)

Product specific configuration filesUnder product installation directory on origin systemText viewerProduct configuration
Product specific log filesUnder product installation directory on origin systemText viewerProduct log file
Product specific image filesNo standard image files collected.Image viewerView the image.

 

Each file opens in a separate page in the appropriate viewer for the file type..  This makes it easier to examine and compare information in multiple files.   For example, you may view a screenshot of a problem to find a value and search for that value in a collected log file.

 

Features of Viewers

ViewerFeatures
Paged XML Viewer
  • Resize columns
  • Sort columns by clicking header
Log Viewer
  • Resize columns
  • Sort columns by clicking header
  • Indicator of highest severity level in the log, count of exceptions if you hover over the colored box in right scroll area
  • Color bars indicate exceptions on the page, hover over them to see details, or click them to jump to that line in the log
  • Build boolean queries to find strings or regular expressions in the log
  • Save queries
  • Load saved queries
  • Filter log to only display lines matching a query
  • View number of matches of query
Text Viewer
  • Find a string or regular expression in files
  • Save queries
  • Load saved queries
  • Filter for lines in file with a string or regular expression
  • Go to a line number in a file
Image Viewer
  • Supports most image file formats.
  • No other features beyond displaying the image.

 

I recently recorded a short video showing how I use the Maintenance Tool to view files in the Log Zipper output file.  This provides an overview of the process I use in general, but I wanted to share two more examples in detail.

 

Checking for missing configuration files

 

When I click Zip Logs, the Log Zipper collects the product configuration files and log files needed for analysis.  The Log Zipper is the tool that collects the files, but all the logic of what files to collect and where they should be found on the system is installed with the product.  This means, if the Log Zipper is unable to collect a file it means the file is not in the location expected by the product itself.

 

Here is how I screen for issues of missing configuration files:

  1. Launch the Maintenance Tool
  2. Click Zip Logs and wait for it to complete.
  3. Click Configuration Log.
    This opens the <product>_configuration_log.txt file in the viewer.  You could also navigate and find the file in the Log Zipper output file.
  4. Choose Edit > Filter
  5. Right-click on the default query ( Details containsIgnoreCase '' ) and add or change clauses as necessary.
    You can filter on "Adding file" to see all the files or directories which Log Zipper attempts to collect but rather than looking at this list, I  constucted a query to search for lines where Details contains "Not found" and Severity is INFO, WARNING, or SEVERE, as shown below:
    Filter_missing_files2.png

The output indicates the file name and location where the file was expected.  Close the filter dialog and uncheck the filter checkbox to return the unfiltered results.  See Searching in the log viewer for more details on defining search queries.

 

After testing the query, you can choose Edit > Save Search Query in the Filter window.

SaveQuery.png

 

Specify a value for Short Description.  For example, I call this search "Missing files".

The other attributes can be left blank.  Long Description, Source Description, and Resolution Description hold details on why you are performing the search, information about the string if found, and how to resolve any issues found.  There attributes are not visible or used in the current Maintenance Tool, so they do not need to be populated.

 

Searching for errors in a log file

 

Another useful activity when analyzing an issue is to search for errors in a log file.  However, there are many different error code prefixes or indications of an error, including:

  • ARERR
  • BMC- 
  • ErrorCode
  • Error msg
  • ORA-
  • SQL Server
  • Exit code
  • ReturnCode
  • MessageNum
  • java.lang
  • SEVERE

 

Searching for each of these strings individually takes time and familiarity with the log.  It also makes it difficult to see patterns of errors such as when one error is always preceded by a different error.

 

To simplify this process, I use a query to search for all common error prefixes.

 

When viewing logs in the log viewer, this query is constructed using an OR match type, and a list of strings.  Filter_errors_log_Viewer2.png

Typically, this query is not needed in the log viewer because the Severity column indicates when an error has occured, but the query can filter these lines to ones with specific product errors.

 

When a file is open with the text viewer:

  1. Choose Edit > Filter
  2. Check the regex option, to enter the search using a regular expression.
  3. Enter the list of search expressions, separated by a pipe character so it will return results on any of the search strings, as shown below, and click Filter.

 

Filter_errors_text_viewer.png

This will show just the lines which contain one of these search strings.

 

Save the query for future use by choosing Edit > Save Search Query.

 

Note: This technique of using a regular expression to search for several indications of an error is not unique to the Maintenance Tool.  Many text editors also have the ability to search on a regular expression.  However the Maintenace Tool allows you to do this in one pane when viewing a log, and then easily switch to the OperatingSystemsData file to view information about the system configuration or environment variables, and perform the same sort of analyis across several files to compare related information.

 

I hope this information in this blog post is helpful, perhaps as a quick check before doing a deep analysis of an issue. I am interested in feedback on this topic. How does this technique work for you? Are there other quick searches worth saving and running routinely as a pre-analysis?