As a long-time member of Customer Support, I spend a lot of time thinking about how to isolate and resolve issues more quickly. Once I find the root cause, I ask myself, “How could I have found it more quickly?” There are differences between BMC products and the issues that occur, but also similarities in the investigative process. I have found some practices that work well for me and in this blog, I hope to share those tools, techniques and resources that have been useful across several BMC products.
If you are not familiar with it, the Maintenance Tool is installed with many BMC products to simplify diagnostic and maintenance activities, including:
- BMC Cloud Lifecycle Management
- BMC Atrium CMDB Suite
- BMC Atrium Orchestrator
- Most of the BMC Remedy products and applications, including BMC Remedy AR System Server, BMC Remedy IT Service Management Suite
- BMC ProactiveNet Performance Management.
The exact name, location, and features are specific for the product and version which installed the maintenance tool but it is typically in a subdirectory of the product with a file name such as:
- <product name>MaintenanceTool.cmd (Windows)
- <product name>MaintenanceTool.sh (UNIX)
The most commonly used feature of the Maintenance Tool is Zip Logs, used to collect diagnostics to a file which can be provided to Customer Support. This output file collects the most commonly needed product, environment, and diagnostic information. In the most recent product versions, the Maintenance Tool also includes features to review these collected diagnostics and perform elementary analysis. These are the features I will describe in this post.
If the Browse to Log button appears on the Logs page, the features described in this post should be available in the Maintenance Tool. You can use any Maintenance Tool that has these features to view a Zip Logs output file, even if the zip logs file was created from an earlier version of the product or from a different BMC product.
To open files:
- Click Browse to Log
- In the navigation tree, open a directory or Zip file to view the included files.
- Double-click a file to open it in the viewing pane.
The file opens in the best viewer for that file.
Below are some files included and the viewer which opens to view them:
|ProductRegistry.xml||Windows directory (Windows)|
etc directory (UNIX)
|Paged XML viewer|
Product and feature installation history.
|User temporary directory on the origin system||Paged XML viewer||Environment details at the time Zip Logs was performed. Includes environment variables and may include System Memory, Partitions, Registry Keys, status of Ports, or other system details.|
|<product>InstalledConfiguration.xml||User temporary directory on the origin system||Paged XML viewer||Environment information and installation options during the last successful installation.|
|<product>InstallingConfiguration.xml||User temporary directory on the origin system||Paged XML viewer||Environment information and installation options during the last attempted installation.|
|User temporary directory on the origin system||Log viewer||Install logs and uninstall logs.|
|<product>_configuration_log.txt||User temporary directory on the origin system||Log viewer|
Activity in the Maintenance Tool, including
Zip Logs and Health Check (in some products)
|Product specific configuration files||Under product installation directory on origin system||Text viewer||Product configuration|
|Product specific log files||Under product installation directory on origin system||Text viewer||Product log file|
|Product specific image files||No standard image files collected.||Image viewer||View the image.|
Each file opens in a separate page in the appropriate viewer for the file type.. This makes it easier to examine and compare information in multiple files. For example, you may view a screenshot of a problem to find a value and search for that value in a collected log file.
Features of Viewers
|Paged XML Viewer|
I recently recorded a short video showing how I use the Maintenance Tool to view files in the Log Zipper output file. This provides an overview of the process I use in general, but I wanted to share two more examples in detail.
Checking for missing configuration files
When I click Zip Logs, the Log Zipper collects the product configuration files and log files needed for analysis. The Log Zipper is the tool that collects the files, but all the logic of what files to collect and where they should be found on the system is installed with the product. This means, if the Log Zipper is unable to collect a file it means the file is not in the location expected by the product itself.
Here is how I screen for issues of missing configuration files:
- Launch the Maintenance Tool
- Click Zip Logs and wait for it to complete.
- Click Configuration Log.
This opens the <product>_configuration_log.txt file in the viewer. You could also navigate and find the file in the Log Zipper output file.
- Choose Edit > Filter
- Right-click on the default query ( Details containsIgnoreCase '' ) and add or change clauses as necessary.
You can filter on "Adding file" to see all the files or directories which Log Zipper attempts to collect but rather than looking at this list, I constucted a query to search for lines where Details contains "Not found" and Severity is INFO, WARNING, or SEVERE, as shown below:
The output indicates the file name and location where the file was expected. Close the filter dialog and uncheck the filter checkbox to return the unfiltered results. See Searching in the log viewer for more details on defining search queries.
After testing the query, you can choose Edit > Save Search Query in the Filter window.
Specify a value for Short Description. For example, I call this search "Missing files".
The other attributes can be left blank. Long Description, Source Description, and Resolution Description hold details on why you are performing the search, information about the string if found, and how to resolve any issues found. There attributes are not visible or used in the current Maintenance Tool, so they do not need to be populated.
Searching for errors in a log file
Another useful activity when analyzing an issue is to search for errors in a log file. However, there are many different error code prefixes or indications of an error, including:
- Error msg
- SQL Server
- Exit code
Searching for each of these strings individually takes time and familiarity with the log. It also makes it difficult to see patterns of errors such as when one error is always preceded by a different error.
To simplify this process, I use a query to search for all common error prefixes.
Typically, this query is not needed in the log viewer because the Severity column indicates when an error has occured, but the query can filter these lines to ones with specific product errors.
When a file is open with the text viewer:
- Choose Edit > Filter
- Check the regex option, to enter the search using a regular expression.
- Enter the list of search expressions, separated by a pipe character so it will return results on any of the search strings, as shown below, and click Filter.
This will show just the lines which contain one of these search strings.
Save the query for future use by choosing Edit > Save Search Query.
Note: This technique of using a regular expression to search for several indications of an error is not unique to the Maintenance Tool. Many text editors also have the ability to search on a regular expression. However the Maintenace Tool allows you to do this in one pane when viewing a log, and then easily switch to the OperatingSystemsData file to view information about the system configuration or environment variables, and perform the same sort of analyis across several files to compare related information.
I hope this information in this blog post is helpful, perhaps as a quick check before doing a deep analysis of an issue. I am interested in feedback on this topic. How does this technique work for you? Are there other quick searches worth saving and running routinely as a pre-analysis?