Recently I needed to troubleshoot an issue with BPPM Publishing where Impact Models were not being successfully published in BPPM.
Some smaller Models were successful, other larger Models would just not publish.
The Models showed up correctly in Remedy (Impact Designer), but when attempting to publish to BPPM they would time-out or throw a large number of errors in the BPPM logs.
Looking at the log files from AR System, I could see lengthy queries being executed from BPPM that were taking upwards of 20 minutes to execute before ultimate failure of the publishing occurred. SQL analysis showed queries that were using "Null" values, so adding indexes was out of the question to speed up the queries and help the process.
Looking into the API calls that were executed, I could see that BPPM was executing an API query using the "CMDBGraphWalkQuery" function with no batching and no levels (which was effectively pulling the full model down across all classes) to "walk" the Model and obtain all the CI's and Relationships.
Service Models for this client were loaded via a 2 step process:
- Main Model was imported via ETL (AI Jobs/Transformations) from an Excel spreadsheet
- The remainder of the Model was imported from multiple external data sources, again via ETL (AI Jobs/Transformations)
This caused a problem when it came to troubleshooting the root cause as there was no "one source" of information where the full Model resided. We analysed the Excel spreadsheet for anomalies and corrected, but no amount of troubleshooting and diagnostics was able to pin point the actual issue across the full Model.
For BPPM to successfully pull down the Model (CI's and Relationships), it needs to know what BPPM Cell to publish the the Model too. In a large BPPM implementation, there maybe more than one BPPM Cell configured and the BPPM environment can span multiple networks, be segmented off based on security requirements, etc, so knowing where to Publish a Model too (target Cell) is key to a successful publishing attempt.
Eventually we determined through the BPPM logs that there were multiple CI's that were missing the "HomeCell" and "HomeCellAlias" values in the CMDB records as required by BPPM. This may have been due to the way the Models were loaded, or another issue with the propagation of the Cell values via the Reconciliation process.
With the larger Models having upwards of 50k + CI's (and Relationships), identifying each and everyone of the CI's related to a particular Model was practically impossible due to the way the data was loaded (and Infrastructure CI's could be used across more multiple Models). So we were left with going line by line through the error logs to identify the offending CI's and updating manually, or coming up with another method to extract the data.
We eventually determined that the only way to get past this issue was to "emulate" what the BPPM Publishing process was doing to be able to see what CI's form part the Model we were attempting to Publish.
From here, we could also create a "self documenting" method for a Model where all the related CI's were exported to a file where we could see where the values were missing and then schedule a bulk update to set the "HomeCell" and "HomeCellAlias" values.
Attached is the Java application that was created that uses the "CMDBGraphWalkQuery" API call to "walk" the Models from the top to bottom and produce a "csv" file showing all related CI's to that Model, including the the "HomeCell" and "HomeCellAlias" values.
It can walk a single Model or multiple Models (using the configuration file to set parameters).
There are options to update the values on the fly or just walk the Model and produce an output file (please see the Instruction file included).
To use, just extract the zip to a directory of choice (the file structure in the zip needs to be kept intact) and run the program either using the command line options or the configuration file.
I hope this helps anyone else facing a similar issue with publishing to BPPM.
mpsCMDBGraphWalkQuery.zip 16.0 MB