In this month’s Remedy Product blog we will discuss Unified Data Management (UDM). This feature was introduced starting with IT Service Management 8.x Application version. In this blog post, I will share tips on how it works and how to avoid common issues. I will cover the following topics:
- UDM Features
- Initial Configuration
- UDM Components
- Creating and Running Jobs
- Server side logging
- Troubleshooting load issues Error handling
- Some useful links related to UDM
Data management help to reduce the manual input of numerous records throughout the system and ease the process of creating/updating many types of records and on-boarding new system users, support groups, etc.
Here is a simplified way to view implementation of UDM.
There are a number of benefits to the UDM-based method of loading data:
- Unified approach of data management across the solution stack
- Job-based framework
- Automated job scheduling
- Multiple users can perform simultaneous data loads
- Multi-tenancy, which means users can create data load jobs and templates restricted to specific companies
- Centralized error handling
- Case insensitivity correction and alias replacement during validation
- Loading data from various sources;
- Lightweight Directory Access Protocol (LDAP) source files,,,,Microsoft Excel
- Everything is web-enabled and driven from AR Server.
- To improve application performance, system administrators can configure system messages from filters and servers to appear in a message bar instead of in pop-up windows
There are a few steps required when first using UDM.
- Update Unified Data Management Settings
- After configuring the Data Management Load file path verify it by click on the Test button to verify it was entered correctly.
- A user must be a member of the appropriate permissions group in order to access the Data Management (DM) features, either DMT Admin or DMT User.
- More Information you can see on Configuring the BMC Remedy AR System repository for attachments and Spreadsheets
- Setting default preference for all data load jobs. For more information please review: Configuring Data Management application preferences
NOTE: You can review more information on Change in Company column information from the Data Spreadsheet on this link from BMC Documentation portal
Components of UDM:
Below is a diagram of the components of UDM, and below, a brief summary of them.
Atrium Integrator adapter: This plug-in enables you to create and run your own data transformations.
The adapter is used in transformations to import data into BMC Remedy AR System or to extract data from BMC Remedy AR System into other data sources
Load template: The load template functionality enables you to select an out-of-the-box template or previously-saved custom template that you created for a job. An out-of-the-box template already has a set of steps defined for it.
Pentaho Data Integration (PDI) Engine: This engine used for Extraction, Transformation and Load activity. PDI Is packaged with AR System Install By default with AR Server.
It extracts data from particular data source, transforms that data into a particular format as per business need, and Loads that data into the target form.
Spoon : GUI for designing and testing transformation jobs
Repository: Stores Transformation and jobs definition in AR forms basically act as Repository
Carte Server: Separate process with HTTP interface with HTPP based data integration server which executes transformation and jobs. The communication between Carte Server and AR server done by ARDBC plugin
Creating and Running Jobs:
Users can create data load jobs manually or with using Templates. One or more templates can be used in one job. The UDM Job engine will first Load the data then validate it and after successful validation, the data get promoted.
User can run data Jobs immediately, or scheduled (one time, interval etc.). Interval jobs are used more often for loading data from LDAP or other custom sources. For loading Attachment data we recommend to use one time job.
Following log files are useful while troubleshooting the functionality behavior from UDM.
1) arcarte.log — All Execution Instance messages are recorded in this file according to the logging level specified when you created the Execution Instance. ( This file exists in <AR Install Dir>/db directory)
2) arcarte-stdout-<timestamp>.log — All messages that the Carte server prints to the console are recorded in this file.
3) arcarte-stderr-<timestamp>.log — All error messages that the Carte server prints to the console are recorded in this file.
Troubleshooting data load issues:
Below you find some suggestions to investigate if data load process is not executing correctly.
- What if Load process hangs?
This can be due to a slow system, logs may be turned on for some reason or it can be the Initial job creation. The first job will take longer time than normal as the engine needs to build dependency and sequence for Load, Validate, and Promote at the back end.
Additionally make sure escalations are not disabled on Server. To get supportive information of the current data load job status, administrator user can query the UDM:ExecutionStatus form with matching the execution Instance name with the instance ID of Load Step).
If no record found, then Atrium Integrator engine is not invoked
- What if Validate/Promote Steps stuck with ‘In progress’ status?
Check if CAI plugin is running on server or not. For informative error messages about current job please check arerror.log.
Below error message get captured into arerror.log when there are multiple concurrent jobs running on carte server:
java.lang.OutOfMemoryError: Java heap space
NOTE: Load Error panel enables user to understand if any required field data is missing, provided Invalid selection value or incorrect format in Spreadsheets
To validate error messages please open Error Management Console, check your particular job and correct staging data. You can see validation error messages with clicking View data button.
While loading asset management data please wait until CI reconciliation occurs and the Update CI/CI Relationship data. If any errors occur during reconciliation, check Reconciliation job logs.
If any error like “Invalid login ID or Role” occurs then you can correct it using the Error Management Console by selecting a particular job and click on Update Staging data Button.
After updating the data values please execute the validation job with using “Re-run Validate” button
Data Promotion job errors get occur while moving the data to the target form. To get detail information, you may copy the job, create a wait state, and keep minimum logging to identify the problem.
In the case of CI/CI Relationship loads, do not use the Continue button - wait for Reconciliation to complete. Check the status of the Reconciliation Job in the Reconciliation application.
Below please find some useful Links which provides information on UDM data load functionality:
In this post, I tried to summarize my experience supporting UDM over the last few months, and some of the methods I use to investigate issues. I hope it has been helpful, please add comments below to share your own experience.
You can find similar posts like this in BMC Remedy Pulse Blogs.