In this post I will describe the details associated with DSO which will be helpful for Users to understand and troubleshoot DSO related problems.
The topics I’ll cover in the post are:
1. General Information about DSO
2. Key Information required for DSO to work
3. Location of DSO logs
4. Simple DSO health check
5. Examples for most common errors
6. Performance tuning DSO process
1.0 General Information about DSO in CLM
With CLM 2.x / 3.x, the Remedy component was divided into Enterprise AR (EAR) and Cloud AR (CAR) to uniformly balance the load. Whereas Enterprise AR provided a platform to provide ITSM / foundation / Service Request Management related functionalities, Cloud AR was introduced to manage the cloud related Configuration Items (CI’s). Both the server details used to be added to Midtier configuration details (under AR Server Settings), to present a single portal for the User to manage and request CLM related services. As such it was required that common CI’s between the two servers be synced whenever there is an update on these CI’s on either of the servers. Thus the process of DSO was introduced to ensure that whenever there will be an update on the related CI on one of the servers, the DSO process will ensure that the related data on the other server is updated by updating the BMC.IMPORT DatasetID, which in turn will get reconciled to BMC.ASSET Dataset ID by the reconciliation job.
Which forms are mapped to DSO process and where is the information stored?
Distributed Mapping form contains the details of the various CI’s which are updated between the two Remedy servers. The form is present on both the AR servers and states conditions for CI’s which will be updated to the destination server. The screenshot below shows the list of CI’s which are updated from Enterprise to Cloud AR.
Another form “Distributed Logical Mapping” has entry for the Logical Name as “DESTINATION-SERVER” which is used in the Distributed Mapping form. Care should be taken to ensure that the Physical Name corresponds to the hostname of the Target AR Server.
1.1 Configuring DSO
The DSO related parameters in Distributed Mappings forms and ar.cfg file are configured when installing EAR and CAR through Install Planner. The component “DSO Mapping” in the AR server installation through Install Planner is responsible for installing and configuring the DSO Mappings. More information on these parameters and configuration is shared in the below section.
2.0 Key Information required for DSO to work
There are some common parameters and configurations which need to be configured for DSO to work. They are:
- License Information:
DSO requires AR Distributed Server license to be configured in both Enterprise and Cloud AR. Users should also keep a check on the number of AR fixed licenses along with the DSO license (this can be checked by checking the number of AR fixed User licenses which should be more than the number of Users configured for CLM environment).
The licenses can be viewed (as shown in the below screenshot) from AR System Administration Console -->Server -->General -->Add or Remove Licenses.
- Identification of C Process and JAVA Process for DSO:
Armonitor.conf file consists of a list of processes which are enabled when the AR System starts up. This file is usually located in C:\Program Files\BMC Software\ARSystem\Conf for Windows machines and in /etc/arsystem/<server name>/armonitor.conf for Linux machines.
In AR 7.6.04 (which was used by default in CLM 2.x) and below, C processes were used. Therefore the DSO related process to check here is:
"C:\Program Files\BMC Software\ARSystem\serverds.exe" --unicode -i "C:\Program Files\BMC Software\ARSystem" –m
/opt/bmc/ARSystem/bin/arservdsd -s vx-aus-cus-spxx -i /opt/bmc/ARSystem
From AR 8.x onwards, JAVA related processes were used, the DSO process to check in CLM 3.x environment onwards is:
"C:\Program Files\Java\jre7\bin\java" -Xmx512m -classpath "C:\Program Files\BMC Software\ARSystem\dsoj;C:\Program Files\BMC Software\ARSystem\dsoj\ardsoj81_build001.jar;C:\Program Files\BMC Software\ARSystem\arserver\api\lib\arapi81_build001.jar;C:\Program Files\BMC Software\ARSystem\arserver\api\lib\arcmnapp81_build001.jar;C:\Program Files\BMC Software\ARSystem\arserver\api\lib\arutil81_build001.jar" com.bmc.arsys.dsoj.DSOServer --unicode -i "C:\Program Files\BMC Software\ARSystem" –m
/usr/java/jdk1.7.0_21-64bit/jre//bin/java -Xmx512m -classpath /opt/bmc/ARSystem/81/dsoj:/opt/bmc/ARSystem/81/dsoj/ardsoj81_build001.jar:/opt/bmc/ARSystem/81/api/lib/arapi81_build001.jar:/opt/bmc/ARSystem/81/api/lib/arcmnapp81_build001.jar:/opt/bmc/ARSystem/81/api/lib/arutil81_build001.jar com.bmc.arsys.dsoj.DSOServer --unicode -i /opt/bmc/ARSystem/81 -m
The most common occurrence for DSO issues is due to these entries being commented.
Note: Any changes to the entries in armonitor.conf file even removing “#” from the commented entries require AR Server restart for the changes to take effect.
- Parameters to check in ar.cfg file
AR configuration file has various configuration entries for AR Server. This file is usually located in C:\Program Files\BMC Software\ARSystem\Conf on Windows machines and in /opt/bmc/ARSystem/conf on Linux machines.
Some important parameters to note in this file are:
Register-With-Portmapper: is the default configuration setting of the portmapper
TCD-Specific-Port: TCP port for the arserver process. If this option is not set, the dispatcher is randomly assigned to an available port. (TCD stands for transaction control daemon.)
DSO-User-Password: Password for the local DSO server user. This is encrypted password. To check the password or modify it in case of correction, please use the utility mentioned under https://kb.bmc.com/infocenter/index?page=content&id=S:KA404774
DSO-Target-Password: Password used to access the target AR System server through the DSO server.
Use this format: DSO-Target-Password: serverName:encryptedPassword
DSO-Target-Connection: Information for the target AR System server.
Use this format: DSO-Target-Connection: serverName:RPCNumber portNumber
Note: Users are advised to refer the AR Configuration document for the respective AR server version before making a change to the port as it could have a dependency on another configuration or a range in which the Ports can be selected.
3.0 Location of DSO logs
Location of DSO Log (ardist.log): Typically the DSO Log is located in <AR_HOME>\ARSystem\Arserver\Db\ardist.log. There is a separate file created per DSO Pool.
There are a couple of places to enable the DSO logging to troubleshoot DSO issues:
1: AR System Administration Console-->Server-->General-->Server Information-->DSO (tab):
Check the “Log Errors to DSO Pending Error Form”:
2: AR System Administration Console-->Server-->General-->Server Information-->Log Files
Check ‘DSO Log’ and set the log level to “Info”. Click on apply and save.
We can also set the Logging Level to Error, Warning or Info as per the requirement. Thus the standard set of logs which are helpful to troubleshoot the DSO related issues are arerror.log and ardist.log
4.0 Simple DSO health check:
In order to check whether DSO is working on both Enterprise and Cloud AR, we can perform the following steps:
1. E-AR to C-AR: Open BMC.CORE:BMC_Tag form. Select a record with DatasetId = BMC.ASSET
Edit short description and see if it gets updated on C-AR (on same form and same DatasetId= BMC.IMPORT.ENTERPRISE)
2. C-AR to E-AR:Open BMC.CORE:BMC_Tag form. Select a record with DatasetId = BMC.ASSET
Edit short description and see if it gets updated on E-AR (on same form and same DatasetId= BMC.IMPORT.CSM)
3. Go to AR Server (Cloud or Enterprise) Home Page --> AR System Admin Console --> System --> Distributed Server Option --> Double click Pending DSO Operation --> In the 'Pending DSO Form' perform a search if there is no result that means DSO is working fine however if the search displays some results, it does not mean that DSO is not working, it means the records are pending to be DSO’d, however if the list of the pending records does not decrease or reduce over time, it is an indication of DSO not working properly.
Suggestions to make the DSO work:
a. Restart the java process associated with DSO if DSO process is stuck from quite some time.
b. Wait for pending operations to get completed.
c. Check for related license errors in arerror.log for AR server or DSO
d. Check for commented out entries for DSO in the armonitor configuration file.
e. Verify the DSO / AR server parameters mentioned in the section 2.0
5.0 Examples for most common errors
AR Distributed Server has been licensed however the “AR Distributed Server” services are not starting on the machine or the Simple DSO health check fails.
Sample error message from ardist.log file:
Processing item number 29 (Thu Nov 03 2011 13:31:57.9857)
<DIST> Pending Type – 1
<DIST> Source Form – BMC.CORE:BMC_BusinessService
<DIST> Source ID – 000000000000103|000000000342119
<DIST> Pending Other --
<DIST> -m “DIST-CMDB:ECMDB_TO_CLOUD-BMC_BusinessService” –p “Pool4”
<DIST> Get source schema definition (stage 2)
<DIST> Using NEW cache definition for BMC.CORE:BMC_BusinessService (clm-itsm)
<DIST> Get entry details (stage 3)
<DIST> Get mapping details (stage 4)
<DIST> Filter-specified mapping – DIST-CMDB:ECMDB_TO_CLOUD-BMC_BusinessService
<DIST> Mapping DIST-CMDB:ECMDB_TO_CLOUD-BMC_BusinessService not in the cache or expired
<DIST> Replacing logical name (DESTINATION-SERVER) with physical name (clm-cloudar)
<DIST> Mapping matching qualification: InstanceId = “Instance ID>”;
<DIST> Mapping name – DIST-CMDB:ECMDB_TO_CLOUD-BMC_BusinessService
<DIST> Target form – BMC.CORE:BMC_BusinessService
<DIST> Target server –
<DIST> Perform final checks (stage 5)
<DIST> Get target schema definition (stage 6)
<DIST> ** WARNING ** Access problem trying to get target form definition, retry later… (Thu Nov 03 2011 13:32:28.1034)
<DIST> Sleeping for 29:32 minutes (Thu Nov 03 2011 13:32:28.1034)
The above error occurs if there is an issue with the network connectivity between Enterprise AR server and Cloud AR. The connectivity between both the between both Enterprise AR server and Cloud AR should be verified. Both the servers should resolve to each other with the hostname and IP etc. Also validate the Port numbers and parameters discussed in section 2.0
The following steps can be performed to troubleshoot the problem:
1: Connect to the "Cloud AR server” and "Enterprise AR server" through the Remedy User tool or through Midtier and open the "AR System Administration" consoles for both the servers.
2: Click on System-->General-->Server Information and open "Ports and Queues" tab on "Server Information" form on both the servers.
3: Check the setting of "Register With Portmapper". If the User is using a port, this field should not be checked either on EAR or CAR server.
4. Make sure that in case a specific port is being used, it populates in AR Administration Console.
a: Open the "Connection Setting" tab on "Server Information" form at the "Enterprise AR server" side and click on "DSO Server"
b: Fill the "Server Name", "RPC Program Number" and the "Port" values of the Cloud AR server in here if the values are blank. Also make sure that the DSO Server Setting Password Table is pointing to Cloud AR, the DSO password can be verified through the ar.cfg file.
The above steps need to be checked on Cloud AR to check for EAR related DSO entries.
Note: A change to the values here will require restarting of the AR services ("BMC Remedy Action Request System Server")
5. Look for the following messages which are an indication of DSO not working. These messages are also captured in csm.logs
<DIST> Perform mapping (stage 7)
<DIST> ** ERROR ** You are already at the limit of the number of fixed user licenses of the following type (ARERR 30)
<DIST> Connection info: Server -- MyCloudDBSrv, Port -- 0, RPC program number -- 0
<DIST> ** ERROR ** You are already at the limit of the number of fixed user licenses of the following type (ARERR 30)
<DIST> ** ERROR ** Failure during mapping attempt (Fri Sep 09 2011 09:18:36.2970)
<DIST> Mapping CSM:DSO:User retrieved from cache
Suggested remediation: Verify that the User account exists in the Cloud AR instance. The user should also be granted a fixed license. If out of sync, touch the record in enterprise AR and verify that the record is updated in cloud AR. If records are not syncing, check to make sure there are available fixed user licenses on Cloud AR.
08 Sep 2011 17:23:14,781 [INFO] DB - [Thread=btpool0-4(31)] [Class=CloudDBRequestProcessor:getResource] - ##START CLOUDDB User[cloudadmin] RETRIEVE##
08 Sep 2011 17:23:14,922 [INFO] DB - [Thread=btpool0-4(31)] [Class=CloudDBRequestProcessor:getResource] - No records found in CloudDB for class : com.bmc.cloud.model.beans.User having uuid : cloudadmin
08 Sep 2011 17:23:14,922 [INFO] DB - [Thread=btpool0-4(31)] [Class=CloudDBRequestProcessor:getResource] - ##END CLOUDDB User[cloudadmin] RETRIEVE success##
08 Sep 2011 17:23:14,922 [ERROR] LOGIN_SERVICE - [Thread=btpool0-4(31)] [Class=LoginService:authenticate(LoginType)] - User : cloudadmin could not be found in Cloud AR. Authentication failed
Suggested Remediation: Increase the number of AR User Fixed License in Enterprise AR and Cloud AR. This will cause the user account to be populated as expected and users will then be able to access the Cloud Portal successfully.
6. Check if AR Server license has expired. If the evaluation period has expired, AR Server license needs to be renewed to utilize the AR server related functionalities including DSO. This information can be observed by checking the arerror.log for entries like the following immediately after the arservices restart:
This version of the Action Request System(R) is ready for use or evaluation without purchasing or activating an authorization key.
For unlimited capabilities, contact your sales representative.
You have reached the maximum number of database entries permitted with this version of the Action Request System(R).
To purchase the unrestricted version, contact your sales representative
6.0 Performance tuning the DSO process:
To fine tune the DSO process, the following steps can be performed and applied in the environment.
1. For EAR server, open the AR Administration Console --> General --> Server Information --> Ports and Queues
2. Add the Type as “Private”, assign “RPC Prog Number” as a range between 390621–390634, 390636–390669, 390680–390694*, configure Min Threads as “2”, Max Threads as “6”
3. Goto the Connection Settings Tab, click on DSO Server and add the selected “RPC Prog Number” for “DSO Local RPC Program Number” field.
4. Also care should be taken to ensure the CAR DSO details are updated here. In the DSO Server Setting Table, update the RPC Program Number which is configured for CAR Server as “DSO Local RPC Program Number”
Perform the steps 1-3 on CAR server for configuring a Private RPC and make sure to update the EAR RPC Program Number field under DSO Server Setting Table.
Note: It is advisable to restart the AR services for these changes to take effect.
* Users are advised to go through the link:
Trending in Support: Queues & Threads – Utilizing Private Queues to get more information on utilizing and configuring private queues.
In this post, I have covered the various configuration parameters and key information on working and troubleshooting of DSO. I hope it was helpful. I would like to hear from you. Please add your comments below.
You can also join the The specified item was not found. to learn about and provide feedback on ways Customer Support can enable your success.
Visit BMC Cloud Lifecycle Management Support Blogs .You can share the links with others if you find these useful.
Please rate this blog, regarding the usefulness of the shared information, by clicking below option.