Share This:



Logging Table

$AR_SYSTEM_HOME default is "/opt/bmc/digitalworkplace".


Problem Area

ITSM/SRM integrationCombined API/SQL/Filter logging while reproducing the problem, arerror log, remote action log



Workflow DesignerBundle log, F12 Browser Developer Mode/process log$AR_SYSTEM_HOME/db
UIBundle log, Jetty log, F12 Browser Developer mode & Fiddler$AR_SYSTEM_HOME/db
ConnectorsBundle log, arerror log, F12 Browser Developer mode, Connector logging$AR_SYSTEM_HOME/db
ReportingBundle log, arerror log$AR_SYSTEM_HOME/db
DWP Client Integration

Bundle log, debug on the DWP side, F12 Browser Developer mode & Fiddler
DWP Catalog debug level



ApprovalsCombined API/SQL/Filter logging while reproducing the problem, remoteaction.log, bundle.log, arerror log



Installationarsystem_install_log.txt, startup logs



Post Install scriptsarerror log, post install log



RSSOarjavaplugin & bundle log, rsso agent logs, rsso debug$AR_SYSTEM_HOME/db
User Group Syncuser synch log, debug log combined api & SQL logging


"/src" directory

Catalog Workflow*dwpc_process.log$AR_SYSTEM_HOME/db/tenant/dragonfly/coffeecom/db
Catalog communication*dwpc_api.log, dwpc_sql.log


*Available from 19.11 only

Tip: To quicky get the bundle log without having to log into the server, type the following in your browser (where dwpchostname is the name of the Catalog Server).


Similarily, the following will also retrive the DWP log



Catalog File System


Most of the Catalog related files are contained within the "digitalworkplace" directory, some of the more important directories and files are highlighted in the following illustration



OSGI Console


The Catalog application is run through a variety of different bundles which are loaded once the "dwpcontroller" script is run as part of an platform restart. Unlike the DWP Client, we cannot restart the individual components of the Catalog (without negative consequence) however we can check the Catalog's bundle status by using the Eclipse OSGI Console (Open Services Gateway Initiative).


The logging level for OSGI is controlled in the  "$AR_SYSTEM_HOME/conf/" file



# Enable osgi console for debugging


osgi.console = 12666

To connect to the console you simply telnet to the Catalog host and execute the "ss" command.


[root@clm-aus-123]# telnet localhost 12666


Some of the more important Catalog related bundles can be identified by "myservice"


If you notice any of these bundles are in a "RESOLVED" state you can attempt to restart as follows:


Stop "ID#"

Start “ID#”

In the event the Catalog login page becomes unavailable for all users, restart the "com.bmc.myservice.uiservice-war_0.0.1" and try again. If you are still unable to log in please raise a case with BMC Technical Support. Type "disconnect" to end the session !


The connectors and bundles are held within the $AR_SYSTEM_HOME/deployedsmart bundles directory.
























DWP Controller Scripts


The "rxscripts" directory and sub-directories contain many scripts which are called by the Catalog but in some cases you may also run them manually.


/bin/        Checks migrations

/bin/             Gets called by the post install script

/bin/                  Checks the current user who’s in the current session

/bin/tenant/                     Lists all the tenants

/bin/tenant/                 Deletes a tenant

/bin/connector/         Refreshes the available connectors

/bin/         Checks if the Catalog is up or down

/bin/                               Used to log into Remedy from the command line

/bin/             Get's called when DB migrations are required

/bin/                            Sets up the environmental variables to allow the API endpoints

/bin/                  Get’s called by the post install script

/bin/                          Get's current version by running

/bin/service/              Lists all catalog services visible by the current DWP user

/bin/process_def                      Remove/find workflow

In order to run these scripts you need to set up your environmental variables by sourcing the "" script

[root@clm-aus-123 bin]# source


Once you source the "setenv" script you then need to log into the Catalog from the command line and generate an authentication cookie. Once the mycookies key is generated subsequent scripts (provided they are in the same directory as the mycookie.txt) will run.


[root@clm-aus-123 bin]# ./ password




[root@clm-aus-0123 bin]# ls -lrt

-rw-r--r-- 1 root root   610 Jun 15 14:59 mycookies.txt


In the following example I'm running the "" script to obtain information about a problematic tenant. We can also delete and recreate a tenant using the post install scripts (be careful deleting..)

[root@clm-aus-123 bin]# tenant/

{"totalSize":1,"data":[{"name":"coffee","domainIdentifier":"","tenantId":"f53893b1d382","databaseName":"ARTenantCoffee","status":"Enabled","overlayGroupId":"f53893b1d382","activationStatus":"Completed"}]}[root@clm-aus-0123 bin]#

[root@clm-aus-123 bin]# tenant/ coffee   


Stopping & Starting the Catalog


We use the "dwpcontroller" script from the $AR_SYSTEM_HOME directory to restart the Platform and Catalog application

This script also restarts the Platform

[root@clm-aus-0123 digitalworkplace]#./dwpcontroller [-u $user_name] [-p $password] <stop|start|restart>


The "start" option calls “/sb/” which does the following:


  • Runs “$AR_SYSTEM_HOME/bin/arsystem start”
  • Checks if the DB/Port is up
  • If the DB is up it calls another script which checks the bundles status
  • If bundles are up another migration script runs which checks the DB to see if anything has changed (such as JARs) and may update a column or add a new table.


logback_server.xml debug


Logging level is controlled within the "logback_server.xml" which is located in the $AR_SYSTEM_HOME/conf directory


Please backup the logback_server.xml before making any changes, XML mistakes may impact the Catalog on start-up.

level=all    Best level of debug but can impact performance



  <appender name="BUNDLE" class="com.bmc.arsys.logging.ReconfigurableRollingFileAppender">








                <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">



                <!-- Extra param to say log file creation at startup -->



                        <pattern>%d{MM-dd HH:mm:ss.SSS} %-4le %c %m%n</pattern>





From 19.02 + you can control Catalog logging from the settings page.


RSSO debug can be written to the bundle log by adding the following lines under the existing bundle section.


   <logger name="com.bmc.rsso" level="DEBUG" additivity="false">

                <appender-ref ref="BUNDLE" />



RSSO logging is enabled by making the following changes:


a) $AR_SYSTEM_HOME/pluginsvr/log4j_pluginsvr.xml file


<logger name="com.bmc.arsys.pluginsvr">

<level value="debug" />




<priority value ="error" />

<appender-ref ref="PluginLog" />


b) $AR_SYSTEM_HOME/bin/arserverd.conf. Go to the last of the jvm.option.xx. on the next line, add this (make sure the number is +1 of the previous line. In this example, the previous line was 24)


c)  $AR_SYSTEM_HOME/conf/rsso-log.cfg





And restart (dwpcontroller stop/start).


RSSO Diagnostics


Starting with 19.02 we have a new diagnostic tool which checks RSSO configuration is correct. This tool is located in the $AR_SYSTEM_HOME/sb/configure_rsso directory. To run open the "" script and make sure it's the paths and diagnostic file name is correct:




# Path to Catalog directory.






if [ $# -eq 0 ]; then






if [ "X${JAVA_HOME}" = "X" ]; then






${JAVA_EXEC} -jar ${DIAGNOSTIC_JAR} ${params[*]}


Once run the output should look something like this:


[root@clm-aus-123 configure_rsso]# ./

05:13:11.164 -

Common validations:

05:13:11.167 - All RSSO agent jar files are in place in '/opt/bmc/digitalworkplace/deploy' and '/opt/bmc/digitalworkplace/pluginsvr'

05:13:11.167 - All RSSO agent config files are in place in '/opt/bmc/digitalworkplace/conf'

05:13:11.167 - Your RSSO agent version is ''.

05:13:11.167 - It is recommended to have RSSO agent of version '19.5'.

05:13:11.167 - Your RSSO server version is '18.05.00'.

05:13:11.167 - It is recommended to have RSSO server and agent of same version.

05:13:11.167 -

Validating '' file in '/opt/bmc/digitalworkplace/conf':

05:13:11.168 - agent-id is dwpid

05:13:11.168 - Agent id should be the same on all DWP-Catalog nodes

05:13:11.168 - sso-external-url is http://<servername>:8080/rsso

05:13:11.168 - HTTPS is recommended for external URL

05:13:11.168 - sso-service-url is http://<servername>:8080/rsso

05:13:11.168 - http://<servername>:8080/rsso is accessible

05:13:11.169 -

Validating 'rsso.cfg' file in '/opt/bmc/digitalworkplace/conf':

05:13:11.169 - rsso.cfg file is configured correctly


Web Server (Jetty)


Jetty logging can be very useful to capture HTTP traffic and is enabled by un-commenting the following section and changing the log level

       <!-- Jetty Logging - uncomment block if jetty logging is required -->

    <appender name="JettyLog"





                <rollingPolicy class="com.bmc.arsys.logging.StoredRollingPolicy">












                        <pattern>%d{EEE MMM dd HH:mm:ss.SSS yyyy} %m%nopex%n</pattern>



        <logger name="org.eclipse.jetty.server.Server" level="all" additivity="false">


If there are connection issues which are observed in a Server Group, the following command will help analyze the traffic (where ens192 is your network adapter)


tcpdump -A -i ens192 port 8008


Connector logging


Connector logging is available from 19.02. To enable, change the log level to "all"



        <logger name="com.bmc.myservice.connect.remedy" level="all" additivity="false">

                <appender-ref ref="connect-remedy" />


        <logger name="com.bmc.myservice.connect.commons-ar" level="all" additivity="false">

                <appender-ref ref="connect-remedy" />




You can also test the connector from the DWP Catalog Admin page.


The following screenshot is indicating that the DWP ITSM Integration patch needs to be updated.




Process logging

From 19.11 we have a new logfile which records process activity. Please refer to the logging table for further details.

Backup the logback_server.xml and add the following to the end of the file.

<!-- Process -->

        <appender name="ProcessLogToForm" class="com.bmc.arsys.logging.SubscribableAppender">


        <appender name="ProcessLog" class="com.bmc.arsys.logging.SiftingTenantAppender">

                <appender class="com.bmc.arsys.logging.SuspendableRollingFileAppender">



                        <!-- <param name="Encoding" value="UTF-8" /> -->



                        <rollingPolicy class="com.bmc.arsys.logging.StoredRollingPolicy">















  <logger name="" level="debug"


                <appender-ref ref="ProcessLog" />

                <appender-ref ref="AlwaysOnLog" />

                <appender-ref ref="ProcessLogToForm" />


        <logger name="" level="debug"


                <appender-ref ref="ProcessLog" />

                <appender-ref ref="AlwaysOnLog" />

                <appender-ref ref="ProcessLogToForm" />




                level="debug" additivity="false">

                <appender-ref ref="ProcessLog" />

                <appender-ref ref="AlwaysOnLog" />

                <appender-ref ref="ProcessLogToForm" />


        <logger name="" level="debug"


                <appender-ref ref="ProcessLog" />

                <appender-ref ref="AlwaysOnLog" />

                <appender-ref ref="ProcessLogToForm" />


        <logger name="" level="debug"


                <appender-ref ref="ProcessLog" />

                <appender-ref ref="AlwaysOnLog" />

                <appender-ref ref="ProcessLogToForm" />


        <logger name="" level="debug"


                <appender-ref ref="ProcessLog" />

                <appender-ref ref="AlwaysOnLog" />

                <appender-ref ref="ProcessLogToForm" />


Then log into the mid-tier (Catalog) as dwpadmin and navigate to Sever Information > Log Files and tick "Process Log".

You need to perform an "dwpcontroller stop/start" for the changes to take effect. Pease remember to turn off this logging if it's no longer requied.

Startup issues


Navigate to the $AR_SYSTEM_HOME/sb/rxscripts/bin directory and uncomment the following line from the script

#export rx_trace="--trace-ascii ./trace.log"

unset rx_trace


Stop/start the Catalog again and check the trace log for errors.


Catalog Bundle log


Almost every troubleshooting scenario will require analysis of the bundle log. Here's an example of what a "healthy" bundle log looks like.


05-05 15:56:26.419 DEBUG com.bmc.myservice.connectors.service.ConnectorSrvImpl testing URI http://localhost:8008/api/sbe/connect/remedy]

05-05 15:56:26.520 DEBUG com.bmc.myservice.connectors.service.ConnectorSrvImpl connector URIs [http://localhost:8008/api/sbe/connect/baopoc]

com.bmc.myservice.connectors.service.ConnectorSrvImpl connector URIs [http://localhost:8008/api/sbe/connect/ad]


As you can see, DEBUG is enabled and it's constantly looping around checking each connector status. The following example demonstrates how you would use the bundle log to troubleshoot a typical problem.


  • In the screenshot below I have submitted a Service Request in DWP and you can see it's gone into a fault status.





The first step is to open a session in the “$AR_SYSTEM_HOME/db” directory and run a "tail" on the log.

  • [root@clm-aus-0123 db]# tail -f bundle.log

This command gives you a live feed of what's happening within the log.


  • In the example below I can see my Service Request is faulting because a required field is empty...


at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source) ~[na:na]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke( ~[na:1.7.0_95]

        at java.lang.reflect.Method.invoke( ~[na:1.7.0_95]

        at com.bmc.myservice.common.util.SrvInterceptor.invoke( ~[com.bmc.myservice.common-1.0.00-SNAPSHOT.jar:1.0.00-SNAPSHOT]

        ... 94 common frames omitted

Caused by: com.bmc.arsys.domain.etc.ARException: ERROR (100): Required Entry ID is empty.; form: rxn:/myit-sb/ServiceRequest

        at com.bmc.arsys.server.domain.validation.EntryValidatorImpl.validateEntryId( ~[na:na]

        at com.bmc.arsys.server.domain.validation.EntryValidatorImpl.validateGetEntry( ~[na:na]

        at com.bmc.arsys.server.domain.validation.handler.EntryValidationHandler.validateGetEntry( ~[na:na]

        at com.bmc.arsys.server.domain.service.impl.EntryServiceImpl.getEntry( ~[na:na]

        at com.bmc.arsys.server.facade.impl.EntryFacadeImpl.getEntry_aroundBody10( ~[na:na]

        at com.bmc.arsys.server.facade.impl.EntryFacadeImpl$ ~[na:na]

  • Here's another example where I submit another Catalog Request from the DWP Client and the bundle log is telling me that I need to define an Work Order Manager on the ITSM side.

Caused by: java.lang.reflect.InvocationTargetException: null

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95]

        at sun.reflect.NativeMethodAccessorImpl.invoke( ~[na:1.7.0_95]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke( ~[na:1.7.0_95]

        at java.lang.reflect.Method.invoke( ~[na:1.7.0_95]

        at ~[na:na]

        ... 80 common frames omitted

Caused by: com.bmc.myservice.connect.exception.ConnectorException: ERROR (1440853): ; No work order manager group could be found. Manually select a group from the menus. If no group with an individual in the functional role of work order manager is defined, notify your System Administrator.

        at ~[na:na]

        at ~[na:na]

        at ~[na:na]


Most errors are now visible by simply opening the Request from the Console:




Installation and Upgrade


The installation script will inform you if there are any problems. Full details are available within the "arsystem_install" log which is located in the /tmp directory. To obtain a live feed on installation progress, just run a "tail" on the log as follows:


[root@clm-aus-0123 tmp]# tail -1000 /tmp/arsystem_install_log.txt


The following extract is an example of a successful installation.


(Dec 15 2016 03:54:59.677 PM +0530),INFO,com.bmc.install.utility.logging.Log,


  BMC Remedy Action Request System 9.5.00 install succeeded.


(Dec 15 2016 03:54:59.697 PM +0530),INFO,com.bmc.install.product.base.installer.InstallCompletionInstallationTask,

  PROGRESS EVENT {Description=[installCompletion.installCompletion.description],Progress=[100],Detail=[installCompletion.installCompletion.complete]}

(Dec 15 2016 03:54:59.917 PM +0530),CONFIG,com.bmc.install.product.base.project.runner.ProjectRunner,

  LOG EVENT {Description=[COMPLETED InstallationTask],Detail=[com.bmc.install.product.base.installer.InstallCompletionInstallationTask]}

If the installer fails and the arsystem log doesn't identify the root cause, run the uninstall script (make sure you also remove any Catalog DB references if present) and run the script again in verbose mode as follows:

bash -x ./

This command will output a detailed log to the terminal. In the event of a failure the output will stop allowing you to scroll up and identify what was happening at the time.

Please back up the Catalog Databases and file-system before upgrading. In the event you encounter a problem you can quickly roll back and send the logs to BMC Technical Support.

Connector Refresh


There may be occurrences where your connectors are no longer visible in the Console. If this is the case try running the connector refresh script as follows:


[root@clm-aus-0123 rxscripts]# source bin/


./bin/ password


Approvals & ITSM Integration


The 'MyITSB- Approval' process is responsible for sending the approval request from the Catalog to the ITSM Server. When a Service Request is raised from the DWP Client, the corresponding workflow on the Catalog machine is triggered which in-turn calls a Catalog 'Request Approval' process which communicates with the AR Server to create the Manager approval.



Once the Request has been approved by a manager, remedy workflow kicks in and attempts to send the data back to the Catalog via the "remoteaction.bat" file. This file along with the contents of the "remoteaction" directory are implemented by the Integration Server Patch.

Make sure the remoteaction script/batch file has the correct Java path


If there's a problem getting the correct approval status reflected in DWP the first thing to do would be to enable filter/API logging on the ITSM Server before reproducing the problem. This way you should be able to see the relevant filters triggering the approval change and sending it back to the Catalog Server.


The next step would be to open the "SB:ServiceRequestStub" form on the ITSM Server and check the “Error Message” box. In the example below it's complaining about a timeout reaching the Catalog.


Make sure the "SB:LocalApprovalConfiguration" & "SB:RemoteApprovalConfiguration" Approval forms on the ITSM AR Servers have the correct information.

There is also a "remoteaction" log file (enabled in the logback_remoteaction file) which will allow you to track the status transmission back to the Catalog.


ITSM Integration


The following illustration depicts the process flow when submitting a request from DWP. As both Approvals and ITSM use a combination of the remoteaction.xml and associated JAR files in combination with remedy workflow, we will use the same troubleshooting methodology as above for Approvals.



The "Status Change" filters have changed names over the releases...


Browser Tools


Once Developer Mode (F12) is enabled check the network tab response and look for any exceptions. The screenshot below illustrates how I've captured the root cause of a problem by simply clicking on the REST call and selecting the response.



Ensure the DWP Catalog is reachable and the fully qualified domain name is being used.


Don't forget, the users must exist in both the user form on the Catalog Server (which is populated via the sync script) and CTM:People form within ITSM


DWP Client Catalog Debug


Open the logback-dwp.xml from the "/opt/apache/tomcat8.x/external-conf" directory and change these two extracts.


  <!--Logging of DWP Catalog -->

  <property name="LOG_DWP_CATALOG" value="true"/>


  <!--Optional DWP Catalog logger-->

  <if condition='${LOG_DWP_CATALOG}'>


      <logger name="" level="debug" >

        <appender-ref ref="DWP_CATALOG_FILE"/>


      <logger name="" level="debug" >

        <appender-ref ref="DWP_CATALOG_FILE"/>


      <logger name="" level="debug" >

        <appender-ref ref="DWP_CATALOG_FILE"/>





After the scan period of 60 seconds a new log file (dwp-catalog.log) will be created in the DWP/Logs directory.