The Pulse blog shares experiences from BMC Customer Support on topics that are trending in application support. Our goal is to share insights on how to understand features, investigate when there are issues, and get the most out of the solution. Last month’s blog discussed The Request Entry Console. In this blog post, we will explore the Command Automation Interface (CAI) in Service Request Management.
BMC Service Request Management (SRM) creates fulfillment tickets - Incidents, Work Orders and Change Requests. It is integrated with the Incident Management, Change Management and Work Order (WO) systems by way of the CAI and these interface forms.
Sometimes, we face the issues where a backend fulfillment application ticket is not created when the service request is submitted. Below are troubleshooting steps and information useful for all SRM Application administrators facing similar problems.
Troubleshooting Steps -
Below are steps I use to investigate issues. When we see that No Incident/Change/WO is generated, check the Service Request and click on Request Details. Go to Process View tab in Request Details and you will see something like this
In the above figure - ITCON is the Application Object Template (AOT) which was designed when defining the Process of the Service Request Definition (SRD). The incident which was generated has ID - INC00000000105.
In this case the incident has been created for the Service Request. Whenever we don’t find the Incident ID, look for an Event Error. In most of the cases the Event error will be present and it will give us a clue on what exactly is happening and what the reason the tickets are not being generated is.
Troubleshooting Steps for Request coordinator –
Ideally the Service Request status values follow –the Service request lifecycle
The request coordinator can investigate issues by tracking the service request through this process:
- From the Service Request Coordinator Console, click SRs with Errors in the left navigation panel. Alternatively, you can search for your request.
- Double-click the request or click Request Details.
- In the Request Details dialog box, click Process View to see the process related to the request.
This view enables you to see the underlying objects used when creating work orders, incidents, change requests, and so on. Note: If the Event Error button is enabled, it indicates that there is an error in the request that needs to be resolved.
- From the Process form, click Event Error to troubleshoot the request.
- In the Fulfillment Application Instance Detail area, verify that the correct applications are instantiated.
- If the event is successful, the Application Instance Details area shows the application that created the event, its request ID, and its current status.
- If there is an error on the application instance, the event error details appear in the Event Error Details area of the Application Instances dialog box.
- If an event contains an error (shown in Event Error Details area), click View Events.
The Event History form appears, showing the event history of the instance. If the event ran without errors, the entry is automatically deleted. As a result, you cannot view successful events in the Event History form.
- Debug the error with the fulfillment applications.
Typically, you see problems with fulfillment applications if the AOTs, PDTs, or SRDs were not defined correctly.
Some of the errors are listed in this table.
- Click Retry to re-start the application instance command.
- Close the Event History dialog box.
When you have fixed the fulfillment application error, the application generates a request ID and automatically moves the application request into the In Progress state.
- Close the View History dialog box.
The Request Details dialog box shows the fulfillment ID of the fulfillment application.
- Close the Request Details dialog box.
The fulfillment application ID is generated correctly and you can work on the request.
- After the problem is fixed, make sure that the fulfillment worker working on the request completes it in a timely manner.
If necessary, you can track the request to make sure it is approved by the appropriate person.
- Track the request until its status is Completed. Some of the troubleshooting tips are mentioned in the Documentation.
Understanding how CAI works with fulfillment applications -
The CAI configuration settings can be accessed from the console using:
Application Admin Console > Custom Config > Foundation > Command Automation Interface. There are set of defined commands like example -
SRM_OUT_PUSH_SR_INFO / SRM_IN_APP_REQUEST_RESOLVED / SRM_OUT_CREATE_APP_REQUEST
There are more commands than those mentioned above, but we will use this one as an example to examine how it works
The Command tells CAI Plugin to act and hence some Commands are defined Out of the box. Workflow's job is just to pass the command and let CAI plugin pass the data between SRM and Incident/Change/Work Order. When we define a command then we also have to define parameter values passed when that particular command is passed to CAI Plugin. There will be a source form and there will be a destination form and between them there will be commands executing for the Data transfer. The data can be anything like Status of the Service Request or Work info added by End user in the SR which needs to be passed into Incident.
During the run time the flow can be considered as follows -
SRM-> SRM:APPInstanceBridge -> CAI:Events (Read the above links for the form) -> Backend Applications (Incident/Change/WO).
CAI:Events form -
- The interface forms for the CAI not only support communication from BMC SRM to the back-office fulfillment applications but also from the back-office fulfillment applications to BMC SRM.
To learn more about CAI, the following links will be helpful –
CAI Plugin listens to CAI Events at regular intervals. When a new record reaches CAI:Events form it will have the Command Name and based on this command name the plugin will act accordingly.
The plugin is executed when Insert operation is done in the CAI:Events form. It will poll this form and as soon as an event arises to this form. There is some workflow which calls the CAI plugin and then CAI plugin processes this event.
It allows you to retry the event after the correction of data. So these are some of the reasons why CAI plugin is used.
Troubleshooting Issues when CAI Events do not process
We can use the Event Entry ID in Request Details of SR to search on CAI:Events form. We'll get the same result but, in order to move our "In Process" request out of there, first we'll have to fix the error shown on our event and then, from within Service Requests form, click on Retry button. If everything went fine, our "In Process" request will generate a correct Incident.
There may be a case when there are many entries stuck in CAI:Events form.
In this case, take back-up of all the records from CAI:Events form and then delete all the records. Restart the AR-Server with plugin logging on to see if any unusual errors pop up. Once it is back up and running the CAI should pick up those In Progress events. If CAI does not work then you should see all the activities in plugin logs.
If the CAI:Events record is still "In Progress" or "Running" then the fulfillment record will not be created at all until the CAI processes and deletes the record.
Check the "Status" of the events and the "Return Code" from the call.
Once an event has been processed, it will register as "Complete" and an Escalation deletes these records ON regular intervals. Each event has a number of retries (MaxRetries) and a retry counter (RetryCounter) to indicate how many times the system has attempted to re-try the execution of escalation (Escalation - every hour). The Status of the attempts is shown by the "CAIAction" field > "RETRY".
If the event re-send is unsuccessful, and the max retries is achieved, the event will be in an "Error" state as indicated by the "Return Code".
For Service requests which are stuck in CAI:Events in Running status, you can try setting Return Code = "Start" and ActionKeyword = "RETRY"
After saving the Event, it should launch again, creating the associated backend application ticket.
Locating single Service Request which is possibly stuck in the CAI:Events form -
Go to SRM:AppInstanceBridge form and search with the Request ID you want to check if it is stuck in CAI:Events form.
The field you have to use for the search is – SR Req Number.
Once the result is returned, please copy the zTmpEventGUID field data in the same form (shown above).
Open the CAI:Events form in search mode and paste the contents you’ve copied earlier in the Event GUID field of CAI:Events form and search for the record –
Please note that the results will be returned only if the request is stuck in CAI:Events form due to some error or any other reasons discussed above.
The entry will not be returned if the request has been processed successfully.
Note: If you are passing the values from a text question from BMC Service Request Management to a back-end application (for example, Incident Management), make sure you set the proper field length values. Otherwise, you might see errors in the CAI events trying to create records in the back-end applications, because data from one of the fields in the service request is too long (for example, trying to pass 128 characters to a field that only accepts 100 characters).
Note: Issues with the CAI are usually related to plug-in issues that need to be debugged through the logs to see what is causing the issue e.g. Incorrect Configuration Values, Timeouts, etc. Helpful logs in this case are - CAI plugin log / arjavaplugin.log / Filter, API and SQL logs.
To configure the CAI for BMC Service Request Management
- From the IT Home page, open the Application Administration console.
- Click the Custom Configuration tab.
- Choose Foundation > Advanced Options > Command Automation Interface - Application Registry.
- From the CAI Application Registry form, search for the entries that were created for BMC Change Management or BMC Incident Management.
- On the Connection tab, update the connection information to indicate the BMC Remedy AR System server and port number of the BMC Change Management or the BMC Incident Management computer.
- In the Server field, enter the server name where the application you are registering is installed.
- If you are using a load balancer environment, enter the name of the load balancer in the Server field. The server name (fully-qualified or not) must be the same on the BMC Remedy Mid-Tier, in the load balancer, and in the CAI:Registry form.
- In a server group environment, set the Server field value in the CAI Application Registry form to the common server name alias that you defined for the server group in the Server Name Alias field of the BMC Remedy AR System Administration: Server Information form. Make sure that you also enter the unique server name in the AR System Server Group Operation Ranking form to define which server takes over if the primary server fails.
- Do not enter a login name or password.
Leave these fields blank except for remote installations of BMC Change Management or BMC Incident Management.
- Update other information on the CAI Application Registry form, as needed.
- Click Save.
Copying the CAI plug-in to remote servers
Remember when making changes to CAI plugin configuration, to make the change to all servers in the server group if applicable. –Click Here for more details on this.
To configure the CAI:PluginRegistry form to define and configure thread pools for specific commands:-
Thread pools for different commands are configured in the CAI:Commands form. Configuring thread pools defines dedicated CAI threads for use by the specific command. This ensures that requests from a command use only the dedicated threads, and performance of other command requests to the CAI is not affected.
To configure the CAI:PluginRegistry form -
- In a browser, open the CAI:PluginRegistry form by opening the Application Administration Console, selecting the Custom Configuration tab, and selecting Foundation > Advanced Options > Command Automation Interface - PlugIn Registry.
- In the Private Queue # field, type the private queue number that you generated in generating a private server queue.
- In the CAI Pool Configuration table, select any CAI pool and click View to edit the number of threads for the selected pool. To add a new pool, click Add, fill in the details, and click Save. This updates the total number of threads automatically. The pool that you created here will be used to configure the outbound messages and commands for a specific pool.
- Note - On the Commands form, you can configure one of the pool numbers that you defined in step 3. If the pool number is not defined, the default pool (#0) is used.
- Ensure that the maximum number of threads that you specified in Generating a private server queue is less than the computed threads in step 3.
- In the Log Level drop-down list, select the desired level for the CAI plug-in log entries. The WARN level is the recommended default value.
- Click Close to save the entry, and close the CAI:PluginRegistry form.
- Restart BMC Remedy AR System so that CAI picks up the new private queue information.
Finally, there are a few known issues with the CAI which have workarounds or are fixed in later service packs. Please see following knowledge articles for more information:
KA287697 - CAI:Events and CAI:Event Params forms fill up with entries which crashes the arplugin server
KA303117 - TMS_OUT_GET_DATA in CAI:Event form causes problems with Service Request creation.
KA382534 - Is there way to configure a notification for the CAI Events form when a Service Request is submitted with the Return Code of 'Error' in CAI:Events form?
KA341717 - Troubleshooting Application Plugin Problems in ITSM 7
KA325419 - How CAI plugin works in server group?
In case of defects, it is always advisable to upgrade to the latest service packs and versions that address them.
I hope this article is useful to understand how the CAI works, the role it performs in creating fulfillment tickets from requests in BMC Service Request Management, and how to investigate if issues occur. Please share your feedback, experiences, and thoughts by adding comments below.
Also note that you can use Ideas in Communities to submit and vote on features that you would like to see in the product.