Share:|

Lately, several users have reported issues with us stating ‘Approvals not processed’ or “Approvals stuck in Application Pending form’. There could be several reasons to this problem . Hence, I thought of consolidating investigation path for Approvals Stuck issues in this post.

 

In this post, we will cover-

What is purpose of the Application Pending form? How does it work?  How can you recognize and investigate issues with it? These are a few questions we will cover in this blog post.

 

You can find similar information about the Assignment Engine in the blog post The Pulse: Understanding Assignment Engine.

 

At any point of time, the number of entries in Application Pending form should have 0 or at least a small number when Approvals are processing successfully. However, in some cases we observe records pile up in the Application Pending form. The consequence of that would be incorrect processing of approvals or sometimes approvals stop processing completely.

The Application Pending form is a queue form for several application server processes - which currently include the following:

  • arsvcdsp, apsvcdsp - the Dispatcher, which processes entries where Category = Dispatch
  • brsvcbrie - the SLA Engine, which processes entries where Category = BR-BRIE
  • The Approval Server, which processes entries where Category = Approval
  • The Assignment Engine, which processes entries where Category = AE-Assign

 

*Approval server & Assignment engine are Java Plugins since 8.x.

The design is that a workflow does a Run Process with an Application-Command process name. This creates the entry in the Application Pending form sends a signal to the appropriate server process or plugin above, which processes the records. This ensures requests are processed in sequence, and allows troubleshooting if issues occur.

In this blog, we’ll specifically focus on Approval Server troubleshooting.

 

Sample Problem Statement :

“Customer reported a problem wherein approval signatures were not generated for approvers, hence many change & service requests are stuck in waiting for approval state. When we checked Application Pending form there were thousands of entries; most of them specifically with Category=Approval”

 

Below are the basic steps of our troubleshooting process:

  • Check for collecting records in the Application Pending form with Category = Approval
  • Check when the oldest record was created, and if it is processed in a minute
  • Check ar.cfg/ar.conf file configuration for dispatcher and polling interval
  • If the oldest record is within polling interval, investigate dispatcher problem
  • If not, investigate approval server issue
  • To start investigating the issue, we may need to check ar.cfg entries related to approval to see if approval engine is configured correctly. If it’s a server group environment, we need to identify AR server who actually owns Approval Operation. We can check that by searching in “AR System Server Group Operation Ranking” form.

 

Parameters in ar.cfg/ar.conf file:

  • Approval-Server-Suspended

This parameter should be set to ‘F’ on the Primary server which performs approval operations and should be set to ‘T’ on other servers in server group environment.

 

  • Alternate-Approval-Reg

Approval Server listens for the AR System server’s signal directly (F) or listen for the application dispatcher to signal (T).The default value of this setting is T, which makes sure that the approval server receives the signals sent by the dispatcher. But if it is set to F then approval server will work on polling basis and will poll Application Pending using polling interval set in ar.conf/cfg

 

  • Approval-Polling-Interval

Interval at which the Approval Server polls the AR System server for pending work. This setting is intended as a backup method, not the primary contact method. Under normal circumstances, the AR System server or the Application Dispatcher (depending on the configuration) contacts the Approval Server when work is pending. When this option is specified, the Approval Server polls the AR System server only if it does not receive a signal from the AR System server in the specified time. Specify the interval in seconds. Minimum is 30 seconds. Maximum is 3600 seconds (one hour).

 

We can Check if the dispatcher server process is running, and the process id is not changing. It is arsvcdsp.exe for windows and arservapd for UNIX. To check this you can use Task Manager in windows & ‘ps –ef | grep arservapd’ command for UNIX.  Also check for errors in the armonitor.log file. Here the  about memory is just one example of what may be found in armonitor.log

 

Example:

AR System failed to start a process.

Make sure that your system is not low on resources. (ARERR 33)

No error : "D:\BMCSoftware\BMCARSystem\arsvcdsp.exe" --unicode -d "D:\BMCSoftware\BMCARSystem" -m

 

We can move to next steps to check files specific to approvals-

<ARSystemInsallDir>/Db folder consist of two lck files namely ar.lck.390608 & ar.lck.390609

Make sure these two files are in place. These files are created at AR sever startup and if any of this file is not present in Db directory, approval process won’t work.

 

ar.lck.390608 contains Process ID of plugin server process and ar.lck.390609 contains Process ID of dispatcher process ( arsvcdsp.exe for windows/arservapd for Unix). Please cross check Process ID’s mentioned in these files with actual process ID’s from task manager.

 

To find process ID of plugin server, use following command –

     netstat –nao |findstr <plugin server port no.>

     Suppose 9999 is the port number allocated to plugin server , then use below command –

     netstat –nao |findstr 9999

 

You can see here in with the help of screenshot:

 

Application Pending Form:apllicationPending.jpg

lck file entries:

LCK Entries.jpg

Java Plugin server:

Java Plugin server.jpg

Dispatcher Process:

Dispatcher Process.jpg

 

Example 2:

In one of the customer case we observed a problem wherein above two lck files were not created at AR server startup. After reviewing ar.cfg file , we observed that

Alternate-Approval-Reg  parameter was not present. We added Alternate-Approval-Reg:T to ar.cfg and restarted AR server ; lck files created successfully.

If you passed the above tests and are still observing an issue with approvals, please enable “arjavaplugin.log” in debug mode. To understand how to do it, please click here.

 

let’s take a note on how to check if Approval server is running on (Out-Of-the-Box ) the default plugin server or not.

There are 2 methods from which user can identify if the approval server is running on the default plugin server or not

 

Method 1:

Open armonitor.conf file and search for keyword "arasj" and check the command line.

If line is commented then Approval server is running on default plug-in server.

 

Method 2:

Open AR server installation directory

Then please open -> pluginsvr -> locate "pluginsvr_config.xml" file

Open "pluginsvr_config.xml" and search for word "PREVIEW"

If you find the XML tag and IT IS *NOT* COMMENTED then approval server is configured and running on the default plug-in server.

 

The following list includes how to analyze errors messages, when the Approval server is running on DEFAULT PLUG-IN SERVER

Go to AR Server installation directory -> Arserver -> Db directory

In this directory you will see following files created whenever the approval server is running on the default plug-in server.

 

  • arjavaplugin.log - This is consolidated log
  • arjavaplugin-stdout-<timestamp>.log - This is standard output log file
  • arjavaplugin-stderr-<timestamp>.log - This is standard error log file.

 

If you see reported behavior, then all errors are getting logged in arjavaplugin-stderr-<timestamp>.log including exception.

We recommend reviewing this file to see if there is any exception thrown by Approval server.

If stderr log file is empty then check in consolidated log file i.e. arjavaplugin.log

 

A) If you find the following statement:

2014-05-13 23:45:38,641 FATAL [Main] com.bmc.arsys.pluginsvr.plugins.ARPluginContext (?:?) - <ARSYS.ARDBC.PREVIEW>Exclusive lock could not be acquired for lck file

 

    This error potentially means that ar.lck.390608 file is not in sync with process id of the plug-in server.

    To correct this,

    Please stop AR server and delete ar.lck.390608 file along with ar.lck.390609 file.

    If feasible, then you can take the backup of complete DB directory and then remove all log files from it

   Now start the AR server service.

    These steps will create ar.lck.390608 and ar.lck.390609 files with correct process id which is in sync with the plug-in      server.

 

B) If you find following statement

"failed to create an instance of ARSYS.ARDBC.PREVIEW"

 

     This error potentially means that in ar.conf file, there are some duplicate parameters. (Which are supposed to be unique) for example,      Two entries of "Server-Name" parameter.

     To correct this,

     Please review ar.conf/cfg file and make sure there are NO duplicate entries.

     After correcting ar.conf/cfg file, please restart AR server service or at least plug-in server process.

     In case if you still see the same behavior then again check stderr file.

 

C) If you don't see ar.lck.390608 file in "Db" directory then please check arjavaplugin-stderr-<timestamp> log file.    This file contains information about approval server failure.

     To correct this, please follow the steps from point A.

 

D) If you see following statement,

"Exception in thread "Thread 1" java.lang.OutOfMemoryError: Java heap space"

OR

"java.lang.OutOfMemoryError: GC overhead limit exceeded"

 

     This error potentially means that the heap size is not set correctly.

     To correct this,

     First, please make sure the source of error is "approval" by looking at call stack.

     Secondly, please check system's memory configuration and armonitor.conf file.

     In this file, we need to check,

     If plug-in server is running with proper -Xmx<heap-size> parameter or not.

     If <heap-size> is allocated sufficient or not. (Such as 512Mb then observe by increasing it to 1GB ( or more ).

 

In case if above information would not help, then we recommend to log an issue with BMC support team with providing following information.

  • Complete "Db" directory
  • Use case or approval command in execution at that instance.

 

Now, let’s discuss on how to analyze errors message, when Approval server is running on SEPARATEPLUG-IN SERVER.

Following link from documentation portal space, provide information on how to configure approval server on different plug-in server. We recommend you to check if the provided information is correctly set in your environment or not

Configuring BMC Remedy Approval Server with a separate plug-in server instance

 

There are multiple methods from which user can identify if the approval server is running in Separate plugin server or not

 

Method 1:

Open armonitor.conf file and search for keyword "arasj" and check the command line

If line is *NOT* commented then Approval server is running on Separate plug-in server.

 

Method 2:

Open AR server installation directory

Then please open -> pluginsvr -> locate "pluginsvr_config.xml" file

Open "pluginsvr_config.xml" and search for word "PREVIEW"

If you find the XML tag and IT IS COMMENTED then approval server is configured and running on Separate plug-in server.

 

Method 3:

Open AR server installation directory

Then please open -> pluginsvr -> locate "pluginsvr_config.xml" file

Open "pluginsvr_config.xml" and check <port> tag

e.g. <port>9800</port>

 

This port viz. 9800 should match with port mentioned in ar.conf file against ARSYS.ARDBC.PREVIEW

e.g. for above port entry i.e. 9800 , ar.conf should contain

Server-Plugin-Alias: ARSYS.ARDBC.PREVIEW ARSYS.ARDBC.PREVIEW <server-name>:9800

 

The only difference in this section is the log file names which are created by plug-in server.

Go to AR Server installation directory -> Arserver -> Db directory

In this directory you will see following files created whenever approval server is running on Separate plugin server.

  • approvalplugin-stdout-<timestamp>.log - This is standard output log file
  • approvalplugin.log - This is consolidated log
  • approvalplugin-stderr-<timestamp>.log - This is standard error log file

 

These file names are provided in the ‘log4j_pluginsvr.xml file’ from Approval server /bin directory.

 

In case you see any error messages in these files, then please refer and follow the troubleshooting steps from default plugin serve

Apart from above mentioned troubleshooting steps, you can also try to run approval server on dedicated plugin port. By default, approval server runs on port used by plugin server. To know how to do it, click here.

 

Useful KA:

KA366004-Approval transnational records are getting stored into application pending form, after upgrading AR server to 7.6.04 SP3 application version


KA373461-Approval server operation records are not getting proceed instantly for Asset/Change Application request

 

KA380042-Some of the change requests are getting stuck into approval phases although there is no pending approver is attached to change request

 

KA364898- How to turn off/on arjavaplugin logging?

 

I hope this post is useful for investigating issues with Approval requests collecting in the Application Pending form.

Please rate this blog to let me know if it was useful and add comments with your experiences.

To see more like this, see BMC Remedy Support Blogs