Remedy - Server - How to collect Java heap dumps (and thread dumps) for AR Server troubleshooting - ARS 9.x

Version 2

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    Remedy AR System Server


    AR System


    BMC Remedy AR System Server 9.x


    Support has asked for heap and thread dumps from AR Server (9.x and above) - how can these be collected?


    Please see attached PDF document, showing an example of using JVisualVM (and in this case, executed on a Window machine) to collect heap dumps, thread dumps and application snapshots.

    For non-Windows environments, jstack could be another alternative - or the X display can be directed back to a Windows instance running an X Server.

    Perform the following steps to create and prepare files to send to Support:

    1.   Install Java SDK (JDK)  on the system you will perform the steps on.
    NOTE: This system is typically a Windows system other than the AR Server system.

    2.   View the ar.cfg (conf)  or use the Centralized Configuration UI to check the Jmx-port setting. 
          a.  The default is 61500

    3.   View the arserver.config/arserverd.conf file to ensure the following settings are present.  Note that the exact option number is not critical:
          b.  jvm.option.14=-XX:+HeapDumpOnOutOfMemoryError
          c.  jvm.option.15=-XX:HeapDumpPath=./Logs

             - For Windows: the file is called arserver.config and it is located in ....\BMC Software\ARSystem
             - For Unix/Linux: the file is called arserverd.conf and it is located in .../ARSystem/bin

    4.   Ensure that the AR System server's drive (C:\ on Windows) contains at least enough free diskspace for each heap dump you take.
    Each dump will be roughly the size of the Arserver Java memory used, so if ARServer Java process takes 6GB and you are taking 4 dumps, ensure that you have more than 24 GBs available plus some extra room.

    5.   Launch JVisualVM from the JDK's bin directory.

    6.   Click on  File / Add JMX Connection...

              User-added image

          a.  Connect to the ARServer host using the Jmx port.
          b.  Example Connection string: service:jmx:rmi:///jndi/rmi://<server Name>:61500/ARServer
          c.  Check the box to 'Use security credentials' and provide a Remedy administrator user and password.
          d.  Check the box to no require SSL
          e.  Click OK to connect

    7.   Double-click on the new Connection below the AR Server name. For example, Demo@service:jmx:rmi:///jndi/rmi://<server Name>:61500/ARServer (pid 8504)

    8.   Click on the Monitor tab

    User-added image

    9.   Click on the Heap Dump button.   This will generate a large .hprof file on the AR Server system.
          a.  Provide the full path and filename of the dump file.


    10.  Click OK to continue.

    11. Click on the Threads Tab.

    12.  Click on the Thread Dump button to generate a dump of thread activity.

              User-added image

    13.  Repeat step 12 several times to capture multiple Thread Dumps ( we need multiple thread dumps so that we can identify possible  trends in thread utilization) 

    14.  Right-click on the Remote Connection below the AR Server name. For example, Demo@service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer (pid 8504)
          a.  Select Application Snapshot
          b.  This will generate a "point in time" snapshot of the environment details and thread dumps to a single file
          c.  This does not include the .hprof files that are on the AR Server system

              User-added image

    15.  Right click on the new Snapshot item and choose Save As to create a .apps file.
    This will save to the local drive that you are running JVisualVM from.

    16.  Gather, Zip,  and send to Support the following files:
          a.  The .hprof files created in step 9
               Note:  These files will be on the AR System server system

          b.  The .apps file created in step 15
               Note: This files will be on the system from which you are running JVisualVM

    17.  You can alternatively use the jstack and jmap utilities if there is a JDK installation on the remedy server.  The utilities are located under the JDK bin directory.  In some cases the jvisualvm may not respond and using the command line versions would be the next option. These are command line utilities.  

    Below are examples of the command syntax to use where you redirect the output to a file.
    The pid will be the process ID of the remedy server java process.  Look for the Java with the java heap parameters and also referencing the SNAPSHOT jar file as in the example below.

    In some cases the '-F' parameter will encounter an error when executing the command.  If this occurs just omit the -F.
    The 2nd parameter with jstack below is the lower case letter l.

    jstack -F -l [pid] > stack.log
    Example: jstack -F -l 1234 > stack.log

    jmap -F -dump:format=b,file=[dump_file_name] [pid]
    Example: jmap -F -dump:format=b,file=[c:\temp\heapdump.bin] 1234

    Remedy Server java process:
    /usr/lib/jvm/jre-1.8.0-openjdk- -Xms8589934592 -Xmx8589934592 -cp ./../lib/com.bmc.arsys.boot-9.1.02-SNAPSHOT.jar

    Note:  Even after zipping the files, they may be rather large.
    You may need to connect to,  create a directory under the 'incoming' directory, upload the files and alert Support to the location and filename(s).

    Please follow the steps explained in the following article on how to use BMC's FTP site:  BMC Remedy How to upload files on
    Contact BMC Support if you require any assistance to access/create a FTP directory in the BMC FTP server.

    Related Article(s)
    Remedy - Email Engine - v9.1.x How to capture Thread and Heap dump for Email Engine process (Windows)


    Article Number:


    Article Type:


      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles