Remedy - AR System Server - How to collect Java heap dumps (and thread dumps) for AR Server troubleshooting - ARS 9.x - INCLUDES VIDEO

Version 86
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    Remedy AR System Server


    Remedy AR System Server


    BMC Remedy AR System Server 9.x


    Support has asked for heap and thread dumps from AR Server (9.x and above) - how can these be collected?


    Please see attached PDF document, showing an example of using JVisualVM (in this case, executed on a Windows machine) to collect heap dumps, thread dumps and application snapshots.

    For non-Windows environments, you can run JVisualVM from a remote Windows machine and point to the Unix system running ARServer

    Perform the following steps to create and prepare files to send to Support:

    1.   Install Java SDK (JDK)  on the system you will perform the steps on.
    NOTE: This system is typically a Windows system other than the AR Server system.

    2.   View the ar.cfg (conf)  or use the Centralized Configuration UI to check the Jmx-port setting. 
          a.  The default is 61500

    3.   View the arserver.config/arserverd.conf file to ensure the following settings are present.  Note that the exact option number is not critical:
          b.  jvm.option.14=-XX:+HeapDumpOnOutOfMemoryError
          c.  jvm.option.15=-XX:HeapDumpPath=./Logs

             - For Windows: the file is called arserver.config and it is located in ....\BMC Software\ARSystem
             - For Unix/Linux: the file is called arserverd.conf and it is located in .../ARSystem/bin

    4.   Ensure that the AR System server's system drive (C:\ on Windows) contains at least enough free diskspace for each heap dump you take.

      Each dump will be roughly the size of the Arserver Java memory used; so if ARServer Java process takes 6GB and you are taking 4 dumps, ensure that you have more than 24 GBs available plus some extra room.
    5.   Launch JVisualVM from the JDK's bin directory.

    6.   Click on  File / Add JMX Connection...

              User-added image

          a.  Connect to the ARServer host using the Jmx port.
          b.  The  Connection string will look like: service:jmx:rmi:///jndi/rmi://<server Name>:<jmx port>/ARServer
      For example: service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer
             c.  Check the box to 'Use security credentials' and provide a Remedy administrator user and password.
          d.  Check the box to no require SSL
          e.  Click OK to connect

    7.   Double-click on the new Connection below the AR Server name. For example, Demo@service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer (pid 8504)

    8.   Click on the Monitor tab

    User-added image

    9.   Click on the Heap Dump button.   This will generate a large .hprof file on the AR Server system.
          a.  You will either get an option to save the dump or it will open it up in a new tab.
               1.  If you get prompted, Provide the full path and filename of the dump file.

             2.   Otherwise, if the heap dump opens in a new tab, look at the navigation pane on the left for this new dump, right-click on it and choose Save-As

    10.  Click OK to continue.

    11. Click on the Threads Tab.

    12.  Click on the Thread Dump button to generate a dump of thread activity.

              User-added image

    13.  Repeat step 12 several times to capture multiple Thread Dumps
      Support needs multiple thread dumps so that we can identify possible  trends in thread utilization
    14.  Right-click on the Remote Connection below the AR Server name. For example, Demo@service:jmx:rmi:///jndi/rmi://RemedyProd:61500/ARServer (pid 8504)
            a.  Select Application Snapshot
            b.  This will generate a "point in time" snapshot of the environment details and thread dumps to a single file
            c.  This does not include the .hprof files that are on the AR Server system

              User-added image
          d.  In some cases, you will need to generate an Application Snapshot from a large timeframe, often 24 hours

               1.   To change interval to 24 hours from the default: Choose “Tools | Options” and enter 1,440 for Charts Cache

      User-added image

              2.    After sufficient data is collected, select the node in the navigation tree, right-click, and choose “Application Snapshot”

    15.  Right click on the new Snapshot items and choose Save As to create a .apps file.
    This will save to the local drive that you are running JVisualVM from.

    16.  Gather, Zip,  and send to Support the following files:
          a.  The .hprof files created in step 9
               Note:  These files will be on the AR System server system

          b.  The .apps file created in step 15
               Note: This files will be on the system from which you are running JVisualVM

    Other methods:
    You can alternatively use the jstack and jmap utilities if there is a JDK installation on the AR System server.  The utilities are located under the JDK bin directory.  In some cases the jvisualvm may not respond and using the command line versions would be the next option. These are command line utilities.  

    Below are examples of the command syntax to use where you redirect the output to a file.
    The pid will be the process ID of the remedy server java process.  Look for the Java with the java heap parameters and also referencing the SNAPSHOT jar file as in the example below.

    In some cases the '-F' parameter will encounter an error when executing the command.  If this occurs just omit the -F.
    The 2nd parameter with jstack below is the lower case letter l.

    jstack -F -l [pid] > stack.log
    Example: jstack -F -l 1234 > stack.log

    jmap -F -dump:format=b,file=[dump_file_name] [pid]
    Example: jmap -F -dump:format=b,file=[c:\temp\heapdump.bin] 1234

    Remedy Server java process:
    /usr/lib/jvm/jre-1.8.0-openjdk- -Xms8589934592 -Xmx8589934592 -cp ./../lib/com.bmc.arsys.boot-9.1.02-SNAPSHOT.jar

    Note:  Even after zipping the files, they may be rather large.
    You may need to connect to,  create a directory under the 'incoming' directory, upload the files and alert Support to the location and filename(s).

    Please follow the steps explained in the following article on how to use BMC's FTP site:  BMC Remedy How to upload files on
    Contact BMC Support if you require any assistance to access/create a FTP directory in the BMC FTP server.

    Related Article(s)
    Remedy - Email Engine - v9.1.x How to capture Thread and Heap dump for Email Engine process (Windows)

    Online Documentation -
    JVM Monitoring



    Article Number:


    Article Type:


      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles