Share This:

I thought I would show my work for how I finalized the steps to Monitoring the TrueSight Server Automation Application Server with JMX.  This article will cover setting up a simple JMX monitor (jmxtrans) and monitoring of non-TSSA attributes and then configuring the application server to expose TSSA-specific attributes.  While I'm presenting this in a step by step fashion to show the process I went through, you can always read the other article to see the final configurations for the complete setup.  As always, the standard disclaimer: the below procedure is not supported, your mileage may vary, etc etc.

 

What you will need

Jmxtrans and a system to install it on, ideally separate from the TSSA application server

TSSA application server

JDK that matches the version included in the in TSSA version you are using (to use the standalone jconsole for testing).

 

Setting up a JMX Monitor to monitor non-TSSA attributes

Jmxtrans is an open-source tool to pull data from a JMX source and send the data to a logging, graphing, or monitoring engine.  Setting up an entire monitoring, logging, and/or graphing infrastructure is beyond the scope of the article and I'm going to stick with some simple configurations to prove out the functionality.  Since we are using standard JMX configurations, you should be able to adapt these examples to your particular tool.

 

Initial configuration of the TSSA application server for JMX

 

To enable the standard JMX interface I configure the TSSA application server's JVM by starting it with these parameters: -Dcom.sun.management.jmxremote.port=<some port> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

 

This is accomplished by running blasadmin against the appserver instance I want to setup monitoring for.  If you have multiple instances on each system, you will need to allocate a unique port for each instance.

# blasadmin -s default set app jvmargs "-Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
# blasadmin -s default show app jvmargs
JVMArgs:-Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

 

I need to restart the application server service to pickup the change and I can confirm my instance is now listening on 9099:

# netstat -anp | grep java

tcp        0      0 192.168.8.70:23034      192.168.8.70:4750       ESTABLISHED 20633/java        

tcp6       0      0 :::9700                 :::*                    LISTEN      20591/java        

tcp6       0      0 :::9701                 :::*                    LISTEN      20591/java        

tcp6       0      0 :::9702                 :::*                    LISTEN      20591/java        

tcp6       0      0 :::9099                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9840                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9841                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9842                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9843                 :::*                    LISTEN      20633/java        

PID 20633 is my appserver instance (listening on 984*) and I see the same process is bound to the jmxreport.port (9099) I provided.

 

Connect with jconsole to confirm what attributes you can see with this configuration.  I'm using the jconsole from the JDK I downloaded previously.  You will get a warning about an insecure connection which you can ignore.

 

I can see various attributes now on the MBeans tab:

As expected, I cannot see any of the TSSA-specific attributes:

You may have noticed that you are able to execute some operations under various mbean nodes.  We will talk about securing those later.

 

Configuring jmxtrans for logging

After provisioning a Centos 7 system and installing the RPM for jmxtrans, I set wrapper.java.memory=512m in /etc/jmxtrans/wrapper.conf to ensure I have enough memory.

 

On the jmxtrans host I create a /var/lib/jmxtrans/blapp2002.json file with the following contents to pull some of the attributes I do have access to so I can confirm my monitoring is working.   While jmxtrans has several types of output plugins,  I will use the KeyOutWriter which simply writes values to a text file on disk.

{
  "servers" : [ {
    "port" : "9099",
    "host" : "blapp2002.example.com",
    "queries" : [ {
      "obj" : "java.lang:type=OperatingSystem",
      "attr" : ["OpenFileDescriptorCount","ProcessCpuLoad","SystemCpuLoad","SystemLoadAverage" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  }
 ]
 } ]
}

 

After starting the jmxtrans daemon and waiting for a couple minutes, I see some entries in the /tmp/blapp2002.txt file:

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.OpenFileDescriptorCount,340,1585143272

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.ProcessCpuLoad,0.045454545454545456,1585143272

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemCpuLoad,0.06060606060606061,1585143272

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemLoadAverage,0.46,1585143272

 

That shows the hostname, object name, attribute, value, and time in epoch.  The output of your JMX tool may differ.  Now that I have jmxtrans connected to my TSSA appserver pulling the exposed attributes and values, I can move on to the next step and expose the TSSA-specific attributes and monitor them.

 

 

Exposing TSSA attributes via JMX

 

Configuring the TSSA application server

 

I want to expose the TSSA-specific attributes, in a read-only way.  After reviewing Oracle's JMX documentation about JMX configuration it looks like I can accomplish this with the jmxremote.access.file and jmxremote.password.file directives.  I create a NSH/br/jmxremote.access file that contains the line jmxtrans readonly and a NSH/br/jmxremote.password file with the line jmxtrans bladelogic.  On Linux, these must be owned by bladmin:bladmin and the jmxremote.password must be only owner read and write:

-rwxr-xr-x. 1 bladmin bladmin  134 Feb 14 17:04 jmxremote.access

-rw-------. 1 bladmin bladmin   31 Feb 14 17:04 jmxremote.password

 

I will also need to disable the session support in the TSSA application server for the JMX interface.  Expose this setting as configurable by editing the NSH/br/deployments/<deploymentName>/BlAdmin.xml and removing the internal="yes" from this line:

<option name="DisableSessionSupport" description="Disable Session Support [true,false]" internal="yes" type="boolean"/>

becomes:

<option name="DisableSessionSupport" description="Disable Session Support [true,false]" type="boolean"/>

 

I add the new JMX configuration to my blasadmin settings:

#blasadmin -s default set appserver jvmargs "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Djava.rmi.server.hostname=blapp2002.example.com -Dcom.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access -Dcom.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password"
#blasadmin -s default show appserver jvmargs
JVMArgs:-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Djava.rmi.server.hostname=blapp2002.example.com -Dcom.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access -Dcom.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password

 

Disable session support for the JMX interface:

blasadmin -s default set management disablesessionsupport true

 

After restarting the application server service, I connect with the jconsole, using the username and password I specified in the jmxremote.password file:

 

I can now see the previously protected TSSA attributes:

And trying to run any of the various Operations results in a permission error, which is desired as this user will only be used for read access to monitor various attributes.

 

Configuring jmxtrans with additional attributes

 

I update my jmxtrans json file (/var/lib/jmxtrans/blapp2002.json) with the username and password and a few TSSA-specific attributes:

{
  "servers" : [ {
    "port" : "9099",
    "host" : "blapp2002.example.com",
     "username" : "jmxtrans",
     "password" : "bladelogic",
    "queries" : [ {
      "obj" : "java.lang:type=OperatingSystem",
      "attr" : ["OpenFileDescriptorCount","ProcessCpuLoad","SystemCpuLoad","SystemLoadAverage" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  },
        {
      "obj" : "Bladelogic:type=ApplicationServer,Job Manager=Job Manager,name=Job Manager",
      "attr" : ["NumIdleThreads", "NumberOfRunningJobs" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  },
  {
    "obj" : "Bladelogic:type=ApplicationServer,Connections=Connections,name=Client Connection Service",
    "attr" : [ "ActiveConnections", "IdleConnections", "OpenConnections", "NumIdleClientWorkerThreads" ],
    "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  }
 ]
 } ]
}

 

After a few minutes, I see the additional attributes and values show up in my /tmp/blapp2002.txt file:

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.ActiveConnections,0,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.IdleConnections,2,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.OpenConnections,2,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.NumIdleClientWorkerThreads,10,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_job_JobManagerImpl.JobManager.NumIdleThreads,100,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_job_JobManagerImpl.JobManager.NumberOfRunningJobs,0,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.OpenFileDescriptorCount,330,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.ProcessCpuLoad,9.714087321070273E-5,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemCpuLoad,0.04471864054127741,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemLoadAverage,0.02,1585162606

 

 

 

Tidying Up

To make the configuration a little easier to read and work with, we can move most of the configuration settings into a file and pass the -Dcom.sun.management.conf.file setting in blasadmin and then put the rest of the configuration settings in a jmxremote.config file:

 

blasadmin -s default set app jvmargs "-Dcom.sun.management.config.file=/opt/bmc/bladelogic/NSH/br/jmxremote.config

 

jmxremote.config:

com.sun.management.jmxremote.port=9099

com.sun.management.jmxremote.ssl=false

com.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password

com.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access

com.sun.management.jmxremote.authenticate=true

java.rmi.server.hostname=blapp2002.example.com

 

Summary

The above steps can be used to configure a rudimentary JMX monitor (jmxtrans) with file-based logging, for the purposes of validating that TSSA-specific JMX attributes can be monitored with a standard JMX monitor.  You could use one of the other writers and send the JMX information into a graphing solution like Graphite or StatsD.