Skip navigation
1 2 3 Previous Next

TrueSight Server Automation

150 posts
Share This:

TrueSight Server Automation (TSSA) allows servers to be "enrolled" more than once. The only requirements are that the Server name be unique. Servers enrolled by more than one name can cause issues when running jobs, especially when the same server is included multiple times in the same job run due to multiple enrollments in TSSA.

 

Common names servers are often enrolled as include;

  • FQDN - Typically this is the name most sites use normally
  • Short Host Name - With no domain name
  • IP Address
  • DNS Alias
  • Secondary End Point Names - DB network name, Storage network name, etc

 

If you search here on BMC Communities, you will find several postings on this issue with no real solution. The problem being there is no accurate single server "ID" available to correlate enrolled servers by. We had some success here at our site comparing ??TARGET.NAME?? to ??TARGET.FQ_HOST??. However not 100% of our servers use the same DNS name as used in FQ_HOST. I suspect that is also true for most customer sites as well. So a different method was needed.

 

The solution we arrived at was to have each server create a empty file called "TSSA-??TARGET.NAME??" in a existing location on each server endpoint, /tmp on Linux/Unix and C:\temp on Windows. We choose ??TARGET.NAME?? as it will be a unique name in TSSA. We can then create a  normal TSSA compliance policy to verify if the count of these TSSA-* files is exactly 1 on each server end point. If the count is greater than 1, then we have detected a server that is enrolled in TSSA more than once. And we can easily identify the enrollment names by looking at the TSSA-* files on the server.

 

Here's how it looks when you review a compliance job run;

SS-TSSA-Enrollment-Results.png

 

It appears we have 6 servers in the group, however 1 server has been enrolled 3 times. Once by IP address, once by short host name, and once using FQDN. Our compliance run has detected them as non-compliant and selecting the "count File:/tmp/TSSA-** =1 where" line in red for one of the non-compliant servers shows the "Left Value" count to be 3. Confirming that we detected this server has been enrolled 3 times in TSSA.

 

The compliance rule to do this is really simple;

SS-TSSA-Enrollment-Rule.png

 

For those familiar with TSSA compliance rules, you may notice there is both a check for count = 1 and also a foreach loop to detect the compliant status. Using just the count would be sufficient to determine compliance. The foreach rule was added so you can "see" the names of the TSSA-* files in the job run results view for a server that is non-compliant. Simply so you don't actually have to go look for the TSSA-* files, they will be listed in the compliance run results view for the any non-compliant components.

 

The Component Template we used as a proof of concept is available as a ZipKit here: Blade ZipKit - TSSA Enrollment

 

This method of detecting "Multiple Server Enrollment" has proven to be very effective in helping us here at Customer Zero clean up our multiple enrolled servers in TSSA.

Share This:

Please note the following Support Flash regarding the upcoming (May 25, 2020) expiration of the TSSA Live Reporting license file shipped with all versions of TSSA up to and including TSSA 20.02

 

https://docs.bmc.com/docs/tssa2002/notification-of-a-critical-action-required-by-users-of-truesight-server-automation-live-reporting-910749158.html

 

If the license file is not applied, the following error will be encountered when attempting to launch TSSA Live Reporting after May 25, 2020:

 

The software license has been breached.

Please go to the License Management page for more information.

 

Please see Knowledge Article 000170998 for full steps on how to obtain the updated license file from BMC Customer Support.

Share This:

TrueSight Server Automation (TSSA) Windows Patch Analysis results, and the success of subsequent Patch Remediations, can be affected by the target server's "Pending Reboot" status.

 

If the server is in the Pending Reboot state this can mean a patch is only partially installed when the Patch Analysis job is run in TSSA.  This can result in misleading Patch Analysis results and also cause Patch Remediation/Deployment jobs to fail.

 

If a server was in the Pending Reboot state when the Patch Analysis job was run, a warning will be reported in the job run results.

Expanding the job run results under the Server View, Failed and Successful targets are displayed as follows:

 

 

To see which Server(s) generated the warning, right-click on the run to display the job run log.  Here each target is displayed with their run status. The Server(s) which completed with a warning display the yellow triangle icon.  In the run messages for the target the following warning message will be present:

 

"Reboot is pending on this machine, analysis results may be in-correct."

 

See KA 000145107. for additional details on this scenario.

 

In order to avoid this scenario and the Patch Analysis job warnings the target servers can be rebooted prior to executing the analysis job.  A TSSA Compliance Job could be executed against the targets to verify none are in the Pending Reboot state.  The Windows Patch Readiness ZipKit Component Template is available for TSSA versions 8.9 and above in the BMC Communities here.  The Template contains rules to check several items, including the Pending Reboot status.


 

 

A server in the Pending Reboot state can affect the patch remediation process as well.  A patch may have been applied to a target server earlier, but that specific patch may require a reboot to be performed before it is considered fully installed. With the server in the Pending Reboot state, the patch may only be partially installed.  Attempting to apply the same patch via TSSA, because it was earlier flagged as Missing during Patch Analysis, could result in the deployment failing with exit code 2359302.  Additional detail of this scenario is covered in KA 000086137.

 

 

There are currently two enhancements under consideration to help avoid this scenario as well.

The first enhancement (DRLPJ-131) is to include a new node within the Patch Analysis Job result’s Server View to make it easier to notice servers which are in the Pending Reboot state.  A third node, along with Failed and Successful, would include Servers which were successful and pending a reboot.

The second enhancement (DRLPJ-142) is to add an option to Patch Analysis Jobs to reboot a server prior to analysis, if found to be in the pending reboot state.

These enhancements are being considered for a future release of TSSA.

Share This:

“Sean, what the [heck] is this thing and how does it relate to these other, similar-sounding products”

 

For the more in-depth article, check out John's extensive and well-researched post here: Helix Support: Planning Your Deployment of TrueSight Smart Reporting for Server Automation (TSSR-SA) 19.2.02

 

Anthony’s BDSSA EOL announcement

 

 

What is TrueSight Smart Reporting (TSSR)?

 

  • TrueSight Smart Reporting for Server Automation (TSSR-SA solution) is the replacement for BMC Decision Support for Server Automation (BDSSA).  It’s a solution made up of two parts: the Data Warehouse component, which gets data from the core application database to a data warehouse database, and the Smart Reporting Platform, which includes the reporting engine, and the out of the box reports content. 
  • The Smart Reporting Platform is shared across solutions, including Helix ITSM, TSOM, TSCO, and TSNA, and will be able to be used and reused across multiple data sources (server, network, capacity).

 

Which product components do I need to install TSSR?

 

  • The latest version of the TSSR solution is 20.02 (As of April 2020).  It’s made up of two components:
    • TSSA-DW (Data Warehouse) 8.9.04.04 and
    • TSSR-SA (Reporting Platform) 19.2.02.
  • It is compatible with all 8.9 versions of Server Automation, including BMC Server Automation 8.9.0, 8.9.1, and TrueSight Server Automation 8.9.2, 8.9.3, and service packs.

 

Can I use TSSR without removing BDSSA?  I want to migrate my reports slowly, while continuing to use BDSSA.

 

 

How do I get a TSSR license?

 

  • If you already have/had a license for BDSSA, you should contact your Sales Representative who can help you get a permanent license for TSSR, and a trial license if necessary.  If you’ve never had a license for BDSSA, you may need to buy a license: contact your Sales Representative.

 

What’s Yellowfin?

 

  • Yellowfin is a reporting product that BMC is using for both Smart Reporting and Live Reporting.  They are installed separately and using different instructions.  Start here (TBD) for Live Reporting, and here (TBD) for Smart Reporting.

 

What’s the difference between Smart Reporting and Live Reporting?

 

  • Both use the Yellowfin engine, but Smart Reporting runs reports against the data warehouse database, while Live Reporting reports run against the core application database.
  • There are many out of the box reports in Smart Reporting, covering the main use cases of Server Automation, including Compliance, Patching, Job Utilization, RBAC, etc.   Smart Reporting reports can be customized, and “broadcast” via email or FTP.  You can also build custom reports in Smart Reporting. 
  • Live Reporting has near real-time reports for key use cases, including: Patching, Compliance.
  • To prevent performance impacts to the core application, Live Reporting doesn’t support customization of reports.  This doesn't mean you -can't- customize reports, but you should expect them to get wiped out on upgrade.  This is by design.

 

FeatureLive ReportingSmart Reporting
Use CasesPatching, CompliancePatching, Compliance, Job Utilization, RBAC, others
Real Time Reports?YesNo: typically as soon as ETL completes
Customizable?Not supportedYes
Database UsedCore Application DB ("bladelogic")Data Warehouse (TSSR_DW or BSARA_DW etc.)
Impacts Core App Performance?YesNo, run as many as you want!
Share This:

I thought I would show my work for how I finalized the steps to Monitoring the TrueSight Server Automation Application Server with JMX.  This article will cover setting up a simple JMX monitor (jmxtrans) and monitoring of non-TSSA attributes and then configuring the application server to expose TSSA-specific attributes.  While I'm presenting this in a step by step fashion to show the process I went through, you can always read the other article to see the final configurations for the complete setup.  As always, the standard disclaimer: the below procedure is not supported, your mileage may vary, etc etc.

 

What you will need

Jmxtrans and a system to install it on, ideally separate from the TSSA application server

TSSA application server

JDK that matches the version included in the in TSSA version you are using (to use the standalone jconsole for testing).

 

Setting up a JMX Monitor to monitor non-TSSA attributes

Jmxtrans is an open-source tool to pull data from a JMX source and send the data to a logging, graphing, or monitoring engine.  Setting up an entire monitoring, logging, and/or graphing infrastructure is beyond the scope of the article and I'm going to stick with some simple configurations to prove out the functionality.  Since we are using standard JMX configurations, you should be able to adapt these examples to your particular tool.

 

Initial configuration of the TSSA application server for JMX

 

To enable the standard JMX interface I configure the TSSA application server's JVM by starting it with these parameters: -Dcom.sun.management.jmxremote.port=<some port> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

 

This is accomplished by running blasadmin against the appserver instance I want to setup monitoring for.  If you have multiple instances on each system, you will need to allocate a unique port for each instance.

# blasadmin -s default set app jvmargs "-Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
# blasadmin -s default show app jvmargs
JVMArgs:-Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

 

I need to restart the application server service to pickup the change and I can confirm my instance is now listening on 9099:

# netstat -anp | grep java

tcp        0      0 192.168.8.70:23034      192.168.8.70:4750       ESTABLISHED 20633/java        

tcp6       0      0 :::9700                 :::*                    LISTEN      20591/java        

tcp6       0      0 :::9701                 :::*                    LISTEN      20591/java        

tcp6       0      0 :::9702                 :::*                    LISTEN      20591/java        

tcp6       0      0 :::9099                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9840                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9841                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9842                 :::*                    LISTEN      20633/java        

tcp6       0      0 :::9843                 :::*                    LISTEN      20633/java        

PID 20633 is my appserver instance (listening on 984*) and I see the same process is bound to the jmxreport.port (9099) I provided.

 

Connect with jconsole to confirm what attributes you can see with this configuration.  I'm using the jconsole from the JDK I downloaded previously.  You will get a warning about an insecure connection which you can ignore.

 

I can see various attributes now on the MBeans tab:

As expected, I cannot see any of the TSSA-specific attributes:

You may have noticed that you are able to execute some operations under various mbean nodes.  We will talk about securing those later.

 

Configuring jmxtrans for logging

After provisioning a Centos 7 system and installing the RPM for jmxtrans, I set wrapper.java.memory=512m in /etc/jmxtrans/wrapper.conf to ensure I have enough memory.

 

On the jmxtrans host I create a /var/lib/jmxtrans/blapp2002.json file with the following contents to pull some of the attributes I do have access to so I can confirm my monitoring is working.   While jmxtrans has several types of output plugins,  I will use the KeyOutWriter which simply writes values to a text file on disk.

{
  "servers" : [ {
    "port" : "9099",
    "host" : "blapp2002.example.com",
    "queries" : [ {
      "obj" : "java.lang:type=OperatingSystem",
      "attr" : ["OpenFileDescriptorCount","ProcessCpuLoad","SystemCpuLoad","SystemLoadAverage" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  }
 ]
 } ]
}

 

After starting the jmxtrans daemon and waiting for a couple minutes, I see some entries in the /tmp/blapp2002.txt file:

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.OpenFileDescriptorCount,340,1585143272

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.ProcessCpuLoad,0.045454545454545456,1585143272

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemCpuLoad,0.06060606060606061,1585143272

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemLoadAverage,0.46,1585143272

 

That shows the hostname, object name, attribute, value, and time in epoch.  The output of your JMX tool may differ.  Now that I have jmxtrans connected to my TSSA appserver pulling the exposed attributes and values, I can move on to the next step and expose the TSSA-specific attributes and monitor them.

 

 

Exposing TSSA attributes via JMX

 

Configuring the TSSA application server

 

I want to expose the TSSA-specific attributes, in a read-only way.  After reviewing Oracle's JMX documentation about JMX configuration it looks like I can accomplish this with the jmxremote.access.file and jmxremote.password.file directives.  I create a NSH/br/jmxremote.access file that contains the line jmxtrans readonly and a NSH/br/jmxremote.password file with the line jmxtrans bladelogic.  On Linux, these must be owned by bladmin:bladmin and the jmxremote.password must be only owner read and write:

-rwxr-xr-x. 1 bladmin bladmin  134 Feb 14 17:04 jmxremote.access

-rw-------. 1 bladmin bladmin   31 Feb 14 17:04 jmxremote.password

 

I will also need to disable the session support in the TSSA application server for the JMX interface.  Expose this setting as configurable by editing the NSH/br/deployments/<deploymentName>/BlAdmin.xml and removing the internal="yes" from this line:

<option name="DisableSessionSupport" description="Disable Session Support [true,false]" internal="yes" type="boolean"/>

becomes:

<option name="DisableSessionSupport" description="Disable Session Support [true,false]" type="boolean"/>

 

I add the new JMX configuration to my blasadmin settings:

#blasadmin -s default set appserver jvmargs "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Djava.rmi.server.hostname=blapp2002.example.com -Dcom.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access -Dcom.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password"
#blasadmin -s default show appserver jvmargs
JVMArgs:-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Djava.rmi.server.hostname=blapp2002.example.com -Dcom.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access -Dcom.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password

 

Disable session support for the JMX interface:

blasadmin -s default set management disablesessionsupport true

 

After restarting the application server service, I connect with the jconsole, using the username and password I specified in the jmxremote.password file:

 

I can now see the previously protected TSSA attributes:

And trying to run any of the various Operations results in a permission error, which is desired as this user will only be used for read access to monitor various attributes.

 

Configuring jmxtrans with additional attributes

 

I update my jmxtrans json file (/var/lib/jmxtrans/blapp2002.json) with the username and password and a few TSSA-specific attributes:

{
  "servers" : [ {
    "port" : "9099",
    "host" : "blapp2002.example.com",
     "username" : "jmxtrans",
     "password" : "bladelogic",
    "queries" : [ {
      "obj" : "java.lang:type=OperatingSystem",
      "attr" : ["OpenFileDescriptorCount","ProcessCpuLoad","SystemCpuLoad","SystemLoadAverage" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  },
        {
      "obj" : "Bladelogic:type=ApplicationServer,Job Manager=Job Manager,name=Job Manager",
      "attr" : ["NumIdleThreads", "NumberOfRunningJobs" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  },
  {
    "obj" : "Bladelogic:type=ApplicationServer,Connections=Connections,name=Client Connection Service",
    "attr" : [ "ActiveConnections", "IdleConnections", "OpenConnections", "NumIdleClientWorkerThreads" ],
    "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "outputFile" : "/tmp/blapp2002.txt",
        "maxLogFileSize" : "10MB",
        "maxLogBackupFiles" : 200,
        "delimiter" : ",",
        "debug" : true,
        "typeNames" : ["name"]
      } ]
  }
 ]
 } ]
}

 

After a few minutes, I see the additional attributes and values show up in my /tmp/blapp2002.txt file:

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.ActiveConnections,0,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.IdleConnections,2,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.OpenConnections,2,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_client_ClientConnectionManager.ClientConnectionService.NumIdleClientWorkerThreads,10,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_job_JobManagerImpl.JobManager.NumIdleThreads,100,1585162606

blapp2002_example_com_9099.com_bladelogic_om_infra_app_service_job_JobManagerImpl.JobManager.NumberOfRunningJobs,0,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.OpenFileDescriptorCount,330,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.ProcessCpuLoad,9.714087321070273E-5,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemCpuLoad,0.04471864054127741,1585162606

blapp2002_example_com_9099.sun_management_OperatingSystemImpl.SystemLoadAverage,0.02,1585162606

 

 

 

Tidying Up

To make the configuration a little easier to read and work with, we can move most of the configuration settings into a file and pass the -Dcom.sun.management.conf.file setting in blasadmin and then put the rest of the configuration settings in a jmxremote.config file:

 

blasadmin -s default set app jvmargs "-Dcom.sun.management.config.file=/opt/bmc/bladelogic/NSH/br/jmxremote.config

 

jmxremote.config:

com.sun.management.jmxremote.port=9099

com.sun.management.jmxremote.ssl=false

com.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password

com.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access

com.sun.management.jmxremote.authenticate=true

java.rmi.server.hostname=blapp2002.example.com

 

Summary

The above steps can be used to configure a rudimentary JMX monitor (jmxtrans) with file-based logging, for the purposes of validating that TSSA-specific JMX attributes can be monitored with a standard JMX monitor.  You could use one of the other writers and send the JMX information into a graphing solution like Graphite or StatsD.

Share This:

The TSSA application server exposes various attributes about itself and its operations via a customized JMX interface which is accessible via the bljconsole and jmxcli. Both of these interfaces require authentication via BLSSO before allowing retrieval of the various TSSA specific attributes and values.  This poses a challenge when trying to use JMX-based monitors as they cannot perform BLSSO authentication and therefore cannot retrieve the TSSA-specific attributes.  Fortunately we can configure the TSSA application server to provide JMX access to these attributes without needing BLSSO authentication.  As always, the standard disclaimer: the below procedure is not supported, your mileage may vary, etc etc.

 

Configuration changes to monitor TSSA-specific attributes via a standard JMX interface

Edit the NSH/br/deployments/<deploymentName>/BlAdmin.xml and remove the internal="yes" from this line:

<option name="DisableSessionSupport" description="Disable Session Support [true,false]" internal="yes" type="boolean"/>  

becomes:

<option name="DisableSessionSupport" description="Disable Session Support [true,false]" type="boolean"/>  

 

Create a NSH/br/jmxremote.password file with the following contents (use whatever username and password you like):

jmxuser password

Create a NSH/br/jmxremote.access file with the following contents (you must use the same username as in the password file):

jmxuser readonly

Create a NSH/br/jmxremote.config file with the following contents:

com.sun.management.jmxremote.port=<port>

com.sun.management.jmxremote.ssl=false

com.sun.management.jmxremote.password.file=/opt/bmc/bladelogic/NSH/br/jmxremote.password

com.sun.management.jmxremote.access.file=/opt/bmc/bladelogic/NSH/br/jmxremote.access

com.sun.management.jmxremote.authenticate=true

java.rmi.server.hostname=<appserver hostname>

replace <port> with a distinct port for this appserver instance, and <appserver hostname> with the appserver's fqdn.

 

Ensure the file permissions on these files match the below:

-rw-r--r--. bladmin bladmin  jmxremote.access

-rw-r--r--. bladmin bladmin  jmxremote.config

-rw-------. bladmin bladmin  jmxremote.password

 

Run the below blasadmin commands:

blasadmin -s <instance name> set app JVMargs "-Dcom.sun.management.config.file=/opt/bmc/bladelogic/NSH/br/jmxremote.config

blasadmin -s <instance name> set management disablesessionsupport true

If you have multiple instances of the appplication server on the same host, you should create multiple jmxremote.config files and specific a different port for each instance in each file, and then specify the per-instance file in each instance's JVMArgs value.

 

After making all of the above changes, restart the application server service.

 

Attributes to Monitor with JMX

There are several attributes available for monitoring in the JMX interface and below is a list of attributes that should provide a good view into what is happening in your TSSA application server:

Bladelogic:type=ApplicationServer,name=Application Server:

FreeJvmMemory, MaximumJvmMemory, ResidentSetSize, TotalJvmMemory, UsedJvmMemory

 

Bladelogic:type=ApplicationServer,BlExec Service=BlExec Service,name=BlExec Service

NumPollingRequests, NumProcessingRequests, NumWaitingRequests

 

Bladelogic:type=ApplicationServer,Connections=Connections,name=Authentication Service

PendingAuthenticationRequests

 

Bladelogic:type=ApplicationServer,Connections=Connections,name=Client Connection Service

ActiveConnections, IdleConnections, OpenConnections

 

Bladelogic:type=ApplicationServer,Connections=Connections,name=Nsh Proxy Service

ActiveNshProxies, IdleNshProxies, OpenNshProxies

 

Bladelogic:type=ApplicationServer,Connections=Connections,name=SSL Connection Service

ActiveConnections, IdleConnections, OpenConnections

 

Bladelogic:type=ApplicationServer,Database Service=Database Service,Client-Connection-Pool=Client-Connection-Pool,name=Client-Connection-Pool

MinConn, NumAvailableConnections, NumConnections

 

Bladelogic:type=ApplicationServer,Database Service=Database Service,General-Connection-Pool=General-Connection-Pool,name=General-Connection-Pool

MinConn, NumAvailableConnections, NumConnections

 

Bladelogic:type=ApplicationServer,Database Service=Database Service,Job-Connection-Pool=Job-Connection-Pool,name=Job-Connection-Pool

MinConn, NumAvailableConnections, NumConnections

 

Bladelogic:type=ApplicationServer,Job Manager=Job Manager,name=Job Manager

NumberOfRunningJobs, NumIdleThreads

 

Summary

The above steps can be used to configure a TSSA application server for standard JMX monitoring.

Share This:

One of the core competencies of TrueSight Server Automation (TSSA) is automating the patch update process for servers. TSSA makes this easy for the typical monthly cycle of patching servers in large groups, but in no particular order for each server in the group. Many applications are hosted on multiple servers, often in a HA (High Availability) and/or failover cluster to limit service outage. Automation to patch these type of services needs to update the nodes in a specific order usually one at a time, to prevent service outages during the patch update process. We call this Service Aware Patching.

 

Microsoft Exchange is one of these services, and I wanted to share here how we have implemented Service Aware Patching entirely in TrueSight Server Automation here at Customer Zero (IE BMC-IT).

 

Key Concepts

 

When automating any cluster service, we need to be able to idle the current target node by moving services off to the other nodes. Experience has taught is that changing the cluster state requires a careful methodical approach to be successful.

 

Key "milestones" in the process;

  1. Verify the target cluster node is in the "normal" state BEFORE attempting to change the cluster state
  2. Move services off the target node to idle the node (IE moving the node into Maintenance Mode)
  3. Verify the services successfully moved to another node and the target node is actually idle
  4. Use normal TSSA Patch Analysis and Deployment to update the patch level of the target node
  5. Verify the node and services are still in the idle state, as patch deployment probably restarted the node
  6. Move the services back onto the target node (IE move the node out of Maintenance Mode)
  7. Verify cluster node and cluster is in the "normal"  state BEFORE moving on the the next node is the sequence

 

Notice we use a careful logical sequence to move the target node through the required phases, we call this the "chain" of steps. Important point here, is that if ANY step in the chain fails the automation procedure needs to stop and NOT PROCEED to the next step. if something unexpected happens in the chain of steps on the cluster, ignoring a failed step and continuing on will most likely cause a service outage due to the automation not halting at a failed step in the chain.

 

TSSA takes care of the actual patch updates, the question is how do we implement the steps to handle the transitions of the cluster nodes?

 

Microsoft Exchange automation requires knowledge on how to verify the Exchange service and how to move the Exchange node in and out of "Maintenance Mode". Fortunately there are Exchange experts that have created existing powershell scripts to do what we need here for Service Aware Patching.

 

We used the powershell scripts available here;

 

The first issue we encountered is common when attempting to automate existing scripts of any type. If the script was designed for interactive use and not called by automation, then the scripts typically do not do things like set the exit code on failure. They often rely on the errors to STDOUT being read by the admin running the scripts. This was true with these powershell scripts, so one of our Exchange admins made a copy of the scripts we needed for the all important "verify" steps in our Service Aware Patching chain and modified them to set a non zero exit code if the script failed in any of the key operations or checks. Our TSSA deploy jobs will automatically detect and report failure if the script exits with a non zero exit code.

 

Once we had the automation procedure steps in the chain defined and the powershell scripts to perform the steps needed to manipulate the Exchange cluster, all the hard work was done! Now we just need to create the TSSA jobs required to execute the steps in the chain in the correct order!

 

Implementation in TrueSight Server Automation

 

We used TSSA batch jobs to implement the chain of steps required in executed in a prescribed order.

 

Note: Important options to set in the TSSA Batch jobs;
  • "Continue executing batch when individual jobs return non-zero exit code" is NOT set
  • "Execute jobs sequentially"

 

The main TSSA batch job to run our Service Aware Patching and control the node by node sequence in the cluster;

 

TSSA Batch Job: "Exchange 2013 - FULL Sequence Maintenance Mode Operation (PROD)"
  • Exchange 2013 - Maintenance Mode Operation (Node #1)
  • NSH - SleepDelay (5 minutes)
  • Exchange 2013 - Maintenance Mode Operation (Node #2)
  • NSH - SleepDelay (5 minutes)
  • Exchange 2013 - Maintenance Mode Operation (Node #3)
  • NSH - SleepDelay (5 minutes)
  • Exchange 2013 - Maintenance Mode Operation (Node #4)
  • NSH - SleepDelay (5 minutes)
  • Exchange 2013 - Maintenance Mode Operation (Node #5)
  • NSH - SleepDelay (5 minutes)
  • Exchange 2013 - Maintenance Mode Operation (Node #6)

 

This main TSSA batch job is the controller job that calls the child job for each node in the prescribed order, and any failure will cause the batch job to stop and not continue to the next node.

 

Note: In the case of any failures in the chain of steps
Manual intervention by the Exchange admins would be required to determine how to resolve without service impact.

 

Each of the child jobs is where our chain of steps are executed against each specific node in the cluster;

 

TSSA Batch Job: "Exchange 2013 - Maintenance Mode Operation (Node #?)"
  • Deploy - Verify Exchange READY for Maintenance Mode
  • Deploy - Move Exchange to Maintenance Mode
  • Deploy - Verify Exchange Maintenance Mode
  • PAJ - Exchange 2013 Servers
  • Build Provisioning (Windows) - CHECKPOINT Wait for Server
  • Deploy - Move Exchange to Normal Mode
  • Deploy - Verify Exchange Normal Mode

 

Note: There is one child job per node.

 

Here's a screenshot showing a successful run of one of the child jobs;

 

SAP-Exchange-MMO-1.png

 

Overview of the TSSA low level jobs and the associated powershell script

 

Deploy - Verify Exchange READY for Maintenance Mode
  • verify-server-is-NORMAL_MODE.ps1
Description: Script we created to check mailbox database (Get-MailboxDatabase) status is OK

 

Deploy - Move Exchange to Maintenance Mode
  • Start-ExchangeServerMaintenanceMode.ps1
Description: Script from VanHybrid site to move Exchange into "Maintenance Mode"

 

Deploy Verify Exchange Maintenance Mode
  • verify-server-is-IN-maintenance-mode.ps1
Description: Script we created to check various Exchange components are in the Inactive state and the cluster node state is "Paused"

 

Deploy - Move Exchange to Normal Mode
  • Stop-ExchangeServerMaintenanceMode.ps1
Description: Script from VanHybrid site to move Exchange out of "Maintenance Mode"

 

Deploy - Verify Exchange Normal Mode
  • verify-server-NOT-in-maintenance.ps1
Description: Script we created to check various Exchange components are in Active state and the cluster node state is "Up"

 

Summary

 

Our Service Aware Patching use case for Exchange Service uses a single TSSA Batch job to run the entire sequence end to end, moving one node at a time into maintenance mode and then updating the patch level using normal TSSA patching jobs, and then return the node to service. All nodes will be sequenced through and the end result will be the entire Exchange cluster patched with no service outage. All using automation.

Share This:

TrueSight Smart Reporting for Server Automation (TSSR-SA) 19.2 Patch 2 released on February 27th, 2020.

 

The official product version is TSSR-SA 19.2.02. As of July 9 2020, 19.2 Patch 2 is still the latest version and this blog posting will be updated after the next release in August 2020.

 

TSSR-SA is the Smart Reporting solution for Truesight Server Automation (TSSA) environments and is the replacement product for BDSSA which reached its End-Of-Life in August 2019.

 

The TSSR-SA solution consists of two main products which currently use different versioning schemes. Although documented, this can sometimes result in confusion around which versions of the product components are compatible and should be downloaded and installed together for a valid TSSR-SA solution installation.

 

The goals of this month’s blog are to clarify details around:

 

  1. Which products make up the TSSR-SA 19.2.02 solution.
  2. Which versions of the constituent products are compatible.
  3. Which versions of TSSA are compatible with TSSR-SA 19.2.02
  4. Which files should be downloaded from the BMC EPD site in order to install or upgrade to TSSR-SA 19.2.02
  5. When to follow the documented install path vs the upgrade path

 

All of this information can be found in the official product documentation and documentation links will be provided when referenced.

 

Since some of this information exists in the TSSR-SA documentation space and some exists in the TSSR Platform documentation space, this blog aims to help avoid confusion by highlighting the most important information in one single place.

 

As a first step, its a good idea to read Sean Berry's TrueSight Smart Reporting (TSSR, BDSSA) FAQ blog posting which is a higher level introduction to Truesight Smart Reporting for Server Automation.

 

Thanks,


John O’Toole

Principal Technical Supp Analyst

BMC Software Customer Support

 

 

 

1. What products make up the TSSR-SA Smart Reporting Solution?

 

 

TSSR-SA consists of two main constituent products:

  • TrueSight Server Automation - Data Warehouse (TSSA-DW)
  • TrueSight Smart Reporting – Platform (TSSR Platform)

 

 

The product version information for TSSR-SA 19.2.02, as listed in the Release Notes, is as follows:

 

 

TrueSight Smart Reporting for Server Automation 19.2.02 (Solution)

Component

Version

TrueSight Server Automation - Data Warehouse

8.9.04.004

TrueSight Smart Reporting - Platform

20.02

 

 

For completeness, this can be compared with the product version information for TSSR-SA 19.02.01 (Patch 1) which released in December 2019:

 

 

TrueSight Smart Reporting for Server Automation 19.2.01 (Solution)

Component

Version

TrueSight Server Automation - Data Warehouse

8.9.04.003

TrueSight Smart Reporting - Platform

19.3

 

And with TSSR-SA 19.2 which released in July 2019:

 

TrueSight Smart Reporting for Server Automation 19.2. (Solution)

Component

Version

TrueSight Server Automation - Data Warehouse

8.9.04.002

TrueSight Smart Reporting - Platform

19.2

 

 

 

2. Which versions of TSA-DW are compatible with which versions of TSSR Platform?

 

 

This information can be found in the TSSR-SA 19.2.02 Release Notes:

 

 

 

 

As we can see here, in a valid TSSR-SA environment, there is a tight relationship between the versions of TSA-DW and TSSR Platform.

 

When downloading the TSSA-DW and TSSR Platform installers, it is important that compatible versions are downloaded.

 

 
3. Which versions of TrueSight Server Automation (TSSA) are compatible with which versions of TSSR-SA?

 

This information can also be found in the TSSR-SA 19.2.02 Release Notes:

 

Compatibility with TrueSight Server Automation

The following versions of TrueSight Server Automation are supported with TrueSight Server Automation - Data Warehouse:

  • BMC Server Automation 8.9.0.x and 8.9.02
  • TrueSight Server Automation 8.9.03.x, 8.9.04.x, and 20.02

 

So, every version of TSSR-SA 19.2.X is compatible with every version of BSA/TSSA 8.9.X and TSSA 20.02

 

The version numbers of TSSA and TSA-DW are not tightly coupled i.e. if an environment is currently running TSSA 8.9.03 the version of TSA-DW installed does not need to match.

 

What is important is that the version of TSA-DW installed is compatible with the corresponding version of TSSR-Platform. (see previous compatibility matrix)

 

For example, the following would be a valid combination:

 

TSSA 8.9.03

TSSA-DW 8.9.04 Patch 4          (part of TSSR-SA 19.2.02)

TSSR Platform 20.02                 (part of TSSR-SA 19.2.02)

 

 

4. Which files should I download in order to install a TSSR-SA 19.2.02 environment?

 

As mentioned previously, the TSSR-SA 19.2.02 solution consists of the following product versions:

 

TrueSight Smart Reporting for Server Automation 19.2.02 (Solution)

Component

Version

TrueSight Server Automation - Data Warehouse

8.9.04.004

TrueSight Smart Reporting - Platform

20.02

 

 

Details on which files to download for both TSA-DW 8.9.04.004 and TSSR Platform 20.02 can be found in the “Downloading the Patch” section of the TSSR-SA 19.2.02 release notes.

 

 

 

a) Locating and downloading the installer files for TSSA-DW 8.9.04.004

 

Since TSSA-DW 8.9.04.004 is a patch, the installation files are found under the “Product Patches” tab on EPD as highlighted below:

 

 

 

From here, we navigate to the TSSA-DW 8.9.04 area:

 

 

Under here, the TSSA 8.9.04.004 installation files are all dated 02/27/2020:

 

 

 

b) Locating and downloading the installer files for TSSR Platform 20.02

 

TSSR Platform 20.02 is a full product release so the installation files are found under the “Licensed Products” tab on EPD as highlighted below:

 

 

 

 

 

From here, we download either the Windows or the Linux installer depending on the OS of the TSSR-SA Server:

 

 

 

 

5. Once the installation files are downloaded, which documentation steps do I follow to install or upgrade TSSR-SA 19.2.02?

 

 

The steps to follow depend on whether this is the initial installation of TSSR-SA in the environment or whether a previous version of TSSR-SA (i.e. 19.2, 19.2.01) was previously installed and is now being upgraded to TSSR-SA 19.2.02.

 

 

Scenario A – Installing TSSR-SA for the first time in an environment:

 

This is the initial installation of TSSR-SA in this environment.

 

BDSSA is the TSSA reporting solution currently in use and we want TSSR-SA to use the same Warehouse DB that BDSSA has been using. This is sometimes referred to as a "migration" from BDSSA to TSSR-SA.

 

These are also the steps to follow if neither BDSSA nor TSSR have previously been installed in this environment.

 

In this scenario, we want to follow the Installing section in the TSSR-SA documentation.

 

Since the installation process includes steps from both the TSSA-DW and TSSR Platform documentation spaces, they are listed here in order:

 

  1. Review the TSSR-SA 19.2.02 Release Notes
  2. Review the TSSR-SA 19.2.02 Orientation for BDSSA users
  3. Review the TSSR-SA 19.2.02 Getting Started section
  4. Complete the Planning activities for TSSA-DW. Note: If the TSSA-DW installation will be using the same databases as BDSSA, no new databases or schemas need to be created for TSSA-DW.
  5. Complete the Prepare to install activities for TSSA-DW
  6. Complete the TSSA-DW Installation steps
  7. Complete the TSSA-DW Post-Installation tasks. Note: Step 2 of the Post-Installation tasks is the step where we provide TSSA-DW with the database information for the Warehouse, two ETL databases and the TSSA database.

 

 

As this point, we should have a functioning TSSA-DW installation and should be able to perform an ETL run as mentioned in step 6 of the Post Installation tasks.

 

If ETL is successful, we can continue to the TSSR 20.02 Platform installation which consists of the following steps:

 

8. Complete the TSSR Platform 20.02 Planning activities.

9. Complete the TSSR Platform 20.02 Preparing To Install activities. ( See point 6B below)
10. Complete the TSSR Platform 20.02 Installation steps
11. Complete the TSSR platform 20.02 Post Installation tasks which consists of two main steps:

  1. Adding the TSA-DW Component to TSSR Platform
  2. Configuring TSSR Platform settings

 

 

Scenario B – Upgrading an existing TSSR-SA environment:

 

A previous version of TSSR-SA (19.2 or 19.2.01) has already been installed in this environment and the goal is to upgrade it to TSSR-SA 19.2.02.

 

In this case, we want to follow the Upgrade steps in the TSSR-SA documentation.

 

  1. Review the  TSSR-SA 19.2.02 Release Notes
  2. Review the TSSR-SA 19.2.02 Upgrading page
  3. Complete the TSSA-DW Preparing to upgrade activities
  4. Complete the TSSA-DW Upgrade steps.
  5. Complete the TSSR Platform 20.02 Preparing to upgrade activities
  6. Complete the TSSR Platform 20.02 Upgrade steps
  7. Complete the TSSR-SA 19.2.02 Post-upgrade tasks. These steps are very important in order to avoid post-upgrade performance, resource usage, data-refresh or authentication issues

 

 

6. Important points and common questions/pitfalls

 

This section will be updated as common TSSR-SA 19.0.02 install/upgrade issues and questions are encountered.

 

A) Warehouse DB data condition to check before running TSSA-DW post-installation steps.

 

Make sure step 1 of the TSSA-DW Post-installation steps is followed if this is the initial migration of a BDSSA environment to TSSR-SA. Skipping this step may result in errors during the Warehouse DB upgrade.

 

"If you are performing the initial migration of BMC Decision Support for Server Automation warehouse to TrueSight Server Automation - Data Warehouse, you must run a diagnostic SQL query to verify that the version of BMC Decision Support for Server Automation warehouse is as expected. For more information, see this knowledge article"

 

 

 

B) Creating the TSSR Repository DB/Schema

 

When doing a fresh installation of TSSR-SA in an environment where BDSSA is currently installed, and the same Warehouse and ETL DBs are to be used by TSSR-SA, there can sometimes be confusion around the new TSSR Repository DB which TSSR Platform uses to store information such as metadata, users, and user permissions for reports.

 

Unlike the Warehouse and ETL DBs, this is not a DB which was present or used by BDSSA. This is a net-new DB required by TSSR Platform.

 

Oracle Environments:

For Oracle DB environments, the TSSR Repository user must be created in advance of running the TSSR Platform installer.

The installer is then provided with the DB connection information. See the "Setting up Oracle as the repository database" section of the Setting up the TSSR Repository Database page.

This page contains details on creating the Oracle TSSR Repository user, the tablespace (if a separate tablespace is preferred) and granting the required privileges.

 

 

SQL Server Environments:

For SQL Server environments, the user can choose whether to create the TSSR Repository DB and user in advance of running the TSSR Platform installer or to allow the installer to create the TSSR Repository DB and user.

 

If the option to have the installer create the DB and User is selected, then the installer requires Database Administrator credentials to be provided so that these tasks can be completed.

If the option to create the DB and User manually in advance is selected, then Database Administrator credentials will not be required by the installer.

Share This:

We are glad to announce the release of new version of TrueSight Server Automation 20.02.00. This new release brings in enhancements to the core features which help to stabilize the functionality.

 

RSCD Agent Enhancement:

The RSCD agent now includes a new Smart Agent feature to perform the following tasks:

  • Monitor the status of RSCD agent and share the status information with the Application Server.
  • Send a request to the Application Server to register the new server without manual intervention.

 

Patch Enhancement:

TrueSight Server Automation now supports the rebooting of Windows servers before applying patches to prevent patch job failures. When you run a patch job on the Windows server that is waiting for a reboot, the server is first rebooted and then the job is started.

 

Compliance templates available for CIS Windows 2019 and DISA STIG Windows 2019

 

Security enhancements

 

BLCLI support:

New BLCLI Utility - simpleExportAuditResultV2" command is added to export the results of an Audit job.

 

RESTful API support:

New REST APIs are added to the following resources: Deploy jobs, NSH script jobs, Patching jobs, and BLPackage

You can use these new APIs to perform the following operations:

  • Execute with approval operation on Deploy, NSH Script, and Patching jobs.
  • Update schedules for NSH Script and Deploy jobs.
  • Retrieve the details of a BLPackage.

 

For more details, please refer to the online documentation - 20.02 enhancements - Documentation for TrueSight Server Automation 20.02 - BMC Documentation

Share This:

We are glad to announce the release of a newer version of BMC’s Automation Console 20.02.

This console is an interface built on top of BMC Server Automation to simplify the OS patching experience for servers as well as for providing a single platform for vulnerability remediation and patching.

This console is available as a service, called BMC Helix Automation Console (SaaS), and as an on-premises product, called TrueSight Automation Console.

Here are the key highlights for this release:

Ability to create change ticket and approvals:

As part of the vulnerability remediation operation, you can now create a change request in the change management system, which tracks the operations, and goes through a change approval process. After a change request is approved, the operation runs according to the schedule.

Approvers can also reschedule the operation if needed.

 

ChangeApprovalManagement.png

Blind spot detection:

Automation Console integrates with BMC Discovery to find servers in your environment and were not scanned for vulnerabilities. Such servers or assets are blind spots and can be a potential security risk as there might be critical undiscovered vulnerabilities on those servers. The Discovered Assets page lists such assets. Key Performance Indicators (KPIs) on the Discovered Assets page show information about the total number of discovered assets, assets that are discovered but not mapped to endpoints in Server Automation, and assets that are not yet scanned. You must ensure that the discovered assets are scanned for missing patches and vulnerabilities.

Business Service Context:

On vulnerability dashboard, you can also see top 10 business services or applications with the maximum number of vulnerabilities and impacted assets. This will help you in prioritizing the remediation based on the business service impact.

 

These are just 3 key highlights, to deep dive into whole set of features delivered as part of this release, please go through the documentation link.

Please contact us if you would like to share any feedback.

Share This:

This question has come up a few times:  How can I remove the patch payloads from a patch repository that were downloaded but will likely not be used again ?  You can't just delete the underlying file off the file system because TSSA still has it flagged as downloaded and if you happen to need the patch again (maybe you are patching a newly provisioned system), you will get errors because TSSA can't find the patch file.

 

This script solves that problem for Windows catalogs.  It uses the DATE_CREATED property value (when the patch was added to the catalog) to find old patches whose payload exists on the repository file system, determines if the patch object has no dependencies, deletes the file, and then resets the downloaded flag to no.  If the patch is needed in the future, it will be re-downloaded.  The script will not run against: offline catalogs, catalogs with Download from Vendor checked, or catalogs that share a repository location with other catalogs in the same TSSA environment.  This should not be run against a TSSA environment that shares the catalog location with another TSSA environment.

 

Standard disclaimer: not supported, may not work for you, may cause small fires, don't test this in production.

 

Basic usage:

 nsh removeOldPatchesFromCatalog.nsh -P defaultProfile -R BLAdmins -c "<full path to catalog>"  -r <retention in days >

example:

 nsh removeOldPatchesFromCatalog.nsh -P defaultProfile -R BLAdmins -c "/Workspace/Patch Catalogs/Windows 2019"  -r 30

 

Will remove patches 30 days or older from the /Workspace/Patch Catalogs/Windows 2019 catalog.

 

The retention value should be longer than your patch cycle.  If it normally takes you about a month to patch all your servers, then use a retention value of 45 to 60 days.

 

Feedback is welcome.

Share This:

Depending on your licensing agreement with RedHat, the entitlements to the repositories you are using in a single Patch Catalog may be spread across multiple certificates.  Alternatively, you may need to maintain multiple Patch Catalogs, each with separate repository servers, and each needing their own set of entitlement certificates, and you cannot use the single set defined in the Patch Global Configuration.  Currently it is not possible to handle either of these situations.  With a small update to the yum_metadata_generator.sh both of these situations can be handled.  Also, since this will bypass the Patch Global Configuration settings for the entitlement certificate and key, you no longer have to worry about the case when RedHat revokes the certificates and subscription-manager on your repository server gets new certificates in /etc/pki/entitlements.

 

Standard disclaimer here:  this is not supported, will be removed by an upgrade, may not work in future versions, and may not work in your environment, etc etc.

 

What we need to do is look at the url defined in the repository configuration generated by the Catalog Update Job and search for it in the entitlement certificates present on the system.  Then we can update the generated repository configuration with the appropriate certificate present on the repository server.

 

First, locate the support-files-1.0-SNAPSHOT.jar file in the NSH/br/stdlib directory on an appserver.  Extract the script from this file into a temporary directory:

#unzip /opt/bmc/bladelogic/NSH/br/stdlib/support-files-1.0-SNAPSHOT.jar -d /tmp com/bmc/sa/patchfeed/redhat/yum_metadata_generator.sh
Archive:  /opt/bmc/bladelogic/NSH/br/stdlib/support-files-1.0-SNAPSHOT.jar
  inflating: /tmp/com/bmc/sa/patchfeed/redhat/yum_metadata_generator.sh  

 

Around line 348, just above:

#main
print_log "Started executing yum metadata generator script" 

 

add the following function:

updateEntitlements()
{
    while read repo
        do
        echo "REPO: ${repo}"
        for cert in /etc/pki/entitlement/*[0-9].pem
            do
            echo "Checking ${cert}..." | tee -a $log_file_name
            if grep -q "$(awk -v val="URL:" '{if ($1 == val) print $2 }' <<< "$(rct cat-cert "${cert}")"  | sed "s/\$basearch/${repoArch}/g;s/\$releasever/\.\*/g")" <<< "$(awk -v val="baseurl" '{if ($1 == val) print $3}' <<< "$(yum-config-manager -c ./yum.conf ${repo})")"
                then
                echo "Using ${cert} for ${repo}"  | tee -a $log_file_name
                yum-config-manager -c ./yum.conf --save --setopt=${repo}.sslclientkey=${cert%.*}-key.pem
                yum-config-manager -c ./yum.conf --save --setopt=${repo}.sslclientcert=${cert}
                break
            fi
        done
    done <<< "$(awk -v val="repo:" '{ if ($2 == val ) print $3}' <<< "$(yum-config-manager -c ./yum.conf )")"
}

 

After the export LD_LIBRARY_PATH="" around line 353 (after the addition above) call the function:

echo "Started executing yum metadata generator script" 

cd $repoDir
export LD_LIBRARY_PATH=""
updateEntitlements

 

Copy the support-files-1.0-SNAPSHOT.jar to the root of the temporary directory where you extracted it earlier.  Then replace the yum_metadata_generator.sh in the zip with the altered version:

cd /tmp
# cp /opt/bmc/bladelogic/NSH/br/stdlib/support-files-1.0-SNAPSHOT.jar .
# zip support-files-1.0-SNAPSHOT.jar com/bmc/sa/patchfeed/redhat/yum_metadata_generator.sh 
 updating: com/bmc/sa/patchfeed/redhat/yum_metadata_generator.sh (deflated 75%)

 

Stop the application server services.

Make a backup of the original support-files-1.0-SNAPSHOT.jar somewhere outside of the application server install directory.

Delete the NSH/br/stdlib/support-files-1.0-SNAPSHOT.jar

Copy the altered support-files-1.0-SNAPSHOT.jar into NSH/br/stdlib

On Linux the file should be owned by root and be owner read and write, group read, and everyone read (644)

Start the application server services

 

This will now look for entitlement certificates in the /etc/pki/entitlements directory on the repository server, ignoring what is defined in Patch Global Configuration.  As long as that directory contains your entitlement certificates for all of the repositories you have in your Patch Catalog, the Catalog Update Job should run without errors related to missing entitlements for one of the repositories being downloaded.

 

The same alteration can be performed to the offline downloader, with the support-files-1.0-SNAPSHOT.jar existing in the <downloader dir>/lib directory.

 

Since an upgrade will overwrite the altered jar file, you will need to perform the above modifications after an upgrade and re-test the Catalog Update Job after an upgrade.

Share This:

On January 7th, 2020 there was a change to the URL used by TrueSight Server Automation (TSSA), versions 8.9 Service Pack 3 (SP3) and 8.9 Service Pack 4 (SP4), to retrieve the xml metadata file from Ivanti.  TSSA version 20.02 and above are not effected.  When executing a Windows Catalog Update Job it may fail with the following error:

Error 01/17/2020 12:07:05 Validation Error :- BLPAT1004 - Http Url is not accessible.
Possible remediation steps :- Please check the proxy login credentials
Please check the vendor url
Please check network access to vendor url
https://content.ivanti.com/data/oem/BMC-Bladelogic/data/partner.manifest.xml

 

 

Note the URL in the error message “https://content.ivanti.com/data/oem/BMC-Bladelogic/data/partner.manifest.xml”.  This is now no longer a valid URL.  In support of the newer Shavlik SDK version 9.3 the URL has changed to a new location.  In some instances the old/original URL may work, but will eventually be fully disabled.  Ivanti now provide the metadata file at this URL:

https://content.ivanti.com/data/oem/BMC-Bladelogic/data/93/manifest/partner.manifest.xml

 

In TSSA the URL location to partner.manifest.xml should be updated within the Patch Global Configuration’s Shavlik URL Configuration tab.

GlobalPatchConfig_URLConfig.png

 

 

See BMC Knowledge Article 000181848 for further detailed steps to update the Patch Global Configuration.

When using the Windows off-line downloader utility the patch-psu.properties files within the resources sub-directory will need to be modified.  Update the “windows.shavlik.manifest_url” parameter to reflect the new URL location, so it will look like:

 

windows.shavlik.manifest_url = https://content.ivanti.com/data/oem/BMC-Bladelogic/data/93/manifest/partner.manifest.xml

 

 

For BladeLogic Server Automation (BSA) version 8.9 Service Pack 2 (SP2) or lower Windows Patch Analysis is no longer supported, as of September 30, 2019.  An upgrade will be required to use this feature again as per the earlier notification here:

https://docs.bmc.com/docs/tssa89/notification-of-action-needed-for-users-of-truesight-server-automation-microsoft-windows-patching-815403716.html

 

 

For continued status information of Patch Analysis functionality follow the “OS Patching Vendor Health Dashboard” in the BMC Communities. 

Also be sure to subscribe to the TrueSight Automation Video Channel for related videos and notifications as new video content is published.

Share This:

Hello Everyone,

 

Windows Patch Management is one of the most heavily used features of TrueSight Server Automation (TSSA) and involves the interaction between TSSA, Ivanti Technologies (previously known as Shavlik) and the underling Microsoft Windows Operating System and patch.

 

The most common TSSA Windows Patch Management support cases we receive tend to fall into the one of the following categories:

 

  1. A Windows Patch is believed to be installed on a target server but is reported as missing by the TSSA Patch Analysis job.
  2. A Windows Patch is not installed on a target server but is not reported as missing by the TSSA Patch Analysis job.
  3. How to add new products to the list of filters available in a TSSA Windows Patch Catalog.
  4. TSSA Windows Patch Analysis issues related to target server reboots.
  5. How Microsoft Servicing Stack Updates (SSUs) can affect TSSA Windows Patch Analysis results.
  6. In May 2019, Windows Catalog Updates began failing due to changes in how Oracle JAVA patches are distributed.

 

The TSSA Customer Support team has been working on enhancing our knowledge base articles and video content in these areas and, in this month’s blog, we wanted to highlight the most useful, and frequently used, knowledge articles and videos for each of the problem categories listed above.

 

All of these issues are also discussed in the  Connect with TrueSight - TSSA: Troubleshooting Best Practices for Windows Patching Webinar which was presented on May 12 2020 by Rohit Sharma. If you have not already viewed this webinar, it is 86 minutes and discusses all the most common TSSA Windows Patching support cases we encounter from Catalog Updates to Analysis to Deployment/Remediation.

 

 

1) False Positives – Windows patch is believed to be installed but reported as Missing by TSSA Patch Analysis

 

 

Knowledge Articles:

 

000099904: TSSA/BSA: Windows Patch Troubleshooting - A Windows Hotfix is reported as missing by TSSA Patch Analysis but is believed to be installed

 

000090870 TSSA/BSA: Windows Patch Troubleshooting - Patch Deploy Job appears to have succeeded but TSSA Patch Analysis still reports the patch as missing

 

Video:

 

 

 

 

2) False Negatives – Windows patch is believed to be missing but is not reported as Missing by TSSA Patch Analysis

 

Knowledge Article:

 

000095840 TSSA/BSA: Windows Patch Troubleshooting - A Windows Hotfix is not installed on a target server but is not reported as missing by TSSA Patch Analysis

 

 

3) How to add additional products to the list of available filters in a TSSA Windows Patch Catalog

(This topic can also surface as a “No mappings were found for the selected product" warning when running a Windows Patch Catalog)

 

Knowledge Articles:

 

000166476 BSA/TSSA: How to add a new product to the list of available filters available in a TSSA Windows Patch Catalog?

000130493 BSA/TSSA: Windows Catalog update job displays "No mappings were found for the selected product" warning in Job Run Log

 

Video:

 

 

 

 

4)     TSSA Windows Patch Analysis issues related to target server reboots

 

Knowledge Articles:

 

000081738 TSSA/BSA: Windows Deploy Job fails to reboot, reports "Reboot required but did not occur. Manual reboot needed to complete operation, exitCode = -4003"

 

000145107 TSSA/BSA: Reboot is pending on this machine, analysis results may be incorrect

 

See also Newton Nyante's April 2020 blog posting specifically about how Pending Reboots can affect TSSA Windows Patch Analysis and Remediation.

 

 

5) How Microsoft Servicing Stack Updates (SSUs) can affect TSSA Windows Patch Analysis results

 

 

Knowledge Article:

 

000167748 TSSA/BSA: How can Microsoft Servicing Stack Updates (SSUs) affect TSSA Windows Patch Analysis results?

 

 

6) In May 2019, Windows Catalog Updates began failing due to changes in how Oracle JAVA patches are distributed.

 

Knowledge Article:

 

000168397 TSSA/BSA: Beginning May 2019 - Windows Catalog Updates failing due to problems downloading Oracle Java patches

 

 

 

Troubleshooting TSSA Windows Patch Analysis cases will often require downloading and running the Ivanti DPD Trace tool in order to gather more detailed information. Please see the following video which demonstrates how to download and run the Ivanti DPD Trace tool:

 

 

 

And Knowledge Article 000096560 which describes how to analyze the Trace.txt and shavlik_results.xml files generated by a TSSA Patch Analysis Job.

 

 

Finally, please make sure to subscribe to the TrueSight Automation Video Channel to find more useful videos and to be notified of new video content as it is published. If you are particularly interested in TSSA Windows Patch Management videos, we have a playlist specifically for this feature.

Share This:

Adding Authorizations to an Acl Template is typically accomplished by running a series of blcli_execute BlAclTemplate addTemplatePermission commands.  Some brief profiling shows that adding 100 Authorizations to a newly created Acl Template takes about 10 seconds.  That's not too bad, but I wonder if it could be faster.  If we look in the unreleased blcli commands documentation for the addTemplatePermission command we can see what it runs:

Reading through the sequence of commands what happens is that it loads up the acl object from the template (the thing that contains the list of role:authorizations), then creates a new acl entry (blAce), adds the new role and authorization, and then updates the acl object with this acl entry.  Then the acl template object is updated with the newly update acl list.  It's likely that instead of immediately running the template update, we can keep adding more and more acl entries to the acl object and do a single update at the end.

 

We want to be able to compare the run times to the current method of adding acls so we should build out a script that does both and compare run times.  I'll just grab a subset of the entire list of authorizations for this test and then run each method, noting the runtime.

 

#!/bin/nsh
# load the date/time module so we can use $EPOCHSECONDS
zmodload zsh/datetime
blcli_setjvmoption -Dcom.bladelogic.cli.execute.quietmode.enabled=true
blcli_setoption serviceProfileName defaultProfile
blcli_setoption roleName BLAdmins
# get just the system authorizations for the test
blcli_execute Authorization findAllByType 1
blcli_execute Authorization getName
blcli_execute Utility setTargetObject
blcli_execute Utility listPrint
blcli_storelocal allAuths
# grab 100, for further profiling get more.  for a real run you'd be reading the list from a file probably.
myAuths="$( tail -100 <<< "${allAuths}")"
echo "Number of auths: $(awk 'NF' <<< "${myAuths}" | wc -l)"
for i in {1..10}
       do
        startTime=${EPOCHSECONDS}
        # create the empty acl template
        blcli_execute BlAclTemplate createAclTemplate Template1 Template1
         # loop through the list of authorizations i pulled and add them to the template
        while read i
                do
                blcli_execute BlAclTemplate addTemplatePermission Template1 BLAdmins "${i}"
        done <<< "$(awk 'NF' <<< "${myAuths}")"
        endTime=${EPOCHSECONDS}
         # get the runtime.
        echo "addTemplatePermission RunTime=$((${endTime}-${startTime}))"
        blcli_execute BlAclTemplate deleteAclTemplateByName Template1

        startTime=${EPOCHSECONDS}
         # create the template, step through the underlying calls for addTemplatePermission
        blcli_execute BlAclTemplate createAclTemplate Template2 Template2
        blcli_execute BlAclTemplate findByName Template2
        blcli_execute Utility storeTargetObject template
        blcli_execute BlAclTemplate getTemplateBlAcl
        blcli_execute Utility setTargetObject
        blcli_execute Utility storeTargetObject blAcl

         # loop over the list of auths to add, add them to the acl object and do the update later.
        while read i
                do
                blcli_execute RBACRole getRoleIdByName BLAdmins
                blcli_execute Utility setTargetObject
                blcli_execute Utility storeTargetObject roleId
                blcli_execute Authorization getAuthorizationIdByName "${i}"
                blcli_execute Utility setTargetObject
                blcli_execute Utility storeTargetObject authId
                blcli_execute Utility setTargetObject
                blcli_execute BlAce createInstance NAMED_OBJECT=roleId NAMED_OBJECT=authId
                blcli_execute Utility setTargetObject
                blcli_execute Utility storeTargetObject blAce
                blcli_execute Utility setTargetObject blAcl
                blcli_execute BlAcl addAce NAMED_OBJECT=blAce
        done <<< "$(awk 'NF' <<< "${myAuths}")"
        blcli_execute Utility setTargetObject template
        blcli_execute BlAclTemplate update NAMED_OBJECT=template
        blcli_execute BlAclTemplate getDBKey
        endTime=${EPOCHSECONDS}
        echo "unreleased RunTime=$((${endTime}-${startTime}))"
        blcli_execute BlAclTemplate deleteAclTemplateByName Template2
done

 

Running the above shows (collapsed lines for space):

addTemplatePermission RunTime=15 unreleased RunTime=2

addTemplatePermission RunTime=8 unreleased RunTime=2

addTemplatePermission RunTime=8 unreleased RunTime=2

addTemplatePermission RunTime=9 unreleased RunTime=2

addTemplatePermission RunTime=8 unreleased RunTime=2

addTemplatePermission RunTime=7 unreleased RunTime=2

addTemplatePermission RunTime=8 unreleased RunTime=2

addTemplatePermission RunTime=8 unreleased RunTime=3

addTemplatePermission RunTime=8 unreleased RunTime=2

addTemplatePermission RunTime=8 unreleased RunTime=2

An average of 8 seconds for the loop of addTemplatePermission commands, average of 2 seconds for the set of unreleased commands.

 

To actually use the new method your script will look something like:

blcli_execute BlAclTemplate createAclTemplate MyTemplate MyTemplate
blcli_execute BlAclTemplate findByName MyTemplate
blcli_execute Utility storeTargetObject template
blcli_execute BlAclTemplate getTemplateBlAcl
blcli_execute Utility setTargetObject
blcli_execute Utility storeTargetObject blAcl

while read auth role
     do
     blcli_execute RBACRole getRoleIdByName "${role}"
     blcli_execute Utility setTargetObject
     blcli_execute Utility storeTargetObject roleId
     blcli_execute Authorization getAuthorizationIdByName "${auth}"
     blcli_execute Utility setTargetObject
     blcli_execute Utility storeTargetObject authId
     blcli_execute Utility setTargetObject
     blcli_execute BlAce createInstance NAMED_OBJECT=roleId NAMED_OBJECT=authId
     blcli_execute Utility setTargetObject
     blcli_execute Utility storeTargetObject blAce
     blcli_execute Utility setTargetObject blAcl
     blcli_execute BlAcl addAce NAMED_OBJECT=blAce
done < /tmp/MyAuthList.txt
blcli_execute Utility setTargetObject template
blcli_execute BlAclTemplate update NAMED_OBJECT=template
blcli_execute BlAclTemplate getDBKey

 

Where /tmp/MyAuthList.txt has the format like:

role authname

Filter Blog

By date:
By tag: