Share This:

Recently, the Support team has seen an increase in the number of PATROL Agent security issues. These security issues can lead to several problems in the TrueSight arena. The most commonly seen error when the PATROL Agent will not connect to Integration Service is found in the PATROL Agent error log:

Enabling Integration Service support subsystem.
ESS Error: ESS_Policy_InitSecurityPolicy() failed >-4<

ESS Error: Security policy is either missing or unreadable: /etc/patrol.d/security_policy_v3.0/proxy.plc/common /etc/patrol.d/security

_policy_v3.0/proxy.plc/client /etc/patrol.d/security_policy_v3.0/proxy.plc/common /etc/patrol.d/security_policy_v3.0/proxy.plc/client >-4<

 

The root cause for almost all of these errors referenced above is permissions on the security files.

 

We also see a PATROL Agent security problem when the Patrol Agent has been deployed some time in an environment where an older version of the Agent is being used. When the older Agent is upgraded to the latest version, some Administrators have been using the overwrite security checkbox set to “Yes” setting. Generally, there really is no reason to set the overwrite security to “Yes”.

 

The recommendation is to leave the security setting to “No” when upgrading the Agent. Using this setting can cause major interruptions in the operation of these Agents if they are used for TSIM or the Integration Service if the Agent is on the same VM or server.  It can also cause issues with certificates and security level settings of the Patrol Agent where the security level is higher than level 2 like the example below in the deployable package creation:

PASecurity.png

  • There are some cases where the PATROL Agent is installed and does not start in the environment. If the install was not done as root, users can run through the details of this Knowledge Article to fix the permissions issue.

 

  • Other cases which looked like a PATROL Security issue ended up being a firewall issue, so it is important to check permissions and firewalls when users cannot connect to the newly deployed PATROL Agents. In the instance where the Agent was not connecting, we found the error “TCP |1537197978| Fd = 924 NumRead = -1 errno = 10054” in the Patrol Console debug log. This error means that the connection got reset by the peer. Error 10054 occurs when the connection is reset by the peer application, usually due to an incorrect firewall configuration. A socket error 10054 may be the result of the remote server or some other piece of network equipment forcibly closing or resetting the connection.
    More details are available at https://docs.microsoft.com/en-us/windows/desktop/winsock/windows-sockets-error-codes-2 and  http://help.logmein.com/articles/en_US/FAQ/What-does-Error-10054-mean
    So, if you are seeing the TCP 10054 error in the Patrol Console debug log, please work with your Network Administrator, to check the firewall setting or any other network setting which is prohibiting cross domain communication.

 

  • We have seen cases where the Patrol Agent is unable to connect to the Integration Service on any IP Address, along with a  Time_Wait when doing a netstat on port 3183. Even after a restart of the OS on both the Integration Server and the Patrol Agent box, the issue remained. Usually if this is still an issue, Support would like to review the following

- %PATROL_HOME%\log\PatrolAgent-hostname-3181.errs on Patrol Agent box
- Agent\pronto\logs folder on IS

 

In this case, the logs did show the common error indicating a permissions issue, see the snippets below:

Enabling Integration Service support subsystem
ESS Error: ESS_Policy_InitSecurityPolicy() failed >-4<
ESS Error: Security policy is either missing or unreadable: /etc/patrol.d/security_policy_v3.0/proxy.plc/common  /etc/patrol.d/security_policy_v3.0/proxy.plc/client /etc/patrol.d/security_policy_v3.0/proxy.plc/common /etc/patrol.d/security_policy_v3.0/proxy.plc/client >-4<
Integration Service Client policy should be at security level 2. Current Level = -1.

 

          This issue was resolved through changing the permissions based on the details below:

          The file /etc/patrol.d/security_policy_v3.0/proxy.plc had been installed with the wrong permissions.

          These are the permissions in place:
          /etc/patrol.d/security_policy_v3.0:
     total 144
          drwxr-xr-x 3 root sys 8192
          drwxr-xr-x 4 root sys 8192
          -rw------- 1 root sys 866 agent.plc
          drwxr-xr-x 2 root sys 96 bak
          -rw-r--r-- 1 root sys 226 client.plc
          -rw-r--r-- 1 root sys 549 esi.plc
          -rw------- 1 root sys 1240 proxy.plc
          -rw-r--r-- 1 root sys 40 signer.plc
          -rw-r--r-- 1 root sys 1459 site.plc
          -rw-r--r-- 1 root sys 46 verifier.plc

                   

              We can see that some of the policy (.plc) files had the correct permissions of 644 (-rw-r--r--), but the agent.plc and proxy.plc files only had 600 permissions (-rw-------).

 

             This meant that the Patrol default account could not read these files. These are the commands run as root during the install or post install using the .log_rootscripts file:

 

         

For the .plc files with correct permissions:

/products/patrol/9.10.00/common/security/config_v3.0/policy_install.sh /products/patrol/9.10.00/common/security/config_v3.0/site_0.plc site.plc FALSE
/products/patrol/9.10.00/common/security/config_v3.0/policy_install.sh /products/patrol/9.10.00/common/security/config_v3.0/esi.plc esi.plc FALSE
/products/patrol/9.10.00/common/security/config_v3.0/policy_install.sh /products/patrol/9.10.00/common/security/config_v3.0/signer.plc signer.plc FALSE
/products/patrol/9.10.00/common/security/config_v3.0/policy_install.sh /products/patrol/9.10.00/common/security/config_v3.0/verifier.plc verifier.plc FALSE
/products/patrol/9.10.00/common/security/config_v3.0/policy_install.sh /products/patrol/9.10.00/common/security/config_v3.0/client.plc client.plc FALSE

The script policy_install.sh has the following line towards the end:
chmod 644 /etc/patrol.d/security_policy_v3.0/$dest_policy_file
This sets the correct permissions.

Now let's look at problem files:

/bin/cp /products/patrol/9.10.00/Patrol3/security/agent.plc /etc/patrol.d/security_policy_v3.0/agent.plc
/bin/cp /products/patrol/9.10.00/Patrol3/security/proxy.plc /etc/patrol.d/security_policy_v3.0/proxy.plc

The files are simply copied with no chmod. If by default, the root account does not create files with a umask of 022, then the permissions on these files will be wrong. In order to resolve this, simply modify the permissions like this:

 

chmod 644 /etc/patrol.d/security_policy_v3.0/agent.plc
chmod 644 /etc/patrol.d/security_policy_v3.0/proxy.plc

Then restart the PatrolAgent. Please note that the Patrol installation assumes that the default umask will be 022. If it is not this, then files may be laid down with

incorrect permissions in some cases.

 

  • The times when the security errors appear in the PATROL Agent error log it is most likely permissions related, please do the following

          Stop the PATROL Agent and take a backup of agent.plc and proxy.plc from /etc/patrol.d/security_policy_v3.0

 

          Copy agent.plc and proxy.plc from /opt/bmc/patrol/patrol/security :

/bin/cp /opt/bmc/patrol3/security/agent.plc /etc/patrol.d/security_policy_v3.0/agent.plc
/bin/cp /opt/bmc/patrol3/security/proxy.plc /etc/patrol.d/security_policy_v3.0/proxy.plc

 

          Ensure that the permissions of the .plc files are 644 (-rw-r--r--). If not, then issue chmod's :      

       chmod 644 /etc/patrol.d/security_policy_v3.0/agent.plc
       chmod 644 /etc/patrol.d/security_policy_v3.0/proxy.plc

 

          Start PATROL Agent and it should now connect to Integration Service.

 

 

Permissions issues are not the only cause of PATROL Security issues, but if you do encounter issues, please open a case with Support for further assistance.

 

Knowledge Articles related to these types of issues

000160869 /etc/patrol.d/patrol.conf permission changes to 600 after Patrol installation/upgrade in some scenarios when umask is set to 0077

000092497 Patrol Agent startup error: Security policy is either missing or unreadable

000099021 Patrol Agent Error: ESS_Policy_InitSecurityPolicy() failed >-4<

000099188 PATROL Agent will not connect to Integration Service

 

 

AMIGO.jpg

TrueSight Operations Management 11.3.02 is out....plan to upgrade.migrate now!

Note: 11.3.02 requires users to upgrade/migrate to 11.3.01 first and the apply the Service Pack to move to 11.3.02

The BMC Assisted MIGration Offering, or AMIGO, is a program designed to assist our customers in planning and preparing for product upgrades from an older, to a newer supported version.  By engaging with BMC Technical Support Analysts, you will be provided with materials containing guidelines and best practices to aid in compiling your own upgrade plan. An upgrade expert will then review your plan, and offer advice and suggestions to ensure success through proper planning and testing.

The AMIGO program consists of a Starter Phase and a Review Phase.  Each phase is initiated by opening a support case, and ends when the case is closed.

In the Starter Phase, an AMIGO Starter case is opened.  Reference material will be provided and a call with a Technical Support Analyst will take place to discuss the details of your upgrade, and address any questions you may have.  The AMIGO Starter case will be closed, and the next step will be for you to prepare a documented upgrade plan.

In the Review Phase, an AMIGO Review case is opened preferably two weeks prior to a set upgrade date.  A call will be scheduled with an upgrade expert to review your detailed plan, providing feedback and recommendations, along with answers to any outstanding questions.  As needed, a follow up discussion with a Technical Support Analyst may take place for feedback after the upgrade is performed.

The AMIGO program includes:

» A “Question and Answer” session before you upgrade

» A review of your upgrade plan with Customer Support

» An upgrade checklist

» Helpful tips and tricks for upgrade success from previous customer upgrades

» A follow-up session with Customer Support to let them know how it went. This will help BMC to enhance the process.

 

To get started, please review the details here:

https://docs.bmc.com/docs/TSOperations/113/amigo-checklist-for-truesight-operations-management-814553031.html

 

Then open a BMC Support issue containing your environment information (product, version, OS, etc.) and the planned date of the installation, if known. We will contact you promptly, and work with you to ensure a successful and timely outcome.

 

Computer.png

 

 

New TrueSight knowledge articles added to the BMC Knowledge Base over the last month:

 

000171371 The -t (timeout) option of mcstat command is not working as expected

 

000171481  Cell blackout policy is reopening event that it did not blackout

 

000171487  Is there a Float option in BMC TrueSight 11.3.x?

 

000171551  Is there a way to create a dynamic event filter in TSPS to show the current User's Assigned Events?


000171914  How to delete an entry from 'tssh properties'?

 

000172006  Secondary TSPS is missing devices and events

 

000172048  Graph Display in TSIM 10.7 FP3 does not show updated thresholds correctly


000172168  TrueSight Presentation Server(TSPS) - is not impacted by 2019-082 Apache Struts security vulnerabilities


000172277 "Infrastructure Management Servers cannot be accessed" seen when viewing TSPS console -> Monitoring -> Events -> Showing events from all Infrastructure Management Servers

 

000172393  How to report the ciphers used for TrueSight Presentation Server and Infrastructure Server