Share This:

The Script is designed to Monitor RSSO, Repository, Grid Manager and Peers for critical attributes likes Available Memory, Used Memory, Total Heap Memory, JVM Uptime, Active Threads, Management Grid Status, Main Grid Status, CDP Grid Components Status (Health, Library Manager, Configuration, Context, Scheduler, Metrics, Job Manager, Adapter Manager, Activity Processor) and all active adapter status active on all peers in the Grid. It also tracks Running, Compensated and Completed Jobs Status of all peers, In case of script find any issue in the grid it will trigger a notification based on the notification method enabled in the Config.xml

 

Enhancements in v3.0

  • Create an Incident ticket in the ITSM tool (Remedy or ServiceNow) to report the issue.
  • Create Event in True Sight Operations Management (TSOM) to report the issue.
  • Monitor Running, Compensated and Completed Job Count in Grid and all peers and notifications will be a trigger for the threshold limit of running jobs.
  • Monitor Embedded and External RSSO Status.

Fix

  • Deprecated library message comes while reading values from Config.xml has been fixed.
  • The use of python minidom library to read XML has been removed as its support in a future version of Python is coming to end.
  • InsecureRequestWarning message issue has been fixed.
  • Logging improved in the console and in logs.

 

Note: Script leverages BAO/TSO REST API and Python web scraping techniques to collect data from RSSO, REPO, Grid Manager and Peers.

 

Which version of BAO/TSO does the script support?

The new version of Script is designed to support TSO 8.x and above including recently release TSO 20.02.00.

 

Which Notification methods does the script support?

The new version of Script is designed to support the below methods to report issues.

  • Email Notifications
  • Remedy Incident Creation
  • SNOW Incident Creation
  • Event Creation in TSOM

 

System Requirements

  • Python 3.6.6 or above.
  • Python Requests Module (Command to install: - pip install requests)
  • For Windows executable file is also available which doesn’t require installation of Python
  • For Linux installation of Python 3.6.6 or above is mandatory along with the installation of request modules.
  • All configurable parameters of the script/tool is controlled by "Config.xml" which should be kept in the same folder as the script.
  • For Remedy Incident creation, script use REST API which needs to be enabled in ARS.
  • For TSOM Event creation and Service Now Incident creation script use REST API.

 

 

Steps to Configure the Script

Getting the Grid ID from your Grid Manager, log in to the Grid Manager and go to Manage>Grids>Select your Grid Name and click on pencil sign for edit. copy the ID and Grid Name as shown below and past it in "Config.xml"

Provide your CDP server IP/FQDN and port details along with BAO/TSO admin account in the "Config.xml", also in "schedule-interval" provide the time interval in secs for the script to check Grid Status, default is set to 270 sec i.e. 4 mins 30 sec if you are using “report Compensated Job” please keep the "schedule-interval" to 4 mins 30 sec only as the Jobs concluded status in kept in the grid only for 5 mins.

 

 

XML TagDescription and excepted value
server
CDP Server Name or FQDN
portCDP Port
schedule-intervalThe interval at which script run (Suggested value is 270 sec i.e. 4 mins 30 sec)
enable-encryptionIf you want to use password encryption set the value as "True" otherwise default value needs to be set as "False"
userAO Admin Username
passwordAO admin password in plain text.
encrypted-passwordEncrypted password string from the encryption tool
GridIDGrid ID which can be collected from Grid Manager console as shown in the above screenshot
GridNameGrid Name of your BAO Environment
AllowInsecureRequestIf you are using self-signed or insecure certificate set this as "True" otherwise set it as "False"
print-onscreenIf you want the script out to be printed on the command prompt or console set this as "True" however if you are running the script in background set it as "False". although in both, the case output will be collected available in logs for reference.
console-loggingIf it is set as "True", INFO, DEBUG or ERROR logging will be also displayed on the console.
monitor-peers

If you want to monitor other peers in the Grid like HACDP, AP's and LAP set this to "True" otherwise set it as "False"

Also if monitor-peer is True you will have to provide peer information as shown below make sure to include protocol like HTTP or HTTPS in value along with the port which peer is utilizing.

 

<peers peerType="HACDP">

        <peer>https://baohacdp.bmc.com:38080</peer>

    </peers>

<peers peerType="AP">

        <peer>https://baoap1.bmc.com:48080</peer>

        <peer>https://baoap2.bmc.com:48080</peer>

</peers>

<peers peerType="LAP">

        <peer>https://baolap1.bmc.com:48080</peer>

</peers>

Threshold-limit

If the threshold limit is hit the script will sending a notification to inform about the same using the notification method enabled.

example:

 

<Threshold-limit>
   <memory>
200</memory>
   <threads>500</threads>
   <runningJob>256</runningJob>
</Threshold-limit>

 

In the above example

if free memory drops below 200 MB a notification will be triggered.

if threads in the BAO cross 500 threads limit a notification will be triggered.

if runningJob are above 256 job limit a notification will be triggered.

repoUrl

URL of the repository

example

<repoUrl>https://baorepo.bmc.com:28080/baorepo</repoUrl>

rsso

URL of RSSO, depending upon whether it was embedded RSSO or External.

example:

<rsso type="embedded">
   <url>https://baorepo.bmc.com:28080/rsso</url>
   <url>https://baocdp.bmc.com:38080/rsso</url>
</rsso>

Or

<rsso type="external">
   <url>https://rsso1.bmc.com:28080/rsso</url>
   <url>https://rsso2.bmc.com:38080/rsso
</url>
</rsso>

report-compensatedJobTo report compensated Job set it as "True"
logging-levelTo Set the Logging level for script default is INFO and other excepted value is DEBUG
Notification

Under the notification tag, different methods of notification are available as

  • Email Notifications
  • Remedy Incident Creation
  • SNOW Incident Creation
  • Event Creation in TSOM

 

Print on Screen and Loggings.

If Print on screens is enabled in Config.xml the script will print status as shown below for all TSO/BAO components. An authentication token will be acquired only once till the time it expires.

The below screenshot shows the status of the RSSO, REPO, GRID along Jobs status in the Grid.

Below screenshot shows the Activity Peer Status along with its respective components and adapters status

Below screenshot show status of CDP components and adapter along with Incident Number which was created to notify one adapter which is in fault state.

If print on screen is turned off, then the details logging are collected in the TSOMonitoring.log depending upon logging level set on Config.xml. In this case, the script can be run in the background.

 

·        Notification Method – Email Setting

 

To turn on email notification set the Tag “enable-email-notification” to “True” and configure the parameter of the EmailSetting as below.

 

Note: For encrypting password uses the encryption tool.

 

------------------------------------------------------

<EmailSetting>
    <SMTPServer>
{SMTP Server Name or FQDN}</SMTPServer>
    <SMTPPort>
{Port- default is 587} </SMTPPort>
    <secure>
{True or False}</secure>

    <starttls>{True or False}</starttls>  

    <EmailUsername>{ Sender Email like - sender@test.local }</EmailUsername>
    <encrypted-password>
{Encrypted Password - V1ZkU2RHRlhOSGhOYWswOQ== }</encrypted-password>
    <NotificationEmailAddress>
receiver@test.local</NotificationEmailAddress>
    <Subject>
BAO/TSO Monitoring tool reported error for monitored Grid</Subject>
    <Message>
Error details are mentioned below</Message>
    <enable-email-notification>
{True or False}</enable-email-notification>
</EmailSetting>

 

----------------------------------------------------

Screenshot on left show notification of Adapter in fault state whereas screenshot on the right show notification compensated job.

 

Notification Method – Remedy Incident Creation

 

To enable remedy incident creation set “enable-remedy-incident-creation” to “True” and configure the parameter of the ARS-Config as below.

 

Note:

For encrypting password uses the encryption tool.

Since the script utilizes Remedy REST API it is compatible with ARS 9.1 and above which has Rest API enabled.

---------------------------------------------------------------

<ARS-Config>
           <ARServer>
{AR Server FQDN or HOSTNAME}</ARServer>
           <ARSAPIPort>
{Port- 8008}</ARSAPIPort>
           <ARUsername>
{ AR Username }</ARUsername>
           <password>
{ Password in Plain Text }</password>
           <encrypted-password>
{ Encrypted Password using encryption tool} </encrypted-password>
           <secure>
{True or False}</secure>
           <first-name>
{First Name of the user from which Incident will be created}</first-name>
           <last-name>
(Last Name of the user from which Incident will be created} </last-name>
           <impact>
1-Extensive/Widespread</impact>
           <urgency>
1-Critical</urgency>
           <default-incident-status>
New</default-incident-status>
           <short-description>
BAO/TSO Monitoring tool reported error for monitored Grid</short-description>
           <description>
Error details are mentioned below</description>
           <enable-remedy-incident-creation>
{True or False}</enable-remedy-incident-creation>
</ARS-Config>

-------------------------------------------------------------

Below screenshot show notification of Adapter in a fault state and a screen of incident created for compensated job

 

·        Notification Method – SNOW Incident Creation

 

To enable SNOW Incident creation set the “enable-snow-incident-creation” to “True”  and configure the parameter of the SNOW-Config as below.

 

Note: For encrypting password uses the encryption tool.

------------------------------------------------------------------------

<SNOW-Config>
     <snow-host>
{Service Now FQDN like test.service-now.com}</snow-host>
     <snow-port>
{Port -443}</snow-port>
     <snow-username>
{Username of SNOW a/c}</snow-username>
     <password>
{plain text password}</password>
     <encrypted-password>
{Encrypted password}</encrypted-password>
     <secure>
{True or False}</secure>
     <impact>
1</impact>
     <urgency>
1</urgency>
     <category>
Software</category>
     <subcategory></subcategory>
     <caller-id>
486e3a122fd74010d72657ab2799b635</caller-id>
     <default-incident-status>
1</default-incident-status>
     <snow-short-description>
BAO/TSO Monitoring script reported error for monitored Grid</snow-short-description>
     <snow-description>
Error details are mentioned below</snow-description>
     <enable-snow-incident-creation>
{True or False}</enable-snow-incident-creation>
</SNOW-Config>

----------------------------------------------------------------------------

To get the caller-id of the user, open user profile in SNOW portal, under additional actions, click copy sys_id, and past in Config.xml as value for <called_id> tag.

The below screenshot shows a sample incident created to report the Adapter in Fault state.

 

Notification Method – TSOM Event Creation

 

To enable TSOM Event creation set the “enable-event-creation” to “True” and configure the parameter of the TSOM-Config as below.

 

Note:

For encrypting password uses the encryption tool.

If Routing is required you can set the CELLNAME else leave it blank and Event will be routed to default Cell.

------------------------------------------------------------------------

<TSOM-Config>
     <tsps-host>
{TrueSight Presentation Server Hostname}</tsps-host>
     <tsim-host>
{TrueSight Infrastructure Manager hostname}</tsim-host>
     <cell-name>
{CELLNAME}</cell-name>
     <userid>
{username}</userid>
     <password>
{password}</password>
     <encrypted-password>
{ Encrypted Password using encryption tool}</encrypted-password>
     <secure>
{True or False}</secure>
     <event-title>
BAO/TSO Monitoring tool reported error for monitored Grid</event-title>
     <message>
Error details are mentioned below</message>
     <enable-event-creation>
{True or False}</enable-event-creation>
</TSOM-Config>

-------------------------------------------------------------------------

Below the screenshot shows a sample event for adapter in a fault state.

 

The below screenshot shows a sample event for compensated Job

 

 

Steps to Encrypt or Decrypt passwords

 

To encrypt the password in Windows run the “passwordtool.exe” and For Linux use “passwordtool.py” provide the password string as shown below along with operation as "encrypt" and copy the encrypted output to the Config.xml file for respective encrypted password tag.

Example for Windows: passwordtool.exe -p P@ssw0rd -o encrypt

Example for Linux: passwordtool.py -p P@ssw0rd -o encrypt

 

 

·         Sample Config.xml

 

<ConnectionDetail>
   <server>baocdp.bmc.com</server>
   <port>38080</port>
   <schedule-interval>270</schedule-interval>
   <enable-encryption>True</enable-encryption>
   <user>aoadmin</user>
   <encrypted-password>V1ZkU2RHRlhOSGhOYWswOQ==</encrypted-password>
   <GridID>urn:jxta:uuid-E5D1AB3B3CB94AA48A93A7E2BE2814A702</GridID>
   <GridName>MyGrid</GridName>
   <AllowInsecureRequest>True</AllowInsecureRequest>
   <print-onscreen>True</print-onscreen>
   <monitor-peers>True</monitor-peers>
<!--    <peers peerType="HACDP">
        <peer></peer>
    </peers>-->
   <peers peerType="AP">
        <peer>https://baoap1.bmc.com:48080</peer>
   </peers>
<!--    <peers peerType="LAP">
        <peer></peer>
    </peers> -->
   <Threshold-limit>
     <memory>200</memory>
     <threads>500</threads>
     <runningJob>256</runningJob>
   </Threshold-limit>
   <repoUrl>https://baorepo.bmc.com:28080/baorepo</repoUrl>
   <rsso type="embedded">
     <url>https://baorepo.bmc.com:28080/rsso</url>
     <url>https://baocdp.bmc.com:38080/rsso</url>
   </rsso>
   <report-compensatedJob>True</report-compensatedJob>
   <logging-level>DEBUG</logging-level>
   <console-logging>False</console-logging>
   <Notification>
        <EmailSetting>
           <SMTPServer>localhost</SMTPServer>
           <SMTPPort>587</SMTPPort>
           <secure>False</secure>
           <starttls>False</starttls>
           <EmailUsername>noreply@test.local</EmailUsername>
           <password>admin123</password>
           <encrypted-password>V1ZkU2RHRlhOSGhOYWswOQ==</encrypted-password>
           <NotificationEmailAddress>support@test.local</NotificationEmailAddress>
           <Subject>BAO/TSO Monitoring tool reported error for monitored Grid</Subject>
           <Message>Error details are mentioned below</Message>
           <enable-email-notification>False</enable-email-notification>
        </EmailSetting>
          <ARS-Config>
            <ARServer>arserver.bmc.com</ARServer>
            <ARSAPIPort>8008</ARSAPIPort>
            <ARUsername>appadmin</ARUsername>
            <password>appadmin</password>
            <encrypted-password>V1ZoQ2QxbFhVblJoVnpROQ==</encrypted-password>
            <secure>False</secure>
            <first-name>Allen</first-name>
            <last-name>Allbrook</last-name>
            <impact>1-Extensive/Widespread</impact>
            <urgency>1-Critical</urgency>
            <default-incident-status>New</default-incident-status>
            <short-description>BAO/TSO Monitoring tool reported error for monitored Grid</short-description>
            <description>Error details are mentioned below</description>
            <enable-remedy-incident-creation>True</enable-remedy-incident-creation>
          </ARS-Config>
        <TSOM-Config>
            <tsps-host>tsps.bmc.com</tsps-host>
            <tsim-host>tsim.bmc.com</tsim-host>
            <cell-name></cell-name>
            <userid>aoadmin</userid>
            <password>P@ssw0rd</password>
            <encrypted-password>VlVWQ2VtTXpZM2RqYlZFOQ==</encrypted-password>
            <secure>True</secure>
            <event-title>BAO/TSO Monitoring tool reported error for monitored Grid</event-title>
            <message>Error details are mentioned below</message>
            <enable-event-creation>False</enable-event-creation>
        </TSOM-Config>
        <SNOW-Config>
            <snow-host>testinstance.service-now.com</snow-host>
            <snow-port>443</snow-port>
            <snow-username>aoadmin</snow-username>
            <password>admin123</password>
            <encrypted-password>V1ZkU2RHRlhOSGhOYWswOQ==</encrypted-password>
            <secure>True</secure>
            <impact>1</impact>
            <urgency>1</urgency>
            <category>Software</category>
            <subcategory></subcategory>
            <caller-id>486e3a122fd74010d72657ab2799b635</caller-id>
            <default-incident-status>1</default-incident-status>
            <snow-short-description>BAO/TSO Monitoring script reported error for monitored Grid</snow-short-description>
            <snow-description>Error details are mentioned below</snow-description>
            <enable-snow-incident-creation>False</enable-snow-incident-creation>
        </SNOW-Config>
    </Notification>
</ConnectionDetail>

 

Steps to Run or Deploy the script or executable.

 

In Windows with .exe: After Configuring the "Config.xml" file double click on executable to run directly or run it from the command prompt.

Example: Open Cmd> BAOMonitoring_v3.0.exe

 

In Windows from .py: After Configuring the "Config.xml" file, Open command prompt and Type "python" then space followed by the script file name.

Example: Open CMD>python BAOMonitoring_v3.0.py

 

To Stop the script press CTRL+C

 

To run in background type pythonw then space followed by the script file name.

Example: Open CMD>python BAOMonitoring_v3.0.py

 

To stop the script which is running in the background you have to find task-id of pythonw and kill it using taskkill command.

 

In Linux from .py: After Configuring the "Config.xml" file, from terminal type python3 then space followed by the script file name.

Example: Terminal>python3 BAOMonitoring_v3.0.py

To Stop the script press CTRL+C

 

To run in background type python3 then space followed by script file name, space followed by "&"

example of running in the background

Example: Terminal>python3 BAOMonitoring_v3.0.py &

To stop the script which is running in the background you must find the process id of python and kill it using kill -9 command.

 

PLEASE NOTE: THIS IS NOT AN BMC OFFICIAL SCRIPT OR TOOL FOR MONITORING BAO/TSO, BMC SUPPORT WILL NOT BE ABLE TO PROVIDE YOU ANY ASSISTANCE FOR THE SAME.