TSI - Service Model (Digital Services) Based Integration with TrueSight Operations Manager or BPPM

Version 3
    Share:|

    Update 4-29: Added Support for TSIM or BPPM servers with Oracle backend. See README attached.

     

    Summary of Integration:

     

    This is a BMC Communities provided and supported integration for TrueSight Intelligence (TSI). The purpose of this integration is to dynamically populate TSI with Metrics (Business and Performance) from TrueSight Infrastructure Manager (TSIM) or BMC ProactiveNet Performance Manager (BPPM) from a Digital Service perspective. Service Models created in TSIM/BPPM (regardless of source: Atrium CMDB, Direct Publishing or manually created) will be the foundation for device and application creation in TSI. TrueSight or BPPM events are also populated on a minute interval within context to devices that are "providers" to a Business Service.

     

    The value of this integration would be the following:

     

        • Digital Services – Service Models created in TSIM/BPPM become the base definition of the Applications in TSI. Regardless if a customer has 1 or +1000 applications, the integration will map all devices from a Service Model to TSI. Accelerate Value Realization of TSI.
        • Analytics – provide an easy and simple mechanism for Application Owners or Line of Business individuals to review, add or remove performance/business metrics which would be synched to TSI for various Analytical Use Cases. Improve efficiency with metric and application definition within TSI.
        • Event Management – Provide visibility of existing or historical faults and/or performance degradations within context to Digital Services.
        • No Scripting/Coding Required - Provide an easy method for Application Owners or Line of Business Analysts to start Analytics Use Cases without needing to script in languages such as Python. All the configuration can be managed in a multi-tabbed spreadsheet without needing to write a single line of code.

     

    This integration is divided into three parts:

     

        1. Metric Creation
        2. Measures
        3. Events

     

    Business Metrics:

     

    TSI Use Cases will have a heavy emphasis on Business Metrics (e.g. Number of Credit Cards processed, Closed Sales Opportunities at the end of the month, ...), which may not be collected by TSOM or BPPM -- therefore, look for ETLs to be posted on BMC Communities (and GitHub) to help facilitate data collection in the short-term from sources like SalesForce and Google Analytics.

     

    End State:

     

    The current version of the integration requires Service Models to be present within TSIM or BPPM for metrics to be dynamically associated with TSI Applications. Click to enlarge the image below:

     

    TSIM_TSI_Sync.png

     

    The final Result in TSI will be Applications created from Business Services, and all the sources (Virtual Machines, Hypervisors, SAN, network, ..) will be mapped dynamically from Provider (BMC_ComputerSystem) CIs in the Service Model. Click to enlarge the image:

     

    TSIM_TSI_Sync2a.png

     

    TrueSight Devices Synched - click to enlarge the image:

     

    TSIM_TSI_Sync3.png

    Note: If you do not currently have Service Models defined in TSIM or BPPM, then consider looking at the following BMC Communities Posts to quickly create a large number of Service Models with very little effort using the vSphere and other KMs as a discovery source:

     

    Automatically Create Virtual Topology using VSM KM and Direct Publishing with Simple Service Model Support - LINUX versi…

    Automatically Create Virtual Topology using VSM KM and Direct Publishing with Simple Service Model Support - WINDOWS version

     

    Who is this integration intended for?

     

    This is BMC Communities provided and supported integration is intended for Application Owners or Line of Buisness Analysts who want to quickly push IT Operations related data to TSI with no scripting or coding.

     

    So let's take a different approach to getting to an end-state ....

    vader-and-son.png

    Metric Creation

     

    This ETL can be executed manually or scheduled on either a daily or weekly basis. The schedule is dependent on how often new KPIs (e.g. Patrol KMs) are created in TSIM/BPPM or additions/changes in Digital Services (Service Models).

     

    The following is the data flow for Metric creation in TSI:

     

    TSIM to TSI Metric flow1.png

     

    The ETL will query the TSIM/BPPM service cell for a list of Business Services and monitor metrics using a combination of an out of the box API (mquery) and web services (REST API). KPI Metrics are dynamically created in TSI with the naming convention: TSIM..<Monitor_Name>..<parameter>. Finally, the ETL will write to three tabs within an MS Excel spreadsheet named “Simple_TSOM_to_TSI_Integration.xlsx”.

     

    Application Owners or Line of Business Analysts can review the defined TSI Metrics and add or remove any automatically selected metrics. Once the review is completed, the spreadsheet will be placed in a working directory (e.g. SharePoint, local directory …) where the file will be referenced in the second ETL (Measures ETL).

     

    Sample Formatted JSON Data by ETL:

     

    {

       "name":"TSIM..VMware_CPU_Container..CPU_Total_Util",

       "description":"VMware CPU Container - CPU_Total_Util",

       "displayName":"VMware CPU Container - CPU_Total_Util in %",

       "displayNameShort":"TSOM_CPU_Total_Util",

       "unit":"percent",

       "defaultAggregate":"avg",

       "defaultResolutionMS":"300000",

       "type":"TrueSight Device"

    }

     

    Measures

     

    This ETL requires being scheduled to run on a 1 or 5-minute interval using Windows Task Scheduler, Cron or other scheduling tools (e.g. Control-M). The schedule/interval is dependent on how often data parameters (values) are updated or streamed to TSIM or BPPM. The suggestion is to schedule this Measures ETL every 5 minutes.

     

    The following is the data flow for Measure creation in TSI:

     

    TSIM to TSI flow2 Measures.png

     

    This ETL will use the spreadsheet as a reference for to determine which metrics will be updated with measures. The measures include device and digital service association.

     

    Sample Formatted JSON Data by ETL:

     

    {

       "source":"172.21.97.150",

       "metric":"TSIM..VMware_CPU_Container..CPU_Total_Util",

       "measure":0.0455,

       "timestamp":1461868916,

       "metadata":{

          "app_id":"CBNA",

          "source_type":"ESXi"

       }

    }

     

    Events

     

    This ETL requires being scheduled to run on a 1-minute interval using Windows Task Scheduler, Cron or other scheduling tools (e.g. Control-M). ETL will synchronize any events associated with devices that are "providers" to the Buisness Services (Service Models). Events are "state-based" and only deltas are synched to TSI. When events are closed in TSOM or BPPM, then the events are closed in TSI.

     

    The following is the data flow for Event creation in TSI:

     

    TSIM to TSI flow Event.png

    Sample Formatted JSON Data by ETL:

     

    {

       "title":"Process IBS_Coll_Sync is above all thresholds. Impact may occur to IBS Collections.",

       "fingerprintFields":[

          "@message"

       ],

       "message":"mc.pncell_tsim1.171fe95a.0",

       "eventClass":"EVENT",

       "status":"CLOSED",

       "severity":"MAJOR",

       "properties":{

          "app_id":"IBS",

          "hostIp":"192.168.0.102"

       },

       "source":{

          "ref":"sme-itsm1",

          "type":"sme-itsm1",

          "name":"sme-itsm1"

       }

    }

     

    3 Simple Steps to the integration:  Easy as 1,2,3

     

    • Step 1 – Add a baroc file to TSIM or BPPM cell (custom data table) and recompile/re-start cell. Then run or schedule a provided perl script to "walk" the tree of the Service Models and populates the data table with business service associations.
    • Step 2 Download/Install JRE (if one does not exist) and Download/Extract PDI 6.1 (or later) on a Windows Server that has access to TSIM/BPPM and Internet connectivity to TSI. Configure ETL Parameters for 3 Jobs (Metric Creation, Measures, Event)
    • Step 3 – Schedule the ETLs with Windows Task Scheduler or another scheduling tool. 

     

    FAQs (Frequently Asked Questions):

    Start first with reading the attached documentation. Here are FAQs

     

    1.What is the ETL engine? Do I need to purchase any additional software?

    Pentaho Data Integration Communities Edition is free of charge and is used as the ETL engine to extract, transform and load data. Download PDI from Sourceforge: (Download 6.1 or later)

    http://sourceforge.net/projects/pentaho/files/Data%20Integration/

     

    2. How does the ETL connect to BPPM or TSIM?

    Based on the 3 ETLs, there is a combination of connections which include:

     

        • REST API
        • Database (SQL) Query
        • MQUERY (TSOM/BPPM API)

     

    Rest API Credentials:

     

    All three ETLs leverage the REST API step to either query TSIM/BPPM or push data (Measures) or events to TSI. The default credentials used for TSIM/BPPM is admin/admin12345. If need to change the web service credentials, then you will need to open the following transformations and edit the REST API step targeted for TSIM/BPPM:

     

        • Metric Creation ETL:  TSI_Metric_Creation_REST_API_from_BPPM_TSIM_Step1.ktr (update TSIM and TSI security token)
        • Metric Creation ETL: TSI_Metric_Creation_REST_API_from_BPPM_TSIM_Step2.ktr (update TSI security token only)
        • Metric Creation ETL: TSI_Metric_Creation_REST_API_from_BPPM_TSIM_Step3.ktr (update TSI security token only)
        • Measures ETL:  TSI_Measures_REST_API_from_BPPM_TSIM_Step2.ktr (update TSIM and TSI security token)
        • Event ETL: TSPS_Event_Dashboard_Event_ETL_Step1.ktr (update TSI security token only)

     

    Database Connection:

     

    Only the Metric Creation ETL requires to connect to the database. The integration supports Oracle as well - there is a zip file named "TSI_Metric_ETL_Step_for_TSIM_BPPM_with_Oracle_backend.zip" which contains a transformation file which you can replace. Again, in the Event Creation ETL (which may run once a day or a week) will connect to the embedded Sybase database warehouse database using JDBC (SQL query). The integration assumes the dba user will be used since Tables (not views) are queried. Attached in the integration is the jconn4.jar which you will need to copy to the {Pentaho_Home}/data-integration/lib directory. Configuration for the integration is done with parameters (parameters tab) on the job named "TSI_Metric_Creation_REST_API_from_BPPM_TSIM.kjb".

     

    If the DB username/password to connect is required to changed (default configured is dba/pronet), then you will need to edit/configure the Input Step for the following transformation and save the transformation: (you can edit step and replace the credentials)

     

        • Metric Creation ETL:  TSI_Metric_Creation_REST_API_from_BPPM_TSIM_Step1.ktr

     

    3.How does the ETL handle changes (deltas)?

    There are 2 different changes that can occur. First, if your service model gets updated (meaning added/removed CIs) then you will need to re-run the provided perl script:

     

    ..\TrueSight\pw\server\bin>perl dashboard_rels.pl -s pncell_tsim1 -m

    Note: pncell_tsim1 = cell name of the TSIM or BPPM server

     

    Once complete, then the Measures ETL will dynamically pick up the Service Model changes.

     

    The other changes is related to events. The Event ETL is configured to be state-based and only new or changed events from the last time the ETL was run (default every 1 min) will process only deltas.

     

    4.How can you schedule the ETL to run?

    The ETL can be run from the command line, so use CRON or some scheduler to execute a sample script. Here are sample bat scripts (Windows). Included in this integration are the Windows Scheduled Task files which you can import and modify file location. Modify the examples below with correct location of PDI and ETL files.

     

    Metric Creation ETL: (Suggested schedule:  once a day or once a week)

     

    @ECHO OFF

    C:\bmc_pentaho\data-integration\kitchen.bat /file:C:\Data\metric\TSI_Metric_Creation_REST_API_from_BPPM_TSIM.kjb

    exit

     

    Measures ETL: (Suggested schedule:  every 5 minutes)

     

    @ECHO OFF

    C:\bmc_pentaho\data-integration\kitchen.bat /file:C:\Data\measure\TSI_Measures_REST_API_from_BPPM_TSIM.kjb

    exit

     

    Event ETL: (Suggested schedule:  every 1 or 5 minutes)

     

    @ECHO OFF

    C:\bmc_pentaho\data-integration\kitchen.bat /file:C:\Data\event\TSI_Event_Creation_REST_API_from_BPPM_TSIM.kjb

    exit

     

    4.How does an Application Owner or Analyst select and modify which metrics (measures) will be updated in TSI?

    The concept of the integration was to simplify the integration process. No scripting is needed. The first ETL (Metric Creation ETL) will generate a multi-tabbed XLSX named "Simple_TSOM_to_TSI_Integration.xlsx". Copy the xlsx (or it can be automated to be emailed to a distribution list) locally and edit with MS Excel. The tab "5.Final_Measures" is the list of metrics per device that will be updated in TSI.

     

    Suggestion is to  Insert --> Table, then you can easily filter which metrics you want or remove. Simply delete the rows that are not wanted. See below - click to enlarge the screenshot:

    XLSX.png

    Not all metrics registered in TSI are selected by default for measures. The tab named "4.TSI_Measures" is the Master list, and if you want to add metrics not appearing in tab "5.Final_Measures".

     

    5.How can I filter which Service Models are synched to TSI?

    Right now, control that in the spreadsheet (see FAQ#4). Once of the Columns that you can filter on is app_id. Simple filter in the XLSX on the models that you do not want, then delete the rows from that tab. See below for an example - click to enlarge the screenshot:

    XLSX2.png

     

    Integration Requirements:

    The following are required for this integration:

    • Pentaho PDI 6.1 or later - Kettle (Open Source ETL)
    • Java SE (JRE) – no specific version required
    • BPPM +9.5,  TSIM +10
    • Embedded Sybase Database
    • Service Models – can be created via Atrium CMDB, Direct Publishing or Manual Creation
    • *Oracle is supported by replacing the following transformationTSI_Metric_Creation_REST_API_from_BPPM_TSIM_Step1.ktr which is included in the zip below named "TSI_Metric_ETL_Step_for_TSIM_BPPM_with_Oracle_backend.zip". Also read the Readme document for 2 steps.

     

    Integration LoEThis base integration will take about ~30 minutes to setup. (assuming Service Models exist in TSIM or BPPM)

     

    easy-button.png

    Known Limitations:

    The following are known limitations with this version:

    • No spaces allowed in the full path for PDI or ETL file location
    • Testing performance - unknown upper limit of metrics that can be processed every 5 minutes. Right now I am able to easy push 180 measures every 5 minutes. Reformatting the JSON body to support bulk will improve performance.

     

    Integration Feedback:

    What do you think? Please provide any feedback on the comments section below. Also, if you have any enhancements or ideas that you can contribute or list, that would be appreciated. Enjoy and keep Communities a sharing environment!

     

    What's Next?

    Next version/update - working on:

    • improving performance (bulk updates to JSON body)
    • Map BMC_Application CIs (not just BMC_BuisnessService) to TSI application (app_id)
    • Map TSPS Application Context Models to TSI application (this would only support TSOM v10.x)
    • Working on improving version control of XLSX ... using Google Spreadsheet
    • also soon to come out soon ... ETLs for Sales Force, Remedy Force and Google Analytics to capture Business Metrics.