TSIM/BPPM - Examples of Event Enrichment using REST API and VMWare PowerCLI

Version 5
    Share:|

    Summary of Event Enrichment:


    This is a BMC Communities provided and supported example of configuring get_external which is a function (rule) which can run an external program and wait for its termination to continue to process the current event, using data retrieved through an interface object. The purpose of this posting is to show examples of how to use the get_external function within a cell (e.g. pncell, service or any remote event cell) to enrich events by running external an ETL (Extract, Transform and Load) using Pentaho's Communities Edition (PDI - OpenSource ETL).


    The value for enrichment would be the following:


    • Event Management - real-time event enrichment with "meta" data of from various sources to assist Operators with triage and resolution process
    • Improve Incident Management process by passing helpful data to the ticket summary or logs
    • Enhance notifications with additional context


    In this posting, I show how to leverage an open-source ETL tool (Pentaho Communities Edition) to complete the following scenarios:


    1. Query a source using REST API (GET) and enrich/refine mc_long_msg slot with results from the query
    2. Query vCenter via VMWare PowerCLI (e.g. Get-Vm) and enrich/refine mc_long_msg with results from the query

     

    Here is a high-level event flow:

     

    Get External Event Flowv2.png

    Note: It is important to note that events are held in the REFINE process stage of the cell till the get_external execution has completed. Therefore, you must be careful and choose a subset of events that have a low volume of events and where enrichment would benefit a business process (e.g. Incident Management) or specific Use Case or scenario. Also, it is recommended to use this function at an enrichment cell level versus normalization (de-duplication) layer of an architecture. Again, be careful with high-volume events!

     

    Let's take a different approach to getting doing Enrichment ...

    vadar_and_son_drive.png

     

    4 Steps to Enrichment:


    Step 1 – Create MC_INTERFACE

                    Define slots of the MC_INTERFACE in a .baroc file (\etc\<cell_name>\kb\classes directory)

      Step 2 – Create a rule (MRL)

                Define the criteria when the get_external enrichment would be met. Be careful and limit the "scope" of the where clause.

      Step 3 – Create a script or batch file to be executed

    In this posting, I have a simple batch file which executes a PDI (Pentaho Communities Edition) Job by calling a utility named kitchen. Slot values from an event are dynamically passed into the ETL for execution.

      Step 4 – (*Optional) Create PDI Transformation(s)

    This is an optional step and could be replaced with shell or perl scripts. If you are not an experienced scripter, PDI maybe an easy way to accomplish many requirements without coding or scripting. For more information on PDI, please goto Community Edition Downloads - Pentaho Community - Pentaho Wiki


    2 Scenarios:


    Use these scenarios as examples and how other Use Cases maybe met with a similiar approach. Again, this posting focuses on using Pentaho, but if you are a strong scripter (e.g. perl) then you can write your own scripts to replace the ETL. The advantage of using Pentaho is that is requires no coding experience and is very easy to pickup, install and configure. Deploying PDI is as easy as unzipping files on Windows or Linux and you are ready to go!

     

    Scenario 1: Enrich with REST API


    This scenario is really to show you that you can query any source via REST API. On requirement is to download PDI (SourceForge) onto a virtual server running one or more cells. In PDI, there is a REST API Client step and a JSON input step which make it very easy to configure queries. In this scenario, the rule executes when a PATROL_EV  OPEN (status) event with mc_object = '"VmMigrationsPerDay" arrives in the cell. In the refine stage, get_external will execute a batch file named "get_rest_api_configuration" and pass values from two slots:  mc_host and mc_object.

    images.jpg

    Let's start from the end, here is the TSIM Operator Console with an enriched event:*Here I am just using msend to inject a simple event:

     

    msend -n pncell_ts-im -a PATROL_EV -r MAJOR -m "This is a sample PATROL Event that will trigger Get_External" -b "mc_host=host38;mc_object=VmMigrationsPerDay;mc_smc_alias='BMC_ComputerSystem:host38"

     

    pix1.png

     

    Event Details: Detailed Message (mc_long_msg) is enriched with TSIM monitoring details and a full REST API url to extract last 24 hrs of data. Note - there is a cost to this type of enrichment. The get_external enrichment took ~5 seconds to complete.

    pix2.png

    Here is the Exported event (baroc format):

    ...

    mc_long_msg='TrueSight Monitor Details:

        monitorTypeName = VMware VM VMotion

        monitorInstanceName = VMotion

        attributeName = VmMigrationsPerDay

        REST URL = http://172.21.86.142/bppmws/api/Device/host38/stats?starttime=2015-10-04T19:16:26&endtime=2015-10-05T19:16:26&idType=Name&statstype=aggregate&attributetype=kpi&monitor=VMware%20VM%20VMotion:VmMigrationsPerDay';

    ...

    *Details on the steps are outline in the APPENDIX sections below. Notice that in the mc_long_msg I have carriage returns after values - I added char(10) after each value in the MRL.


    Scenario 2: Enrich with VMWare PowerCLI


    This scenario is really to show you that you can query vCenter using the VMWare PowerCLI to query and even automate tasks. VMWare PowerCLI will need to be downloaded and installed on a Virtual Machine where remote cell(s) are running. This example could be expanded to other Element Managers or Solutions who have vendor supported APIs.

    powercli.png

     

    Let's start from the end, here is the TSIM Operator Console with an enriched event:*Here I am just using msend to inject a simple event that a virtual machine is not responding to a PING and is determined to be unavailable by the monitoring solution:

     

    msend -n pncell_ts-im -a EVENT -r MAJOR -m "Device dc-docker is not responding to Ping" -b"mc_host=dc-docker;mc_object=PING;mc_smc_alias=BMC_ComputerSystem:dc-docker"

    pix7.png

    Event Details: Detailed Message (mc_long_msg) is enriched with Virtual Machine details from vCenter. Note - there is a cost to this type of enrichment. The get_external enrichment took <10 seconds to complete.

    pix8.png

    Here is the Exported event (baroc format):

    ...

    mc_long_msg='vCenter Details:

    Name:  DC-Docker

    PowerState:  PoweredOn

    Guest:  DC-Docker:

    NumCpu:  4

    MemoryGB:6

    Host:  dcesx3.dclab.bmc.com

    Disk Utilization (%):  15.73

    ResourcePool:  Resources

    UsedSpaceGB:  18.34

    ProvisionedSpaceGB:  116.56;

    ...

    Details on the steps are outline in the APPENDIX sections below. Notice that in the mc_long_msg I have carriage returns after values - I added char(10) after each value in the MRL.

     

    FAQs (Frequently Asked Questions):


    1.What is the ETL engine? Do I need to purchase any additional software?

    Pentaho Data Integration Communities Edition is free of charge and is used as the ETL engine to extract, transform and load data. Download PDI from Sourceforge:

    http://sourceforge.net/projects/pentaho/files/Data%20Integration/


    2. Why not just use Dynamic Data Tables instead of using get_external function for Enrichment?

    This is a GREAT question and really goes back to the requirements. This enrichment is not meant for high volume events, but for events (Availability) that require to query some external source  in real-time. Discuss the scenarios with your TrueSight Administrator or Architect (or even post questions/comments on BMC Communities) for guidance. You need to weigh the cost vs benefit of using get_function.


    GREAT post:  RE: Dynamic Enrichment - which way is best...


    4. Why do you use Pentaho? Can we use BMC Atrium Orchestrator (BAO)?

    The reason for using Pentaho in this posting is to use a powerful ETL tool that does not require scripting knowledge. You can ofcouse use your own scripts instead. BAO could be used, but I see this as a maturity step and follow a Crawl - Walk - Run methodolgy. Get started with simple scripts or PDI to address specific Use Cases, then as you mature to Run stage, then consider using BAO as an alternative solution if requirements drive that direction.


    5. We are a Linux shop, will this work on Linux?

    All the components of these examples should work on Linux. If you download the attached files, you will need to modify any Windows reference (paths and executables). You will need to change the bat.exe to shell or perl scripts. Overall, the ETLs will be re-usable in a Linux envrionment with some minor changes.


    6. I see from the screenshots the icons are different for the hypervisor and virtual machines, how is that done?

    I used customized icons and an ETL to build out the topology dynamically from the VSM KM (Patrol KM). This is not required for this posting, but if you want more information, please check out the following Communities Posting:  Automatically Create Virtual Topology using VSM KM and Direct Publishing with Simple Service Model Support - WINDOWS version


    That Communities posting has custom icons (and customized mc_sm_object.baroc) with the integration. There is a configuration file named component_icon.properties. As inidcated from the VSM integration posting, first copy and backup the existing file component_icon.properties located in the following directories:


              pw\pronto\conf\

              pw\pronto\data\Image\Icon\


    Then copy and replace component_icon.properties from the integration to the directories identified above. Once copied, then restart the jserver by typing:  pw p r jserver


    There are custom icons that have been added to the integration. Unzip the file named "Custom_CIType_icons_9-25-2015.zip" to the following icon directory:


              pw\tomcat\webapps\pronto\jsp\swf\assets\icons\ciType


    Once copied, then restart the jserver by typing:  pw p r jserver


    7. You mention in the posting to place this type of Enrichment in a Enrichment cell tier vs Normalization cell tier, what does that mean?

    You can get more information on the Best Practice architecture deployment, for example, TrueSight Operations Manager v10:  BMC TrueSight Operations Management basic deployment - BMC TrueSight Operations Management 10.0 - BMC Documentation


    Also, the following are videos on a methdology called Event Management Framework. An overview of the EMF and architecture if covered in these short videos:


    Dynamic Downstream Correlation with BMC TrueSight Operations Manager on Vimeo

    EMF Concepts and Configuration on Vimeo

     

    Example Requirements:

    The following are required for this posting:

    • Pentaho PDI - Kettle (Open Source ETL)
    • Java SE (JRE) – no specific version required
    • BPPM +9.5, 10 (TSIM) - again these examples can be configured on remote cells (highly recommended)

     

    Posting Feedback:

    What do you think? Please provide any feedback on the comments section below. Also, if you have any enhancements or ideas that  can contribute or list, that would be appreciated. Enjoy and keep Communities a sharing environment!

     

    APPENDIX

    So how did we do it? 4 steps .. time to peel back the onion

    Onion.jpg

    Example 1:  Enrich with REST API

    Step 1 - GET_CONFIGURATION.baroc

     

    MC_INTERFACE: GET_CONFIGURATION

    DEFINES

    {

            monitorTypeName:              STRING,  default='Not available';

            monitorInstanceName:        STRING,  default='Not available';

            attributeName:                    STRING,  default='Not available';

            url:                                      STRING,  default='Not available';

    };

    END

    *Save this as "GET_CONFIGURATION.baroc" into the kb\classes directory of the cell. Also, add "GET_CONFIGURATION" to the end of the .load file and save.

    Explanation:

    Above defines the slots of interface. When I first started to query TSIM for monitoring details via REST API, the following JSON format was a good starting point for defining the interface. Here is the JSON input file from a sample query:

    ...

    {

                  "monitorInstanceId":"ts-im:501228:4931",

                  "monitorTypeName":"VMware VM VMotion",

                  "statusMsg":"OK",

                  "statusCode":"200",

                  "attributes":[

                      {

                        "attributeName":"VmMigrationsPerDay",

                        "unit":"Number",

                        "precision":2,

                        "kpi":false

                      },

                      {

                        "attributeName":"VmMigrationsPerHour",

                        "unit":"Number",

                        "precision":2,

                        "kpi":true

                      },

                      {

                        "attributeName":"VmMigrationsPerWeek",

                        "unit":"Number",

                        "precision":2,

                        "kpi":false

                      }

                }

                  ],

                  "monitorInstanceName":"VMotion"

    ...

     

    Step 2 - refine_rest_api_monitorlist.mrl

     

    refine rest_api_get_monitor_configuration: PATROL_EV ($EV)

            where

            [

                    $EV.mc_object    ==  "VmMigrationsPerDay" AND

                    $EV.status              == OPEN

              ]

    {

            # This is an example of making a REST API call to TSIM using an ETL (Pentaho)

            get_external(get_rest_api_configuration,[$EV.mc_host,$EV.mc_object],GET_CONFIGURATION,$PDI);

            $EV.mc_long_msg  = 'TrueSight Monitor Details:' || char(10) || '    monitorTypeName = ' || $PDI.monitorTypeName || char(10) || '    monitorInstanceName = ' || $PDI.monitorInstanceName || char(10) || '    attributeName = ' || $PDI.attributeName || char(10) || '    REST URL = ' || $PDI.url ;

            ntadd($EV,'Get_External: Rest API lookup on ' || $PDI.monitorTypeName);

    }

    END

     

    *Save this as "refine_rest_api_monitorlist.mrl" into the kb\rules directory of the cell. Also, add "refine_rest_api_monitorlist" to the end of the .load file and save.


    Explanation: 
    In the rule above, get_external will call a script named "get_rest_api_configuration". Do not put an extension on the end of the name (such as .bat, .sh ...). $EV.mc_host and $EV.mc_object will be passed as variables into the script. Lastly the get_external interface will return results to a named pipe in the format defined by GET_CONFIGURATION (step1). In this example the results must be formatted in the correctly. For example, the PDI ETL transforms the results and writes to the named pipe in this format:

     

    GET_CONFIGURATION;monitorTypeName='VMware VM VMotion';monitorInstanceName='VMotion';attributeName='VmMigrationsPerDay';url='http://172.21.86.142/bppmws/api/Device/host38/stats?starttime=2015-10-04T19:16:26&endtime=2015-10-05T19:16:26&idType=Name&statstype=aggregate&attributetype=kpi&monitor=VMware%20VM%20VMotion:VmMigrationsPerDay';END

     

    Step 3 - get_rest_api_configuration.bat

     

    D:\bmc_pentaho\data-integration\kitchen.bat /file:C:\pdi_repo\get_external\Get_External_REST_API_List_Monitors.kjb "-param:output_file="%1 "-param:mc_host=%2" "-param:mc_object_class=%3" --level=Basic >> c:\temp\trans.log

     

    *Save this as "get_rest_api_configuration.bat" into the kb\bin\w (this is a Windows env) directory of the cell.

     

    Explanation:

    This script will call the PDI Job via kitchen utility and populate pre-defined paramters with slot values passed from the event. In this case, %1 is the interface named pipe (which will be what the ETL will write results to), %2 is mc_host and %3 is mc_object.

     

    Step 4 - Get_External_REST_API_List_Monitors.kjb

     

    This step requires Pentaho to be downloaded to the server where the cell is running. PDI is supported on Windows and Linux. My scenarios are executed on Windows servers. To download PDI, see the FAQ section of this post. This is the PDI Job which executes a series of transformations:

    pix3.png

    Configuration: if you were to re-use this Job, open the File with Spoon and then double-click on the whitespace (background) of the job. Click on the Parameters Tab and fill in the relevant information (values). LEAVE mc_host and mc_object BLANK since those values will be passed in by step 3.

    pix4.png

    Lastly, open the first transformation named "Get_External_REST_API_List_Monitors.ktr" and edit the REST Client step by double-clicking on the step:

    pix5.png

    Click on the Authentication tab and change the username and password. By default, admin / admin12345 is used:

    pix6.png

     

    The ETL is a workflow that is meant to be used an example. Please take the Pentaho files and re-use if it helps. If you would like more information on the details of the transformation then leave a message below for this posting.

     

    Example 2:  Enrich with VMWare PowerCLI


    Example 2 details/explanation will be updated soon .... the overall process is similiar to Example one but calling Powershell script that I created. It was abit tricky to call a PowerCLI powershell script from the shell command. The following link was absolutely useful to getting example completed!

     

    Execute a process - Pentaho Data Integration - Pentaho Wiki

     

    In the meantime, if you are new to VMWare PowerCLI (like I was), here are some useful links:

     

    Follow this article to download VMWare PowerCLI:

    http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-and-vsphere-articles/general/getting-started-with-vmware-powercli.html

     

    Quick overview of VMWare PowerCLI PartI:

    http://www.virtualizationadmin.com/articles-tutorials/general-virtualization-articles/use-powercli-quick-stats-part1.html

     

    Quick overview of VMWare PowerCLI PartII:

    http://www.virtualizationadmin.com/articles-tutorials/general-virtualization-articles/use-powercli-quick-stats-part2.html