Automatically Create Virtual Topology using VSM KM and Direct Publishing with Simple Service Model Support - WINDOWS version

Version 17
    Share This:

    Summary of Integration:

    This is a BMC Communities provided and supported integration. The purpose of this integration is to populate TrueSight Operations Manager or BMC ProactiveNet Performance Manager service cell directly with VSM KM (from BPPM/TSIM Sybase DB) device and relationship/dependency data using Direct Publishing. Service Models and topology models can be represented regardless whether the Atrium CMDB is integrated with BPPM or not.


    The value of this integration would be the following:


    • Event Management - event enrichment with "meta" data of devices managed from VSM KM (OS details)
    • Topology - provide the Manager of Manager (TSIM/BPPM) visibility into interconnected devices. Quickly answer the question - "What is connected to this device?"
    • Service Impact Management - Simple Service Models are supported with this integration. Also, if you have Entuity and using the Direct Publishing integration (found here: Entuity Device Synchronization with BPPM or TSIM using Direct Publishing v3.0 ) then by default Edge Switches to Hypervisor relationships/Impacts are created and enabled. You can use the Edge Switch topology for Downstream Correlation Use Cases.


    **Note: If you are running TSIM or BPPM on Linux platform, then please goto the following Communities link:

    Automatically Create Virtual Topology using VSM KM and Direct Publishing with Simple Service Model Support - LINUX version


    The following diagram shows the data flow from BPPM or TSIM DB (with VSM KM data) to the BPPM/TSIM Service Cell:


    VSM KM Direct Publishing Flow3.png


    The integration relies on VSM KM (at least 1 Patrol Agent) and BPPM/TSIM installed with embedded Sybase DB. Oracle DB is now supported:


    • Enable creating Resource CIs and relationships such as NICs and Storage Volumes. This is disabled by default.
    • House cleaning. Force all hostnames to lowercase and REGEX FQDN to use shortnames.
    • Enable creating Simple Service Models. Included with this integration is a XLSX which is used to create Application and even Service Models very quickly. See the PPT Appendix for details.
    • Included is a separate ETL, which required Entuity ETL to be configured. This ETL will extract MAC Addresses from the Hypervisors (ESXi Hosts) and then dynamically query the Entuity Export Database for MAC to Port/Host mappings. The ETL can then build relationships from the Edge Switch to ESXi host. By default only ESXi hosts are enabled for Edge Switch Connectivity, but if Entuity is discovering vSwitches (e.g. Cisco nexus100v), then you can enable vSwitch to Virtual Machine relationships as well since the ETL does extract virtual MAC Addresses from vCenter (via VSM KM).
    • NEW - support for building SAN Topology from the Sentry Software EMC Disk Array KM to ESXi servers. Disk Array --> Disk/Raid Groups --> Storage Volumes (naa.ID) --> ESXi Hosts.
    • NEW - custom icons for TSIM or BPPM Ops Console. We now have icons for ESXi, RHL, Windows, Centos SAN and network devices.
    • NEW - Oracle support for environments for BPPM/TSIM


    why_integration.png


    Who is this integration intended for?


    This is a BMC Communities provided and supported integration is intended for customers who do not have Atrium CMDB or ADDM (Discovery) but want to populate the BPPM/TSIM Service cell with CIs and Relationships. This version only supports customers with the embedded Sybase database (Oracle DB not yet supported). Also, if you want to accomplish Downstream Correlation, which a short video can be found here:  Dynamic Downstream Correlation with BMC TrueSight Operations Manager on Vimeo


    So let's take a different approach to getting to an end-state ....

    vadar_and_son_bike.png


    5 Steps to the integration:


    Step 1 – Enable Direct Publish Publishing on BPPM or TSIM server

      Step 2 – Create Publishing Environment on BPPM Server

                    penv open -e VSM -p OriginId=DirectPublish -p HomeCell= cellName

      Step 3 – Install and Configure Pentaho for ETL automation

    To change icons within BPPM, first make a backup of component_icon.properties and then copy and replace the component_icon.properties file attached to this integration to pw\pronto\conf\ AND pw\pronto\data\Image\Icon\ directories.

    Download Pentaho CE (SourceForge) – local to BPPM Server

    Extract VSM ETL files (KJB / KTR) locally on the BPPM/TSIM server

    Configure Variables for ETL  which makes this ETL very portable and simple to install. The default Sybase DB username used in the ETL is dba since the ETL queries tables (not views). Oracle is now supported as well. You may need to edit the Input steps of the transformations and change the username and/or password for table connection to BPPM/TSIM embedded Sybase DB. Here is a screenshot of the variables that you configure for your environment at the JOB level (the variables are passed to all transformations within the JVM):

    Job Parameters.png

      Step 4 – Run ETL and Validate Device Synchronization

      Here is a screenshot of BPPM Admin Console with Component Data Tables updated with Inventory data collected by VSM KM:

    Step4 Admin Console.png

      Here is a screenshot of the BPPM Ops Console with CIs and Relationships. Notice that the icons are different for the Hypervisor (ESXi Host) and the Virtual Machines:

    Pix1.png

    Note: if you have ESXi servers which are not part of a vCluster but have Virtual Machines running, then the ETL will only create dependencies from the Hypervisor (ESXi) --> Virtual Machine (VM) with no vCluster in the topology model or path.

        Step 5 – *Optional – Simple Service Model Support

        Included in this integration is the ability to create hundreds of Simple Service Models by updating a multi-tabbed XLSX (spreadsheet). The ETL will automatically dump BMC_ComputerSystems that are discovered via VSM KM or exist in vCenter to a CSV file. You will just copy all or a subset of the rows to the Simple_Models_TSOM.xlsx. Minimally you will need to list names of Applications within your environment (first tab) then map server to app as seen below: (*note a server can support multiple Applications. Simply insert a new line and copy hostname)

    XLSX_2nd_tab.png

        After quickly completing the multi-tabbed spreadsheet, you will have Simple Service Models created as seen below. Your Simple Service Models can stop at the Application level (minimal configuration) or you can extend to the Business Service level.

    Pix2.png

        Step 6 – *Optional – Connecting Entuity (network) Topology

        This step is dependent on having Entuity and the the following integration deployed: Entuity Device Synchronization with BPPM or TSIM using Direct Publishing v3.0 Included in this integration is an ETL which will extract the MAC Addresses from the Hosts (and Virtual Machines) and query the Entuity Export Database for Port/MAC connections. Default is only to reason connections from the Edge Switch to Physical Host. If Entuity has discovered vSwitches such as Cisco nexus1000v, then the ETL can connect Virtual Machines to the vSwitch. Again, all relationships are deleted and rebuilt when the ETL is run to prevent stale relationships. Here is an example:

    Pix3.png

    Validation: you can validate the EdgeSwitch connectivity or uplinks by opening a vSphere Client and select a hypervisor or ESXi host and then select the configuration tab. Click on Hardware --> Networking on one of more of the Physical NIC(s) or adapters and see the properties. If discovered, then vCenter should have the same switch listed. See below for an example:

    vSphere_Client_validation.png

     

    FAQs (Frequently Asked Questions):

    1.What is the ETL engine? Do I need to purchase any additional software?

    Pentaho Data Integration Communities Edition is free of charge and is used as the ETL engine to extract, transform and load data. Download PDI from Sourceforge:

    http://sourceforge.net/projects/pentaho/files/Data%20Integration/


    2. I see from the screenshots the icons are different for the hypervisor and virtual machines, how is that done?

    Attached with the integration is a configuration file named component_icon.properties. First copy and backup the existing file component_icon.properties located in the following directories:


              pw\pronto\conf\

              pw\pronto\data\Image\Icon\


    Then copy and replace component_icon.properties from the integration to the directories identified above. Once copied, then restart the jserver by typing:  pw p r jserver


    NEW - there are custom icons that have been added to the integration. Unzip the file named "Custom_CIType_icons_9-25-2015.zip" to the following icon directory:


              pw\tomcat\webapps\pronto\jsp\swf\assets\icons\ciType


    Once copied, then restart the jserver by typing:  pw p r jserver


    3. How does the ETL connect to BPPM or TSIM database?

    The input step and Lookup steps will connect to the embedded Sybase database warehouse database using JDBC (SQL query). The integration assumes the dba user will be used since Tables (not views) are queried. Attached in the integration is the jconn4.jar which you will need to copy to the {Pentaho_Home}\data-integration\lib directory. Configuration for the integration is done with parameters (paramters tab) on the job named "VSM_Device_Synch_with_TrueSight.kjb".


    If it is required to change the DB username and/or DB password for the integration, then you will need to edit/configure the Input Step for the following transformations and save the transformation: (you can edit these directly in the Spoon client)


    • VSM_TrueSight_Integration_PPOSTER_BAROC
    • VSM_TrueSight_Integration_PPOSTER_BAROC_vCluster_Relationships
    • VSM_TrueSight_Integration_PPOSTER_BAROC_VM_Relationships
    • VSM_TrueSight_Integration_PPOSTER_BAROC_Resources_Relationships


    4.How does the ETL handle changes (deltas)?

    As part of the ETL, the Relationships transformation will delete all the existing VSM relationships in BPPM (*only ones created by the integration) and then rebuild every time the ETL is  run. The ETL is very fast so there is no issue with rebuilding relationships every time the ETL is run (scheduled on a daily basis or can be event driven). This will keep the relationships current and prevent stale relationships. The Inventory ETL only adds new devices (ETL does not remove devices – manual step required).


    5.How do I delete a device with Direct Publishing?

    Since the ETL does not remove devices, you just need the mc_udid of the device and create a baroc file. See documentation Appendix on details.  More info on pposter from BMC Docs:

    https://docs.bmc.com/docs/display/public/proactivenet96/pposter


    6.How do I remove all CIs and Relationships from the integrations?

    If you want to reset or remove all CIs from then run munpublish from the BPPM server:

          munpublish -n <cell> "DirectPublish.VSM"


    7.How do I remove the integration or the Direct Publishing environment?

    If you want to remove the DirectPublish.VSM environment, then from the BPPM Server:

          penv close -e VSM -p "OriginId=DirectPublish" -p "HomeCell=<cellName>"


    8.Could the ETL be triggered by an event? Meaning, if BPPM/TSIM received a VMotion event, could the ETL be triggered to rebuild the topology.

    Absolutely! You configure an automatic action or even configure an Operator Initiated action from the console. The benefit is to rebuild the virtual farm relationships in near real time.


    9.Will this integration work if I have Oracle as my backend datastore to BPPM or TSIM?

    Yes - as of 11-20-2015. I worked with a customer who has BPPM 9.6 and Oracle 11.2 Oracle. The ETL files are attached below in this post. If you have Oracle, use the zip file named "VSM_TrueSight_Integration_Oracle_Support_ETL_Files.zip". You will need to open Spoon and Edit the first step (Table Input Step) and edit the connection information for the following transformation (.ktr) files:


        • VSM_TrueSight_Integration_ComputerSystem_PPOSTER_BAROC.ktr
        • VSM_TrueSight_Integration_PPOSTER_BAROC.ktr
        • VSM_TrueSight_Integration_PPOSTER_BAROC_vCluster_Relationships.ktr
        • VSM_TrueSight_Integration_PPOSTER_BAROC_VM_Relationships.ktr
        • VSM_TrueSight_Integration_PPOSTER_BAROC_Resources_Relationships.ktr
        • from the 2nd sub Job (VSM_Device_Synch_with_TrueSight_Sub.kjb) edit VSM_TrueSight_Integration_PPOSTER_Marked_for_Deletion.ktr

     

    Once you edit the connection for the Table Input Steps, replace the sample information with specific information from your environment (Oracle). Here is an example (click to enlarge the screenshot):

    Oracle.png


    10.If I enable Simple Service Models to be created in the ETL, what are the steps needed to be completed to get the models built?

    Here are the steps to enables Simple Service Modeling witin this integration. Please following these steps:


              1. Run the ETL at least one time with the default setting (parameter simple_service_models value set to "No")

              2. The ETL will generate a CSV file in ${output_dir}\Use_Me_for_Simple_Service_Models.csv

              3. Open the Simple_Models_TSOM.xlsx (spreadsheet) from the integration package with MS Excel. Also open the file generated from step #2 (Use_Me_for_Simple_Service_Models.csv) with MS Excel.

              4. Click on the first tab named "1.Applications". Copy and paste (or type in the Application names). You may have a list of Application found in SharePoint, a spreadsheet or even some Element Manager like Troux. Once you type in the Application Names, then provide the importance or priority of the Application. High maps to "PRIORITY_1", Medium maps to "PRIORITY_3" and Low maps to "PRIORITY_5" as seen below: (note if no Application_Importance is selected, then default value "PRIORITY_5" is used)

    Apps_Tab.png

              5. Next, click on the tab "2.Server_to_Apps", copy all or any subset of Hostnames from the exported CSV file (Use_Me_for_Simple_Service_Models.csv) and paste them to the Hostname column. Once pasted, then use the drop down from the Application Name column and map a server to an Application. If a server supports multiple applications, you will need to insert a new row and copy the hostname to the new row and map that hostname to an additional Application. See below as an example:

    Host_to_App_tab.png

              6. You can stop here and this will map Hosts (BMC_ComputerSystem) to Applications (BMC_Applications). If you choose to only map to Applications, make sure to delete all sample data in the "3.Apps_to_Business" tab and move to next step. Otherwise, if you want to build out Business Services, then open the tab "3.Apps_to_Business" and map Applications to Business Services. Forexample: Active Directory, VOIP, Lync and MS Exchange are applications that support a Technical Service named "Messaging". You can also set a Priority at the Business Service level. See below for an example:

    Apps_to_Service.png

              7. Save the spreadsheet named "Simple_Models_TSOM.xlsx" and copy back to the BPPM or TSIM server in the ${output_dir}.

              8. With the Spoon client, open the following file named "VSM_Device_Synch_with_TrueSight.kjb"

              9. Double click on the white space of the job, which will bring up the job properties dialogue box

              10. Click on the Parameters Tab and change the value for the parameter named "simple_service_models" to "Yes"

              11. Re-run the ETL from Spoon or CLI and validate the models.

              Suggestion: create Dynamic Component Folders for Applications, vClusters and ESX servers such as the screenshot below:

    Pix4.png

    11.How can I create additional virtual resources CIs and add them to the models such as HBA Cards,  MACAddresses, and Storage Volumes?

    This is included with the ETL but turned off (disabled) by default. At the top level Job named "VSM_Device_Synch_with_TrueSight.kjb", you can configure the Parameters tab, and set the Parameter value for "create_resourcepool_cis" to "Yes". This will create the addtional CIs and Dependencies as seen in the screenshot below. Note: the impacts from these resources to the Virtual Machine and Host (ESXi) are not active.

    resources_with_cis.png

    12.?Can I create Service or Topology models that include SAN CIs? (e.g. EMC Disk Arrays)

    NEW - the ETL does support linking to EMC Disk Array logical components when using the Sentry Software EMC Disk Array KM. More information will be provided, but the logic is supported in the ETL. You will need to change a Parameter to "create_san_cis" from No to Yes.

    Pix5.png

    Here is a screenshot (much linking VSM KM and Entuity Network Topology) of linking EMC Disk Arrays via naa.ID to ESXi Hosts:

    Pix6.png

     

    Other vendors will be supported. Working with the Sentry Software Team to support Direct Publishing for other vendors such as NetApp and Hitachi etc.


    Integration Requirements:

    The following are required for this integration:

    • Pentaho PDI - Kettle (Open Source ETL)
    • Java SE (JRE) – no specific version required
    • BPPM +9.0, +9.5, 10 (TSIM)
    • VSM KM +v4.0
    • Embedded Sybase Database
    • *Oracle is NOW supported!


    Integration LoE:

    This base integration will take about 20 minutes to setup. The ETLs can be scheduled and run on a nightly basis. See the attached PPT presentation (PDF file) and goto Appendix for details on how to schedule the ETL.


    Getting Started:

    Please review the following before starting the integration:

    1. TrueSight OM - VSM KM Synch for BPPM or TSIM 5-4-2015.pdf (PPT presentation)
    2. VSM_KM_Device_Synch_with_BPPM_and_TSOM_v1 (Install/Config documentation)

    Other related links and videos:

    With the relationships synched to BPPM/TSIM, you could start to do Downstream Correlation Use Cases. Here is a short 8 min Demo video on Downstream Correlation Use Case with TrueSight Operations Manager or BPPM:

    Dynamic Downstream Correlation with BMC TrueSight Operations Manager on Vimeo

     

    Integration Feedback:

    What do you think? Please provide any feedback on the comments section below. Also, if you have any enhancements or ideas that you can contribute or list, that would be appreciated. Enjoy and keep Communities a sharing environment!

     

    Solution Resolved - 5/15 - Customer wants to use this ETL to build out models, but still open L2 or L3 tickets with Remedy. This means the ReconID need to refined to the CIs. Working on a solution for next release to be able to interface with BSR (BMC Service Resolution) without having to publish from the Atrium CMDB. I may post that as a separate ETL if customers have this scenario. If you are interested in this scenario of being able to integrate L2 ticketing with Remedy without needing for BPPM or TSIM to directly publish to the Atrium CMDB, then please email karlis_peterson@bmc.com

     

    Notes: this integration includes minor fixes in the ETL, new icons and also support for Sentry Software EMC Disk Array KM topology linking. This integration does extend the mc_sm_object (e.g. extending PrimaryCapability enumeration), so if you are publishing from the Atrium CMDB and looking to use this integration, you will need to back out some of the custom slots.