Automatically Create Virtual Topology using VSM KM and Direct Publishing with Simple Service Model Support - LINUX version

Version 2
    Share This:

    Summary of Integration:

    This is a BMC Communities provided and supported integration. The purpose of this integration is to populate TrueSight Operations Manager or BMC ProactiveNet Performance Manager service cell directly with VSM KM (from BPPM/TSIM Sybase DB) device and relationship/dependency data using Direct Publishing. Service Models and topology models can be represented regardless whether the Atrium CMDB is integrated with BPPM or not.

    The value of this integration would be the following:

    • Event Management - event enrichment with "meta" data of devices managed from VSM KM (OS details)
    • Topology - provide the Manager of Manager (TSIM/BPPM) visibility into interconnected devices. Quickly answer the question - "What is connected to this device?"
    • Service Impact Management - Simple Service Models are supported with this integration. Also, if you have Entuity and using the Direct Publishing integration (found here: Entuity Device Synchronization with BPPM or TSIM using Direct Publishing v3.0 ) then by default Edge Switches to Hypervisor relationships/Impacts are created and enabled. You can use the Edge Switch topology for Downstream Correlation Use Cases.

    **Note: If you are running TSIM or BPPM on MS Windows, then please goto the following Communities link:

    VSM KM Device Synchronization with BPPM or TSIM using Direct Publishing with Simple Service Model Support

    The following diagram shows the data flow from BPPM or TSIM DB (with VSM KM data) to the BPPM/TSIM Service Cell:

    VSM KM Direct Publishing Flow3.png

    *Note: TSIM or BPPM with Oracle is supported with the MS Windows posting of this ETL, and will be supported soon for Linux. Please post a comment if you are interesting in getting support for the ETL where TSIM/BPPM is installed on Linux with backend of Oracle.

    The integration relies on VSM KM (at least 1 Patrol Agent) and BPPM/TSIM installed with embedded Sybase DB.

    • Enable creating Resource CIs and relationships such as NICs and Storage Volumes. This is disabled by default.
    • House cleaning. Force all hostnames to lowercase and REGEX FQDN to use shortnames.
    • Enable creating Simple Service Models. Included with this integration is a XLSX which is used to create Application and even Service Models very quickly. See the PPT Appendix for details.
    • Included is a separate ETL, which required Entuity ETL to be configured. This ETL will extract MAC Addresses from the Hypervisors (ESXi Hosts) and then dynamically query the Entuity Export Database for MAC to Port/Host mappings. The ETL can then build relationships from the Edge Switch to ESXi host. By default only ESXi hosts are enabled for Edge Switch Connectivity, but if Entuity is discovering vSwitches (e.g. Cisco nexus100v), then you can enable vSwitch to Virtual Machine relationships as well since the ETL does extract virtual MAC Addresses from vCenter (via VSM KM).
    • NEW - support for building SAN Topology from the Sentry Software EMC Disk Array KM to ESXi servers. Disk Array --> Disk/Raid Groups --> Storage Volumes (naa.ID) --> ESXi Hosts.
    • NEW - custom icons for TSIM or BPPM Ops Console. We now have icons for ESXi, RHL, Windows, Centos SAN and network devices.


    Who is this integration intended for?

    This is a BMC Communities provided and supported integration is intended for customers who do not have Atrium CMDB or ADDM (Discovery) but want to populate the BPPM/TSIM Service cell with CIs and Relationships. This version only supports customers with the embedded Sybase database (Oracle DB not yet supported). Also, if you want to accomplish Downstream Correlation, which a short video can be found here: Dynamic Downstream Correlation with BMC TrueSight Operations Manager on Vimeo

    So let's take a different approach to getting to an end-state ....


    5 Steps to the integration:

    Step 1 – Enable Direct Publish Publishing on BPPM or TSIM server

      Step 2 – Create Publishing Environment on BPPM Server

                   penv open -e VSM -p OriginId=DirectPublish -p HomeCell= cellName

      Step 3 – Install and Configure Pentaho for ETL automation

    To change icons within BPPM, first make a backup of and then copy and replace the file attached to this integration to pw/pronto/conf/ AND pw/pronto/data/Image/Icon/ directories.

    Download Pentaho CE (SourceForge) – local to BPPM Server

    Extract VSM ETL files (KJB / KTR) locally on the BPPM/TSIM server

    Configure Variables for ETL  which makes this ETL very portable and simple to install. The default Sybase DB username used in the ETL is dba since the ETL queries tables (not views). Oracle is now supported as well on the Windows posting for this ETL but not yet for the Linux version. This will relatively easy to support Oracle/Linux. You may need to edit the Input steps of the transformations and change the username and/or password for table connection to BPPM/TSIM embedded Sybase DB. Here is a screenshot of the variables that you configure for your environment at the JOB level (the variables are passed to all transformations within the JVM):


      Step 4 – Run ETL and Validate Device Synchronization

      Here is a screenshot of BPPM Admin Console with Component Data Tables updated with Inventory data collected by VSM KM:


    Here is a screenshot of the BPPM Ops Console with CIs and Relationships. Notice that the icons are different for the Hypervisor (ESXi Host) and the Virtual Machines:


      Step 5 – *Optional – Simple Service Model Support

        Included in this integration is the ability to create hundreds of Simple Service Models by updating a multi-tabbed XLSX (spreadsheet). The ETL will automatically dump BMC_ComputerSystems that are discovered via VSM KM or exist in vCenter to a CSV file. You will just copy all or a subset of the rows to the Simple_Models_TSOM.xlsx. Minimally you will need to list names of Applications within your environment (first tab) then map server to app as seen below: (*note a server can support multiple Applications. Simply insert a new line and copy hostname). Click to enlarge the screenshot.


        After quickly completing the multi-tabbed spreadsheet, you will have Simple Service Models created as seen below. Your Simple Service Models can stop at the Application level (minimal configuration) or you can extend to the Business Service level. Click to enlarge the screenshot.



    FAQs (Frequently Asked Questions):

    Start first with reading the attached documentation. Here are FAQs

    1.What is the ETL engine? Do I need to purchase any additional software?

    Pentaho Data Integration Communities Edition is free of charge and is used as the ETL engine to extract, transform and load data. Download PDI from Sourceforge:

    2. I see from the screenshots the icons are different for the hypervisor and virtual machines, how is that done?

    Attached with the integration is a configuration file named First copy and backup the existing file located in the following directories:



    Then copy and replace from the integration to the directories identified above. Once copied, then restart the jserver by typing:  pw p r jserver

    NEW - there are custom icons that have been added to the integration. Unzip the file named "" to the following icon directory:


    Once copied, then restart the jserver by typing:  pw p r jserver

    3. How does the ETL connect to BPPM or TSIM database?

    The input step and Lookup steps will connect to the embedded Sybase database warehouse database using JDBC (SQL query). The integration assumes the dba user will be used since Tables (not views) are queried. Attached in the integration is the jconn4.jar which you will need to copy to the {Pentaho_Home}/data-integration\/lib directory. Configuration for the integration is done with parameters (parameters tab) on the job named "VSM_Device_Synch_with_TrueSight.kjb".

    If it is required to change the DB username and/or DB password for the integration, then you will need to edit/configure the Input Step for the following transformations and save the transformation: (you can edit these directly in the Spoon client)

    • VSM_TrueSight_Integration_PPOSTER_BAROC
    • VSM_TrueSight_Integration_PPOSTER_BAROC_vCluster_Relationships
    • VSM_TrueSight_Integration_PPOSTER_BAROC_VM_Relationships
    • VSM_TrueSight_Integration_PPOSTER_BAROC_Resources_Relationships

    4.How does the ETL handle changes (deltas)?

    As part of the ETL, the Relationships transformation will delete all the existing VSM relationships in BPPM (*only ones created by the integration) and then rebuild every time the ETL is  run. The ETL is very fast so there is no issue with rebuilding relationships every time the ETL is run (scheduled on a daily basis or can be event driven). This will keep the relationships current and prevent stale relationships. The Inventory ETL only adds new devices (ETL does not remove devices – manual step required).

    5.How do I delete a device with Direct Publishing?

    Since the ETL does not remove devices, you just need the mc_udid of the device and create a baroc file. See documentation Appendix on details. More info on pposter from BMC Docs:

    6.How do I remove all CIs and Relationships from the integrations?

    If you want to reset or remove all CIs from then run munpublish from the BPPM server:

          munpublish -n <cell> "DirectPublish.VSM"

    7.How do I remove the integration or the Direct Publishing environment?

    If you want to remove the DirectPublish.VSM environment, then from the BPPM Server:

          penv close -e VSM -p "OriginId=DirectPublish" -p "HomeCell=<cellName>"

    8.Could the ETL be triggered by an event? Meaning, if BPPM/TSIM received a VMotion event, could the ETL be triggered to rebuild the topology.

    Absolutely! You configure an automatic action or even configure an Operator Initiated action from the console. The benefit is to rebuild the virtual farm relationships in near real time.

    9.If I enable Simple Service Models to be created in the ETL, what are the steps needed to be completed to get the models built?

    Here are the steps to enables Simple Service Modeling within this integration. Please following these steps:

             1. Run the ETL at least one time with the default setting (parameter simple_service_models value set to "No")

              2. The ETL will generate a CSV file in ${output_dir}\Use_Me_for_Simple_Service_Models.csv

              3. Open the Simple_Models_TSOM.xlsx (spreadsheet) from the integration package with MS Excel. Also open the file generated from step #2 (Use_Me_for_Simple_Service_Models.csv) with MS Excel.

              4. Click on the first tab named "1.Applications". Copy and paste (or type in the Application names). You may have a list of Application found in SharePoint, a spreadsheet or even some Element Manager like Troux. Once you type in the Application Names, then provide the importance or priority of the Application. High maps to "PRIORITY_1", Medium maps to "PRIORITY_3" and Low maps to "PRIORITY_5.

              5. Next, click on the tab "2.Server_to_Apps", copy all or any subset of Hostnames from the exported CSV file (Use_Me_for_Simple_Service_Models.csv) and paste them to the Hostname column. Once pasted, then use the drop down from the Application Name column and map a server to an Application. If a server supports multiple applications, you will need to insert a new row and copy the hostname to the new row and map that hostname to an additional Application.

              6. You can stop here and this will map Hosts (BMC_ComputerSystem) to Applications (BMC_Applications). If you choose to only map to Applications, make sure to delete all sample data in the "3.Apps_to_Business" tab and move to next step. Otherwise, if you want to build out Business Services, then open the tab "3.Apps_to_Business" and map Applications to Business Services. Forexample: Active Directory, VOIP, Lync and MS Exchange are applications that support a Technical Service named "Messaging". You can also set a Priority at the Business Service level.

              7. Save the spreadsheet named "Simple_Models_TSOM.xlsx" and copy back to the BPPM or TSIM server in the ${output_dir}.

              8. With the Spoon client, open the following file named "VSM_Device_Synch_with_TrueSight.kjb"

              9. Double click on the white space of the job, which will bring up the job properties dialogue box

              10. Click on the Parameters Tab and change the value for the parameter named "simple_service_models" to "Yes"

              11. Re-run the ETL from Spoon or CLI and validate the models.

             **Suggestion: create Dynamic Component Folders for Applications, vClusters and ESX servers such as the screenshot below:


    10.How can you schedule the ETL to run?

    The ETL can be run from the command line, so use CRON or some scheduler to execute a sample script. Here is a sample bat script (Windows) which can be done in shell or perl. Suggestion is to run the ETL 1-2 times daily. Remember you can also setup an automatic action (event driven) to execute the bat or shell script as well.


    C:\<pentaho_dir>\data-integration\kitchen.bat /file:C:\<location_of_ETL>\VSM_Device_Synch_with_TrueSight.kjb


    11.How can I create additional virtual resources CIs and add them to the models such as HBA Cards,  MACAddresses, and Storage Volumes?

    This is included with the ETL but turned off (disabled) by default. At the top level Job named "VSM_Device_Synch_with_TrueSight.kjb", you can configure the Parameters tab, and set the Parameter value for "create_resourcepool_cis" to "Yes". This will create the addtional CIs and Dependencies as seen in the screenshot below. Note: the impacts from these resources to the Virtual Machine and Host (ESXi) are not active. Click to enlarge the screenshot.


    12. Can I create Service or Topology models that include SAN CIs? (e.g. EMC Disk Arrays)

    NEW - the ETL does support linking to EMC Disk Array logical components when using the Sentry Software EMC Disk Array KM. More information will be provided, but the logic is supported in the ETL. You will need to change a Parameter to "create_san_cis" from No to Yes. Click to enlarge the screenshot.


    Here is a screenshot (much linking VSM KM and Entuity Network Topology) of linking EMC Disk Arrays via naa.ID to ESXi Hosts: Click to enlarge the screenshot.


    Other vendors will be supported. Working with the Sentry Software Team to support Direct Publishing for other vendors such as NetApp and Hitachi etc.

    Integration Requirements:

    The following are required for this integration:

    • Pentaho PDI - Kettle (Open Source ETL)
    • Java SE (JRE) – no specific version required
    • BPPM +9.0, +9.5, +10 (TSIM)
    • VSM KM +v4.0
    • Embedded Sybase Database

    Integration LoE:This base integration will take about 20 minutes to setup. The ETLs can be scheduled and run on a nightly basis. See the attached PPT presentation (PDF file) and goto Appendix for details on how to schedule the ETL.Getting Started:Please review the following before starting the integration:

    1. VSM_KM_Device_Synch_with_BPPM_and_TSOM_v3 (Install/Config documentation)


    Other related links and videos:

    With the relationships synched to BPPM/TSIM, you could start to do Downstream Correlation Use Cases. Here is a short 8 min Demo video on Downstream Correlation Use Case with TrueSight Operations Manager or BPPM:

    Dynamic Downstream Correlation with BMC TrueSight Operations Manager on Vimeo


    Integration Feedback:

    What do you think? Please provide any feedback on the comments section below. Also, if you have any enhancements or ideas that you can contribute or list, that would be appreciated. Enjoy and keep Communities a sharing environment!


    Solution Workaround with Atrium CMDB - I recently had a customer wanted to use this ETL to build out models, but still open L2 or L3 tickets with Remedy using BSR. This means the ReconID need to refined to the CIs. Working on a solution for next release to be able to interface with BSR (BMC Service Resolution) without having to publish from the Atrium CMDB. I may post that as a separate ETL if customers have this scenario. If you are interested in this scenario of being able to integrate L2 ticketing with Remedy without needing for BPPM or TSIM to directly publish to the Atrium CMDB, then please email


    Important note about customization: This integration does extend the mc_sm_object (e.g. extendingPrimaryCapability enumeration), so if you are publishing from the Atrium CMDB and looking to use this integration, you will need to back out some of the custom slots.