Solarwinds NPM Device Synchronization with BPPM or TSIM using Direct Publishing

Version 3
    Share:|

    Updated 7-28-15 - I have updated the transformations that was originally posted. There were some issues with Table Input step (MS SQL JDBC). You will need to download the MS SQL JDBC driver from SourceForge:


    http://sourceforge.net/projects/jtds/files/jtds/1.3.1/

     

    You can download the archive and unzip and copy and place the file "jtds-1.3.1.jar" in the <Pentaho_Base>

    data-integration\lib directory (e.g. D:\bmc_pentaho\data-integration\lib) and then re-start Spoon client. I have also added the "jtds-1.3.1.jar" file to the integration download section below. Enjoy and keep Communities a creative environment!


    Summary of Integration:

    This is a BMC Communities provided and supported integration. The purpose of this integration is to populate TrueSight Operations Manager or BMC ProactiveNet Performance Manager directly with Solarwinds NPM device and optionally relationship/dependency data using Direct Publishing. Service Models and topology models can be represented regardless whether the Atrium CMDB is integrated with BPPM or not.The value is of this integration would be the following:


    • Event Management - event enrichment with "meta" data of devices managed from Solarwinds (Manufacturer, Model, Version, Location, ...)
    • Topology - provide the Manager of Manager (TSIM/BPPM) visibility into interconnected devices. Quickly answer the question - "What is connected to this device?"
    • Service Impact Management - by default Edge Switches to servers have Impacts enabled. You can use the topology for Downstream Correlation Use Cases.


    The following diagram shows the data flow from Solarwinds NPM to BPPM or TSIM:

    Solarwinds Direct Publishing Flow2.png

    The base integration can be modified and extended to support specific customer requirements. The event integration from Solarwinds NPM to BPPM is located here:

    Solarwinds Orion NPM integration for BPPM and BEM Integration v1.1

     

    Also, here is a short video on the configuring the device/relationship synchronization:

    Solarwinds NPM Device and Relationships to BPPM / TSOM on Vimeo

     

    Let's take different approach to getting to an end-state ...

    vadar_and_son_baseball.png

     

    5 Steps to the integration:


    Step 1 – Enable Direct Publish Publishing on BPPM or TSIM server

    Step 2 – Create Publishing Environment on BPPM Server

                    penv open -e SOLARWINDS -p OriginId=DirectPublish -p HomeCell= cellName

    Step 3 – Install and Configure Pentaho for ETL automation

    Download Pentaho CE (SourceForge) – local to BPPM Server

    Extract Solarwinds ETL files (KJB / KTR)

    Configure Variables for ETL and Input Step for Solarwinds NPM SQL DB

    Step 4 – Run ETL and Validate Device Synchronization

    Here is a screenshot of BPPM Admin Console with Component Data Tables updated with Solarwinds Inventory data:

    BPPM - Admin console.png

    Here is a screenshot of the BPPM Ops Console with a device or CI and an associated Solarwinds event:

    Event with CI in Ops Console.png

    Step 5 – *Optional – Synchronize Layer-2 and Layer-3 Topology Relationships

    Here is a screenshot of the BPPM Ops Console with Layer-2 and Layer-3 relationships. Default configuration is that ALL network device to network device (Switch --> Switch, Router --> Switch ...) is configured as a Inactive Impact Relationship. Any Edge Switch to Servers, for example, is set to DIRECT and Active. You can configure and enable impacts and PropagationModel (DIRECT, DECREASING, JUST_WARNING, JUST_INFO ..):

    Network Model in BPPM.png

    *Note - the ETL queries all the interface to interface or port to port connections, and summarizes all the connections to a single relationship. One value point is that BPPM can provide dynamic views of Solarwinds managed devices. A user can use the Find CI search in the BPPM Ops Console and quickly understand what network devices are connected to the CI (as seen above).


    FAQs (Frequently Asked Questions):

    1.What is the ETL engine? Do I need to purchase any additional software?

    Pentaho Data Integration Communities Edition is free of charge and is used as the ETL engine to extract, transform and load data. Download PDI from Sourceforge:

    http://sourceforge.net/projects/pentaho/files/Data%20Integration/


    2.How does the ETL connect to Solarwinds NPM?

    The input step for Inventory and L2/L3 Relationships will connect to the Solarwinds NPM database using ODBC (SQL query). You will need to setup a ODBC DSN for the ETL to connect. See the documentation Appendix for details. You will need to configure the Input Step and configure the Connection to the MS SQL database (ODBC) for the following transformations: (you can edit these directly in the Spoon client)


    • Solarwinds_TrueSight_Integration_PPOSTER_BAROC.ktr
    • Solarwinds_TrueSight_Integration_PPOSTER_BAROC_Relationships.ktr


    3.How does the ETL handle changes (deltas)?

    As part of the ETL, the Relationships transformation will delete all the existing Solarwinds NPM relationships in BPPM and then rebuild every time the ETL is  run. The ETL is very fast so there is no issue with rebuilding relationships every time the ETL is run (sheduled on a daily basis). Inventory ETL only adds new devices (ETL does not remove devices – manual step required). This will keep the relationships current.


    4.How do I delete a device with Direct Publishing?

    Since the ETL does not remove devices, you just need the mc_udid of the device and create a baroc file. See documentation Appendix on details.  More info on pposter from BMC Docs:

    https://docs.bmc.com/docs/display/public/proactivenet96/pposter


    5.How do I remove all CIs and Relationships from the integrations?

    If you want to reset or remove all CIs from then run munpublish from the BPPM server:

          munpublish -n <cell> "DirectPublish.SOLARWINDS"


    6.How do I remove the integration or the Direct Publishing environment?

    If you want to remove the DirectPublish.SOLARWINDS environment, then from the BPPM Server:

          penv close -e SOLARWINDS -p "OriginId=DirectPublish" -p "HomeCell=<cellName>"


    Integration Requirements:

    The following are required for this integration:

    • Pentaho PDI - Kettle (Open Source ETL)
    • MS SQL JDBC Driver (jtds-1.3.1.jar needs to be placed in the ..\data-integration\lib file of PDI)
    • Java SE (JRE) – no specific version required
    • BPPM +9.0, +9.5, v10 (*Tested with TSIM v10)
    • Solarwinds NPM +v10
    • SQL User (Read Only) to query Nodes and OrionSwitchPortMapping tables


    Integration LoE:

    This integration will take about 20 minutes to setup. The ETLs can be scheduled and run on a nightly basis. See the attached PPT presentation (PDF file) and goto Appendix for details on how to schedule the ETL.

     

    Getting Started:

    Please review the following before starting the integration:

    1. TrueSight OM - Solarwinds Device Synch via Direct Publishing PPT v1.0b (PPT presentation)
    2. Solarwinds_NPM_Device_Synch_with_BPPM_and_TSOM_doc_v1_0b (Install/Config documentation)
    3. Short Video on integration and configuration: Solarwinds NPM Device and Relationships to BPPM / TSOM on Vimeo


    Other related links and videos:

    With the relationships synched to BPPM/TSIM, you could start to do Downstream Correlation Use Cases. Here is a short 8 min Demo video on Downstream Correlation Use Case with TrueSight Operations Manager or BPPM:

    Dynamic Downstream Correlation with BMC TrueSight Operations Manager on Vimeo

     

    Integration Feedback:

    Please provide any feedback on the comments section below. Also, if you have any enhancements or ideas that you can contribute or list, that would be appreciated. Enjoy and keep Communities a sharing environment!