Discovery:  Improving performance of CMDB Sync

Version 8
    Share:|

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Discovery


    APPLIES TO:

    BMC Discovery



    PROBLEM:

     

    CMDB sync takes too long

    Example: With ~6000 hosts, it took approximately 20 hours to sync to the CMDB

    Some things to look at are shown below ... See the Solution portion.

    1) Look at the "Details" page for your CMDB Sync Connection

    User-added image

    2) Look at the "Status" page to see if there are errors.  Especially see if there are timeouts or connection failures.

    3) Look at the Atrium Core Console (Normalization page) to see the settings for the BMC.ADDM dataset

    User-added image

     


    CAUSE:

    Various causes


    SOLUTION:

     

    Root Cause 1:  Inline Normalization for BMC.ADDM dataset will cause slow performance.
    By default, when creating a new Dataset in the Atrium Core Console (Reconciliation page), Inline Normalization is enabled for that dataset.
    Inline Normalization enabled means that for each CI synchronized to BMC.ADDM Dataset, a request for normalization is triggered.
    This is a large drag on performance.

      

    Solution 1: 
    Go to the AtriumCore Console->Normalization->Edit Configuration.
    Check that Inline Normalization is disabled for BMC.ADDM dataset
    With Inline Normalization disabled, you need to run Normalization as a batch job, and it should be run before Reconciliation.

    The screenshot above (in the Problem statement) shows 2 items that can be unchecked for BMC.ADDM dataset so that performance is better.

    You don't need the "Impact Normalization Feature" because ADDM CMDB Sync creates its own impact relationships (assuming you are using the default data model).

    Root Cause 2:  The specified RPC queue is too busy
    Symptoms of this problem would be connection errors and timeouts.

    The default RPC queue is 30696: the CMDB general administration queue.
    Other RPC queues that you can specify are:  390698 (Reconciliation queue), 390699 (Normalization queue)
    None of these RPC queues is a dedicated queue only for Discovery CMDB Sync.

    In a future release of Discovery (combined with a later version AR Server), there will be additional queues you can specify, so that you can use a dedicated queue for CMDB Sync.

    See this documentation about the CMDB Sync queues:  https://docs.bmc.com/docs/display/DISCO113/Configuring+CMDB+synchronization+through+a+private+queue

    Solution 2:  You can change the RPC queue in the GUI to use 390698 or 390699, but care must be taken so that CMDB Sync is not "competing" with Normalization and/or Reconciliation resources.

    Root Cause 3:  The "Concurrent Workers" on the "Details" page of the CMDB Sync connection in Discovery GUI is set too low or too high.
    1 Concurrent Worker may be too low.
    More than 3 Concurrent Workers may be too high, depending on how many threads are available to the RPC queue that is specified (see Root Cause 2 for discussion of RPC queues), and depending on how busy that RPC queue on the AR side.  i.e. none of those queues is dedicated to Discovery CMDB Sync.

      

    If you have many connection errors and timeouts during CMDB Sync, then the Concurrent Workers is probably set too high.
    If there are no connection errors or timeouts, then you can try setting Concurrent Workers to a higher value.  (more is not always better)

    What is a Concurrent Worker?  A Concurrent Worker is equivalent to approximately 5-10 threads.

      

    Solution 3:  Change the number of Concurrent Workers, using the information provided above.

    Root Cause 4:  The "Batch Size" could be too high or too low
    If there are timeouts and connection errors to the CMDB, the "Batch Size" could be too high.
    If there are no timeouts and no connection errors, increasing the "Batch Size" to get better performance may be possible.
    As a general rule, the "Batch Size" should not be higher than 100.  (There are exceptions, of course).

    Solution 4:  change "Batch Size" on the Details tab

    Root Cause 5:  Slow CMDB Sync or slow SQL on the AR Server side, caused by configuration settings or customizations on the AR Server side.
    Examples could be:  AR Escalations, AR Filters, an overlay on a BMC.CORE form (not recommended!)
    Another example is that indexes may need to be added to the AR Server internal database to improve SQL performance.

    Solution 5: Explore the performance issues with Atrium Core or AR Server team
    (Send these AR side logs:  arsql, arfilter, arescl, arapi)

    Root Cause 6:  If the CMDB version is CMDB 8.1 or CMDB 9.0, there is a patch for the CMDB for "slow Mark As Deleted performance".  (Be sure to have this patch if the version is CMDB 8.1 or 9.0).
    The problem does not exist in other versions of CMDB.

    Solution 6:  Get patch from the CMDB/Atrium Core team.
    The patch numbers are:
        For CMDB 9.0, it is SW00495641
        For CMDB 8.1, it is SW00495598

    Solution 7:  See also KA: 000091632 - ADDM/Discovery: Resynchronization / synchronization may fail with AR 90, 91 and/or 552 errors

      

    Related Products:  
       
    1. BMC Atrium CMDB Suite
    2.  
    3. BMC Atrium Discovery and Dependency Mapping

     


    Article Number:

    000056052


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles