Discovery: Duplicate CI's in the Discovery CMDB dataset (i.e. BMC.ADDM)

Version 5
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Discovery


    COMPONENT:

    BMC Discovery 11.3


    APPLIES TO:

    BMC Discovery



    PROBLEM:

     

    Duplicate CI's appear in the Discovery CMDB dataset (i.e. BMC.ADDM). There are two similar CIs. Neither one is marked as deleted (if one of them is marked as deleted , this is not a duplicate).


     

     


    SOLUTION:

     

    Root cause 1: If continuous sync is turned off, and a host reaches its aging limit and is destroyed, the delete will not be propagated to the CMDB. As a result there could be duplicates if both the "old" and "new" hosts remain in the CMDB. 

    Solution 1: If not already done: restart the continuous sync and force Discovery to "mark as deleted" the destroyed host nodes (see KA 000026778 )

    *******************************************************************************************************************************************************************
    Root cause 2: Two or more different appliances were used to synchronize nodes to the same dataset. 

    Solution 2: Resynchronize the CMDB with the procedure below

      

    - mark as deleted all the CIs in BMC.ADDM (using the Remedy User tool)
    - run reconciliation to propagate the marked as deleted flag into BMC.ASSET
    - run the BMC.ADDM purge RE job to actually delete the CIs from BMC.ADDM
    - configure Discovery to resynchronize everything.

      

    This problem can also happen if synchronization was being done from a scanner, then a new consolidator and is installed and the the sync is now done from the consolidator (without cleaning up the target dataset). See KA 000090788.

    *******************************************************************************************************************************************************************

    Root cause 3: A backup was restored on the CMDB side only (or Discovery only, or on both Discovery/CMDB but the backup were not taken at the same time). The restore added back some nodes that were deleted in the past. Discovery will not update some of the CMDB CIs in this case. 

    Solution 3: See Solution 2.

    *******************************************************************************************************************************************************************

    Root cause 4:  A custom syncmapping pattern uses  sync.BMC_ClassName() instead of sync.shared.BMC_SharedClassName() to insert a CI in the CMDB
    The concept of shared CIs is documented here:

    https://docs.bmc.com/docs/discovery/113/syncmapping-block-788110659.html
    extract: "For deletion to work correctly, the system must know that such CIs are shared"

      

    Solution 4: To confirm the root cause and resolve this type of duplicate:
    - reset the Discovery CMDB dataset
    - disable all custom patterns
    - resynchronize and confirm that there are no duplicates

    *******************************************************************************************************************************************************************

    Root cause 5: The CMDB user account used by Discovery to synchronize to the CMDB is not compliant with the Discovery requirements listed in this article.

    extract: "The username of the BMC Atrium CMDB user that is at least a member of a group having the CMDB Data Change All role. Note that a user part of the "administrator" or "asset admin" group may not comply with this requirement."

    In one case, even with CMDB Data Change All permissions, the data still could not be read by the Discovery user.  However, data could be written.  (This caused many duplicates after a Resync).

    Note: The ADDMIntegrationId is the same when this permission problem happens.

    The root cause is confirmed if the following log messages are seen (and duplicate CIs are observed):

      

    DEBUG: Lookup in CMDB for instance OI-17c7d806e531416a8005003d669bfc67 (BMC_BaseElement) failed! (Message:ERROR (120006): Instance not found.; OI-17c7d806e531416a8005003d669bfc67)
    DEBUG: Could not identify the operation which caused the exception
    ERROR: Sync of: <hostname> An error occurred during the sync transaction.
    com.tideway.integrations.cmdbsync.exception.CMDBAccessException: Cannot check in CMDB for object: OI-17c7d806e531416a8005003d669bfc67

    To further verify root cause, login to the Remedy system as the Discovery User, and see if that User can see data in the BMC.CORE:BMC_BaseElement form.

      

    To further verify the root cause, login to the Remedy system as the user account used for Discovery sync, and see if that user can see data in the BMC.CORE:BMC_BaseElement form.

    If that user can not see data in that form, the problem is confirmed.  (It is possible that the user can write data, but can not read data).

    Solution 5: Update the CMDB user account to meet the requirements described above.

    *******************************************************************************************************************************************************************

    Root cause 6:  Something has modified the content of BMC.ADDM.  For example, if the attribute MarkedAsDeleted is modified (manually or automatically with an AR workflow/escalation), this may lead to duplicates. In this case, cardinality errors can also happen.

    Solution 6: Do not modify the content of BMC.ADDM (manually or otherwise). If CMDB data must be changed, do it in a different dataset.

    *******************************************************************************************************************************************************************

    Root cause 7: Some SNMP devices run two SNMP agents on two different IPs, but both report the same sysname. For example, the storage arrays EMC VNX7500 and Hitachi DF600F. If these are scanned, Discovery creates two nodes with the same name, and logically the CMDB sync will create two BMC_ComputerSystem CIs with the same name.

    Solution 7: It is recommended to keep the two BMC_ComputerSystem CIs as they are.

    *******************************************************************************************************************************************************************

    Root cause 8: AR Server crashes with the following error message in arerror.log :

      

    Thu Nov 14 20:36:51 2013  390635 : AR System server terminated — fatal error occurred in ARSERVER (ARNOTE 21) 

      

    OR: 

      

    Tue Aug 26 12:14:29 2014  AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (<serverName>)  ARERR - 92
    Tue Aug 26 12:14:30 2014  AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501)

      

    (this can happen when AR can't reach its database)

      

    Sometimes, this can lead Discovery to create a duplicate CI instead of updating an existing one. In this case, the ADDMIntegrationID is the same in the duplicate CIs. This can affect any type of classes. This can lead to duplicate graphs of CIs (computers, software servers ....). The indications in the exporter log are unknown, but something like this may be observed in the transformer log:

      

    47726035695936: 2014-08-26 12:09:22,572: cmdb_sync.handler: INFO: Sync Host node 63b0c1fc2412582581d1ef506e486f7374
    [...]
    47726035695936: 2014-08-26 12:09:22,653: cmdb_sync.handler: INFO: Target subgraph has 5 nodes, 6 relationships.
    47726035695936: 2014-08-26 12:09:22,656: cmdb_sync.handler: INFO: Sync to CMDB...
    [...]
    47726035695936: 2014-08-26 12:58:09,836: cmdb_sync.servants: WARNING: Failed to sync Host node 63b0c1fc2412582581d1ef506e486f7374. Retry 1: Transient Error occurred during sync transaction 
    47726035695936: 2014-08-26 12:58:09,837: cmdb_sync.servants: INFO: Re-added node to sync queue. Retry 1.

      

    To confirm the root cause, compare the creation date of the most recent CI or the duplicates, the date of the AR error message and the timestamp of the transformer log "Re-added node to sync queue". If the date is the same, the root cause is confirmed.

    Solution 8: Resolve the AR server crash issue and execute the resynchronization procedure documented in KA 000030970. Note that this KA can't keep the issue from happening again, it can only resynchronize Discovery and CMDB.

    *******************************************************************************************************************************************************************

    Root cause 9:  Identity change. When a new scan result is significantly different from the previous one, Discovery may create a new host. The old one is not immediately deleted. This could be considered as a duplication of nodes in Discovery and CIs in CMDB. The scans following an upgrade may cause this problem (if the new version collects new information that does not match the old scan result)

      
      Solution 9: see KA 000030001.

    *******************************************************************************************************************************************************************

    Root cause 10:  The data from Discovery scanners is migrated to a new appliance, but the data from the consolidator is not (or its data was lost/deleted).
      
       
       If in this scenario the old consolidator is replaced with a new/empty one, the Discovery keys will change. They won't match the ADDMIntegrationID values previously sent to the CMDB, and this will logically lead to duplicates in the CMDB at the next sync.

    Solution 10: 
     
       
       If the version of the consolidator is 10.0 or later+ and the old consolidator is still available, use the tw_root_node_key_export tool to export the existing keys, then use the tw_root_node_key_import tool to import these keys into the new consolidator before restarting the consolidation. - With Discovery 9.x and below, the consolidator has to be migrated with its data before restarting the sync.  
         
        *******************************************************************************************************************************************************************

    Root Cause 11:
    There is a defect (DRUD1-20650, DRUD1-20659) in the Discovery 11.1 version of "Complete Resync" which leaves duplicate CI's in the CMDB.  These duplicates should have been removed during the resync. See KA  000138256.

    Solution 11:
    In Discovery 11.1 versions prior to 11.1.0.6, use "Incremental Resync" instead of "Complete Resync"
    Or, upgrade to Discovery 11.1.0.6 or 11.2, where this defect is fixed. After upgrading, perform another "Complete Resync"

    *******************************************************************************************************************************************************************

    Root Cause 12: 
    There is a defect (DRUD1-22225) in Discovery versions 10.2, 11.0, 11.1, and 11.2 as follows:
    Duplicate CIs created during Resync when AR Server setting “Max Entries Returned by GetList” is set below 5000.
    The problem is that, if the setting is 2000, for example, Discovery thinks there are only 2000 CI's in the CMDB.  Discovery stops asking for more CI's. because it asked for 5,000 and only got 2,000.

    Note:  As of January 2018, the ROD default for “Max Entries Returned by GetList” is 2000.
    The on-premise default is 0 / Unlimited.

    Symptoms of this problem: 
      
        
        The "Prepare" portion of the Resync will be very quick (about 5-10 minutes).  This occurs because Discovery gets very few CI's from the CMDB.
    The results of the Prepare will show that there will be many, many Inserts to be performed, very few updates, and very few deletes.
    If customer allows the "commit" to proceed, the duplicate CI's  will have identical ADDMIntegrationID's.
    Customer should "cancel" the Resync if duplicates are being created in the CMDB.

    See KA 000147834.
       
       
        Solution 12:
    Change the AR Server setting “Max Entries Returned by GetList” to 5000 or greater, or set it to 0 (Unlimited).  
    Or, upgrade to a Discovery 11.3 which contains a fix for DRUD1-22225.  
    After changing the AR Server setting or upgrading to 11.3, perform another Resync.

    *******************************************************************************************************************************************************************
         
        
         Root Cause 13: 
    After reconciliation or normalization of a CI, the ADDMRowLevelSecurity (i.e. "group") of the CI can change so that the Discovery CMDB sync user no longer has permission to see the CI.
    When Discovery can not see the CI, it will eventually create a new one.
    If a different user logs in to Remedy, that other user may be able to see duplicate CI's.
        
         If there are duplicate CI's that have different ADDMRowLevelSecurity, then this could be the root cause.

    Symptoms of this problem: 
    Here are messages from the transformer log indicating that Discovery can not see some CI's:
        
    tw_svc_cmdbsync_transformer.log.2018-10-31:W20-140049166952192: 2018-10-31 19:25:53,888: cmdb_sync.sync: INFO: CMDB_QA: Sync of BMC_ComputerSystem:hosta.bmc.com aborted: Root CI not found     
    tw_svc_cmdbsync_transformer.log.2018-10-31:W18-140010071897856: 2018-10-31 19:32:54,001: cmdb_sync.sync: INFO: CMDB_QA: Sync of BMC_ComputerSystem:hostb.bmc.com aborted: Root CI not found     
        
    To diagnose the problem:     
    Login to Remedy with the       Discovery CMDB sync user, and observe that the user can not see both CI's. It only sees one CI.     
        
         Solution 13:
    Edit the Discovery CMDB sync user (using the User form in Remedy) to add the ADDMRowLevelSecurity group that is causing the problem.
    *******************************************************************************************************************************************************************
       
        
       
      

     


    Article Number:

    000096922


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles