8 Replies Latest reply on Jul 19, 2019 9:10 PM by Jon Trotter

    CMDB sync errors

    Nirmal Sharma
      Share This:

      Hi Experts,

      Discovery version 11.3

      CMDB version 9.1.03

       

      I have noticed sudden increase in sync errors. Below is the transcript of the errors picked using `cat  tw_cmdbsync_exporter.log| grep ERROR` command.

       

      ERROR (9713): The attempted bulk entry transaction failed due to an error in one of the individual operations.; Operation 1 in bulk transaction failed

      ERROR (120006): Instance not found.; OI-0946e4173f674bed877f61965ed5fc03

      ERROR (120006): Instance not found.; OI-0946e4173f674bed877f61965ed5fc03

      ERROR (9713): The attempted bulk entry transaction failed due to an error in one of the individual operations.; Operation 1 in bulk transaction failed

      ERROR (120006): Instance not found.; OI-95f294acdb9b44b3a26ff1f19a34ca07

      ERROR (120006): Instance not found.; OI-95f294acdb9b44b3a26ff1f19a34ca07

      ERROR (9713): The attempted bulk entry transaction failed due to an error in one of the individual operations.; Operation 1 in bulk transaction failed

      ERROR (120006): Instance not found.; OI-80ff3d5701674b5aa568671a8069a617

      ERROR (120006): Instance not found.; OI-80ff3d5701674b5aa568671a8069a617

      ERROR (9713): The attempted bulk entry transaction failed due to an error in one of the individual operations.; Operation 1 in bulk transaction failed

      ERROR (120006): Instance not found.; OI-fe38a028c3904c6fb6995976e3dde857

      ERROR (120006): Instance not found.; OI-fe38a028c3904c6fb6995976e3dde857

      ERROR (120006): Instance not found.; OI-ac2226e3e7774a14b509f3460307d524

      ERROR (9713): The attempted bulk entry transaction failed due to an error in one of the individual operations.; Operation 1 in bulk transaction failed

      ERROR (120006): Instance not found.; OI-09722569af1847669a8cff55b5525ac1

       

      However below is the long output from 1 instance of each different error from above.

       

      RequestProcessor-5: 2019-02-11 09:16:59,984: com.bmc.cmdb.api.CMDBAPILogger: FATAL: -CMDBSetInstance

      ERROR (120006): Instance not found.; OI-751b8ae7c5c6419face14fd2c0bf382b

              at com.bmc.cmdb.api.CMDBProxyJRpcBase.verifyStatus(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxyJRpcBase.cmdbCall(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxyJRpcBase.cmdbCall(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxy.CMDBSetInstance(Unknown Source)

              at com.bmc.cmdb.api.CMDBInstance.update(Unknown Source)

              at com.tideway.integrations.cmdbsync.SyncTransaction.commit(SyncTransaction.java:253)

              at com.tideway.integrations.cmdbsync.SyncProvider.syncInstances(SyncProvider.java:467)

              at com.tideway.integrations.cmdbsync.SyncProvider.syncRels(SyncProvider.java:422)

              at CMDBSyncCORBA.GraphSyncPOA._invoke(GraphSyncPOA.java:182)

              at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:297)

              at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:591)

              at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:734)

      RequestProcessor-5: 2019-02-11 09:22:42,059: com.bmc.cmdb.api.CMDBAPILogger: FATAL: -CMDBEndBulkEntryTransaction

      ERROR (9713): The attempted bulk entry transaction failed due to an error in one of the individual operations.; Operation 1 in bulk transaction failed

      ERROR (120006): Instance not found.; OI-444b510beb7e47f69b5db9906c5e6975

              at com.bmc.cmdb.api.CMDBProxyJRpcBase.verifyStatus(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxyJRpcBase.cmdbCall(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxyJRpcBase.cmdbCall(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxy.CMDBEndBulkEntryTransaction(Unknown Source)

              at com.bmc.cmdb.api.CMDBProxy.CMDBSetMultipleInstances(Unknown Source)

              at com.bmc.cmdb.api.CMDBUtil.CMDBSetMultipleInstances(Unknown Source)

              at com.tideway.integrations.cmdbsync.SyncTransaction.commit(SyncTransaction.java:244)

              at com.tideway.integrations.cmdbsync.SyncProvider.syncInstances(SyncProvider.java:467)

              at com.tideway.integrations.cmdbsync.SyncProvider.syncRels(SyncProvider.java:422)

              at CMDBSyncCORBA.GraphSyncPOA._invoke(GraphSyncPOA.java:182)

              at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:297)

              at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:591)

              at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:734)

       

      Please help.

        • 1. Re: CMDB sync errors
          Shane Smith

          Looks like it is trying to update a CI that it expects to find in the target CMDB dataset.  I would try finding the CI in the dataset by searching for that recon id found in the logs. To resolve it, you will probably need to complete a resync between Discovery and the CMDB dataset.

          1 of 1 people found this helpful
          • 2. Re: CMDB sync errors
            Duncan Grisby

            Something has changed the data in Discovery's dataset in the CMDB, so some CIs that Discovery thinks are there do not exist. Perhaps the CMDB was reverted to a backup?

             

            Discovery knows how to handle this situation, so you don't necessarily need to do anything. The next time it wants to sync those CIs, it will know that they were missing last time so it will create them. If you are drowning in a vast number of errors, you could do a resync, which will ensure that the CMDB dataset contains exactly what Discovery expects.

            4 of 4 people found this helpful
            • 3. Re: CMDB sync errors
              Jon Trotter

              What was the fix for this issue? Assuming one of the items Duncan mentioned?

               

              Nirmal Sharma

              • 4. Re: CMDB sync errors
                Lisa Keeler

                I know you are asking Nirmal...

                 

                We have a KA  000070570 which addresses common causes of ERROR (120006): Instance not found

                 

                Discovery: CMDB sync ERROR (120006): Instance not found.

                 

                 

                2 of 2 people found this helpful
                • 5. Re: CMDB sync errors
                  Jon Trotter

                  Thank you Lisa!

                  • 6. Re: CMDB sync errors
                    Jon Trotter

                    Hi Lisa,

                     

                    Is there a comprehensive list of the errors somewhere or a repository that we can use as a reference for possible future issues? I am guessing we won't run into every one of them or hopefully none, but we are looking to enhance our events that get triggered from these errors and would like to at the very least provide a link to the error code for troubleshooting purposes.Lisa Keeler

                    • 7. Re: CMDB sync errors
                      Lisa Keeler

                      Hi Jon,

                       

                      The error codes, such as 120006, come from the CMDB Java API.  Discovery receives the error from the AR Server, and then presents the error in the log (and on the status screen).

                       

                      Here is a very short list of error codes that Discovery expects to receive from the CMDB API:

                      CMDB_ERROR_INSTANCE_NOT_FOUND             = 120006

                      CMDB_ERROR_RELATION_CARDINALITY_CHECK     = 120039

                      CMDB_ERROR_RELATION_END_PT_DOES_NOT_EXIST = 120040

                      CMDB_ERROR_REL_ENDPOINT_MARK_AS_DELETED   = 120141

                       

                      Of course, Discovery can receive other error codes.  But, the ones above are the ones that Discovery expects and handled in a special way.

                       

                      Here is the doc regarding CMDB API error codes for version 9:  BMC Atrium CMDB C API error messages - Documentation for BMC Atrium Core 9.0 - BMC Documentation

                       

                      Those codes are for the C/API, but they are the same codes for the Java/API.

                       

                      You could receive errors regarding the Product Catalog or Normalization if you run inline Normalization on the BMC.ADDM dataset.

                      The only error I've seen regarding in-line Normalization is one that states that the Normalization plug-in is not running.

                      ERROR: An error occurred during the sync transaction ERROR (8753): Error in plugin; BMC.FILTERAPI.NORM.ENGINE

                      (KA 000113685)

                       

                      And, you could also be running inline Reconciliation and receive errors from Recon.  But, I have never seen errors from Recon ... of course Recon has errors, but it doesn't seem to pass errors back via the Java/API.  Recon seem to just log its errors to a log file.  (Normalization does also).

                       

                      There are also error codes that come from the AR Server such as the timeout errors, such as ARERR 94

                      Here are is the start page for the AR errors:

                      BMC Remedy AR System error messages - Documentation for Remedy Action Request System 18.08 - BMC Documentation

                       

                      Hope that helps.

                       

                      Lisa

                      3 of 3 people found this helpful
                      • 8. Re: CMDB sync errors
                        Jon Trotter

                        Helpful, as always Lisa. Thank you again!