13 Replies Latest reply on Oct 18, 2019 10:39 AM by Sharvan Singh

    CI Duplication in Golden Dataset

    Gajanand Patil
      Share This:

      Hi All,

       

      I am seeing the many CI duplication in Golden Dataset. Below are few scenarios I observed :

       

      • 1 ADDM ci Dataset record and 2 Asset ci Dataset records with same recon id
      • Multiple ADDM and Asset Dataset cis with same name.
      • Multiple ADDM and Asset Dataset cis with same ToeknID
      • Multiple ADDM and Asset Dataset cis with same Serial Number

       

       

      Can somebody assist me how to fix the ci duplication ?

       

       

      Regards,

      Gajanan

        • 1. Re: CI Duplication in Golden Dataset
          Nick Caulfield

          Gajanan,

           

           

           

          Where you have duplicate records in the ADDM dataset by Name, Serial Number or TokenId - you may want to try a resynch from ADDM before you do anything beyond. Resynchronizing a CMDB connection - Documentation for BMC Discovery 11.3 - BMC Documentation

           

          In terms of duplicate reconciliation IDs in the BMC.ASSET dataset, sometimes I get that too and it should be impossible.

           

          Presumably you have seen the document Investigating issues with duplicate data in CMDB - Documentation for BMC Atrium Core 9.1 - BMC Documentation

           

          I have used the SQL query in this as the starting point for identifying records which I have subsequently updated, but I feel it's more appropriate to recommend you contact BMC Support before updating live data.

          3 of 3 people found this helpful
          • 2. Re: CI Duplication in Golden Dataset
            Ganesh Gore

            Please note Dataset and Recon id combination is unique and it is handled at system level. So the statement 'multiple CIs in SAME dataset with SAME recon id' is incorrect. (If it is happening by any chance, you are dealing with some critical defect but I am sure there are any related to this). There can be duplicate CIs in SAME dataset but with different recon ids or record id with 0. In such cases, You can take a report of sample duplicate records and check all the values. Most of the time, reconciliation engine couldn't identify the matching record with the provided qualification and hence couldn't provide the record id. or if values are different (serial number, hostname, tokenids, domain etc), it treat them as new records and couldn't merge with existing and hence create new entry in ASSET with new reconid.

            4 of 4 people found this helpful
            • 3. Re: CI Duplication in Golden Dataset
              Nick Caulfield

              Ganesh, I know it's supposed to work like that but I've observed instances with the same ReconciliationId in the BMC.ASSET dataset in version 19.2

               

              I just ran

               

              SELECT ReconciliationIdentity, COUNT(*) FROM BMC_CORE_BMC_COMPUTERSYSTEM

              WHERE DatasetId = 'BMC.ADDM' 

              GROUP BY ReconciliationIdentity HAVING COUNT (*) > 1 ;

               

              and found two examples just today so whatever is supposed to handle it at a system level does not always work.

              It looks like I set one instance of each to MarkAsDeleted in the most recent clean up I did but they have somehow escaped the purge. I'll delete them manually today.

               

              The more recent of the two examples had been created in July. That slightly pre-dates the upgrade from 18.5 to 19.2 but both examples were created when we were running 18.5

               

              There may have been examples created since we had 19.2 that I since cleaned up

              1 of 1 people found this helpful
              • 4. Re: CI Duplication in Golden Dataset
                Ganesh Gore

                This is strange and must be a defect. (at the time of merge, recon engine throws error ARERR 120092 and should not create duplicates.

                ARERR[120092] The Dataset ID and Reconciliation Identity combination is not unique

                 

                btw here are the steps to identify duplicates:

                AtriumCore Reconciliation - Finding duplicates in dataset and how to find and fix them.

                1 of 1 people found this helpful
                • 5. Re: CI Duplication in Golden Dataset
                  Nick Caulfield

                  Another example, I just found a BMC_IPEndpoint CI which had two instances with the same ReconciliationId in BMC.ASSET

                  Both instances were created 19/08/2019 10:08:48 (so since we have been running 19.2)

                   

                  In one of those instances, the ModifiedDate is also 19/08/2019 10:08:48, in the other, it has been updated since and the modified date is 11/10/2019 21:26:44

                   

                  The updated instance is the one which would count as "first matching"

                   

                  That search brought up 3 examples of IPEndpoint and 2 examples of LANEndpoint CIs

                  2 of 2 people found this helpful
                  • 6. Re: CI Duplication in Golden Dataset
                    Gajanand Patil

                    Hi Nick,

                     

                    Thank you so much for sharing this. Let me go through the links.

                     

                    Regards,

                    Gajanan

                    • 7. Re: CI Duplication in Golden Dataset
                      Gajanand Patil

                      Hi Ganesh,

                       

                      I am using ARS-1808 and I have 2 asset records with same recon id. I  also surprised when I seen this. But it is there and have to correct it.

                       

                       

                      Regards,

                      Gajanan

                      • 8. Re: CI Duplication in Golden Dataset
                        Gajanand Patil

                        I ran the same query mentioned by Nick and I got the same recon id records in Golden Dataset.

                         

                         

                        Regards,

                        Gajanan

                        • 9. Re: CI Duplication in Golden Dataset
                          Stefan Hall

                          My thoughts on the topic, I am currently analyzing and correcting all possible error situations again.

                           

                          ADDM ci Dataset record and 2 Asset ci Dataset records with same recon id

                          I never saw same dataset + reconid combination twice. I‘ve used all AR Plattform version since 8.1, currently 19.02

                           

                          Multiple ADDM and Asset Dataset cis with same name.

                          same as TokenID

                           

                          Multiple ADDM and Asset Dataset cis with same ToKenID

                          Two things are important 

                          • TokenID is completely unimportant for ADDM Sync, so it can happen very quickly. If they come in at the same time and the Recon Identity runs only afterwards, they all get their own Recon id and are merged into the Asset.
                          • Otherwise it could also be due to your identification rules.

                           

                          Multiple ADDM and Asset Dataset cis with same Serial Number

                          same as TokenID

                          2 of 2 people found this helpful
                          • 10. Re: CI Duplication in Golden Dataset
                            Ganesh Gore

                            I would log a ticket a ticket with BMC and let them look into it. You might to need to involve BMC R&D/Product team to look into this. Looks like some issue with Reconciliation engine. This is fundamental principle of the CMDB design and must be fixed if it is not working by any chance.

                            1 of 1 people found this helpful
                            • 11. Re: CI Duplication in Golden Dataset
                              Gajanand Patil

                              Hi Ganesh,

                               

                              Yes. You are right.

                               

                              Also I observed that those ASSET cis where the 2 recon ID with same dataset are created on same time period. I am involving BMC support team as well as I am investing myself on it.

                               

                              Thank you.

                               

                              Regards,

                              Gajanan

                              • 12. Re: CI Duplication in Golden Dataset
                                Gajanand Patil

                                Hi Nick,

                                 

                                I have checked the information available on the links shared by you. I am checking for the discovery resync activity. Thank you so much for sharing this information with me.

                                 

                                Regards,

                                Gajanan

                                • 13. Re: CI Duplication in Golden Dataset
                                  Sharvan Singh

                                  Dataset id and Recon ID combination is unique but this issue occurs sometime(very very rare). If you are running Reconciliation engine as multi threaded (which of-course we do from performance perspective) and you have one record in target dataset(BMC.ASSET) and have 2 records in staging dataset which are duplicates based on your identification rules. while multi threaded processing there are chances both threads will assign same recon id(matching from target dataset) to both records in staging dataset but this happens very very rare.

                                  You wont get much help from support until you replicate this and share recon debug log and API/FLTR/SQL logs. and its hard to replicate this

                                   

                                  There is another use case of having duplication in staging dataset based on identification rules and when Target dataset is empty/no matches. duplicate records will get different recon ID. In both scenarios answer would be Its working as designed, you should not have duplicates in staging dataset

                                  1 of 1 people found this helpful