1 2 Previous Next 15 Replies Latest reply on Jul 8, 2020 2:04 AM by Marek Ceizel

    Purge MarkAsDeleted Yes , not always happening

    Jeff Sikorski
      Share This:

      Version 18.05 , but this happened in old 8.1 as well.


      BMC's "Purge Engine" is, in theory, supposed to "Delete" everything that is marked as MarkAsDeleted = Yes when you run a Purge recon job.  But it seems like there's something else it's looking for.


      I have thousands of CIs that are MarkAsDeleted = Yes in both BaseElement and BaseRelationship and never get purged during a Purge process.


      I'm noticing most (maybe even All) of the CIs that it just overlooks are CIs that I have set to MarkAsDeleted = 1 via an AI job through this type of step:




      Perhaps setting MarkAsDeleted = 1 via this step in Spoon is not triggering "other" back-end workflow that tells the system it's truly ready to be purged. 


      Does anyone know perhaps what else it might be looking for ?  Or what would be wrong with my above step ?


      Other post with similar issues as me:

      Not able to purge MarkAsDeleted relationships




        • 1. Re: Purge MarkAsDeleted Yes , not always happening
          Stefan Hall

          Let's do the same and not have the problem. I don't think it's the Al job.


          Couple of questions to narrow things down:

          - What classes are we talking about?

          - Does your job only handle normalized CI's and are the MAD=yes CI's normalized?

          - does your job only process identified CIs and are the MAD=yes CIs identified?

          - Is the option "check destination/asset" (something like that) activated in purge and are the MAD=yes CIs present in the asset?


          Then we should know what the problem is

          • 2. Re: Purge MarkAsDeleted Yes , not always happening
            Jeff Sikorski

            Just so we don't get off on the wrong track... the purge process works *sometimes*.  It just doesn't work 100% of the time.   I bring up the AI job because the CIs not being purged seem to be related to that type of behavior.


            Also keep in mind that behavior-wise in my environment, I don't ever purge ComputerSystems, Monitors, Equipment, and Printers.  So a lot of these classes are 'old CIs' related to the ComputerSystems.  It's almost like Atrium is expecting me to Purge the ComputerSystem along with the software, IPs, etc that are related to the ComputerSystems.


            However -- this goes back to my original question:

            If a CI is MarkAsDeleted = Yes, I would expect the Purge behavior to Delete it, regardless of whether or not it's also able to purge related items.



            My Laptop, Jeffs_Laptop, is a ComputerSystem CI.  It's being discovered by SCCM and relates IP addresses, software, processors, etc.  When Jeffs_Laptop gets picked up and goes to Inventory, my AI job sets all the related items (IP addresses, software, processors, etc) to "MarkAsDeleted = Yes".  However, I'm keeping Jeffs_Laptop and not purging that.  So I still want to keep Jeffs_Laptop, but I want all the related items gone.


            It doesn't seem to be working this way.


            Right now I have 1.95 million BaseElement and BaseRelationship CIs that have "MarkAsDeleted = Yes" , but when the Purge job runs, it doesn't recognize that there are any CIs ready to be purged




            - does your job only process identified CIs and are the MAD=yes CIs identified?

            I'm not sure what you mean by this





            - What classes are we talking about?


            These are all the different classes and counts where MarkAsDeleted = 1 , and it's not being purged ... so basically everything






            • 3. Re: Purge MarkAsDeleted Yes , not always happening
              Stefan Hall

              Jeff, this is how I understood you, the process works and you have 1.9 million undeleted entries


              Seriously, there are some situations where a MAD=yes cannot be transferred from the source, such as an earlier normalized CI which currently cannot be normalized due to a different model name. My questions were aimed at these specific cases.


              Now I saw that you want to purge in the asset dataset. So the questions are not so important anymore.


              With us products, IPs, MACs etc. are removed without any problems. These are some 10k elements every day.

              With us is always the CI and the corresponding relationship MAD=yes. We do not remove the main element either.


              We also set MAD via the API.

              Hmm, can you show me another screenshot of your "purge" activity. There should be some checkboxes in the lower right corner.

              • 4. Re: Purge MarkAsDeleted Yes , not always happening
                Jeff Sikorski


                The BaseElement table actually seems to be getting purged ok.  It's the BaseRelationship table that is struggling and not following the "Purge if MarkAsDeleted = 0"


                and full disclosure -- I am using a qualification for my purge activity to avoid purging those "physical hardware" classes.  But I am 100% sure it's not impacting anything.


                Qualification is:
                ( 'ClassId' != "BMC_COMPUTERSYSTEM" AND 'ClassId' != "BMC_EQUIPMENT" AND 'ClassId' != "BMC_MAINFRAME" AND 'ClassId' != "BMC_PERSON" AND 'ClassId' != "BMC_PRINTER" AND 'ClassId' != "BMC_MONITOR" AND 'ClassId' != "BMC_APPLICATION" ) OR 'OK_to_Purge' = 0


                OK_to_Purge is a custom attribute we added so I could purge those classes if needed by flipping that switch.


                I've tried the below qualification with both the "relationship" qualification and without. 


                And it still doesn't explain why many relationships do get purged just fine.  It's still hit/miss.  Even though 1.9 million unpurged relationships seems like a lot.. if this wasn't working at all, I would probably have 10+ million unpurged relationships.  So it *IS* working... just sometimes.









                Here are the relationships that aren't getting purged.  The bulk of them seem to be BMC_COMPUTERSYSTEM to BMC_PRODUCT relationships, which would be created by our SCCM integration.



                • 5. Re: Purge MarkAsDeleted Yes , not always happening
                  Rafael Rodriguez

                  Hello Jeff,


                  Since it's working sometimes and not other times my recommendation is to follow the following troubleshooting techniques to determine the root cause. It could be something related to orphaned or identification discrepancies. https://docs.bmc.com/docs/display/ac2002/Troubleshooting+CMDB+reconciliation+issues+by+collecting+diagnostics


                  Once you see the error in the logs it would provide some insight to what is fundamentally happening.

                  • 6. Re: Purge MarkAsDeleted Yes , not always happening
                    Jeff Sikorski

                    Thanks Rafael I will try.  I know I've looked in the past, but will try again.


                    The issue is that these aren't erroring out or anything during the recon job.


                    When the Purge recon job runs -- it just acts as though these 1.9 million CIs in BaseRelationship don't exist and don't have MarkAsDeleted = Yes.


                    No errors, no nothing.  I wish it would show an error somewhere because then I would have something to work with and understand why the Purge is failing on them.



                    • 7. Re: Purge MarkAsDeleted Yes , not always happening
                      Rafael Rodriguez

                      Jeff even if there is no error there should be something in the log indicating what is going on and why its being skipped/ignored.

                      • 8. Re: Purge MarkAsDeleted Yes , not always happening
                        Jeff Sikorski

                        I have opened Case 00857448 with BMC Support and uploaded the logs there.


                        Thanks all !

                        • 9. Re: Purge MarkAsDeleted Yes , not always happening
                          Stefan Hall

                          Jeff, I know the effect. But I rarely have it.


                          The Purge is based on CI's. If the target no longer exists, e.g. product, these orphan relationships are no longer found. In our case, this happened when we had problems for a while and jobs didn't go through clean. We removed the relationships on database level.


                          BTW in my rule for relationships I use 1=1, behind purge is already MAD=1, but it shouldn't make any difference.

                          • 10. Re: Purge MarkAsDeleted Yes , not always happening
                            Jeff Sikorski

                            I don't mind running a delete from BMC_CORE_BMC_BASERELATIONSHIP where datasetid = BMC.ASSET and MarkAsDeleted = 1  , but I was told this was a very bad way to do it.


                            The only other way is to do it through the form itself, but I can only delete 9,999 at a time (that's the limit to how many show up in searches).


                            If the only possible solution is to delete these manually -- then I'm actually okay with that.  But I can't keep trying to delete 1.9 million CIs 9,999 at a time.  I either need to delete from the database directly, or somehow increase the 9,999 to like 99,999.



                            • 11. Re: Purge MarkAsDeleted Yes , not always happening
                              Stefan Hall

                              Interventions at database level are never good, but sometimes there is no other way.


                              It wouldn't be good if you did it in a source dataset, because you would lose your relationship to the ASSET dataset.


                              But you are in the asset dataset and nothing depends on it anymore.


                              However, in the first step I would only delete the relationship where at least the source or target no longer exists in the asset dataset. These relationships definitely don't make sense anymore.

                              • 12. Re: Purge MarkAsDeleted Yes , not always happening
                                Jeff Sikorski

                                Actually you gave me some ideas and I started digging a little deeper.


                                The relationship is set to MAD = 1, but the BMC_PRODUCT CI in BaseElement is not.  That's probably partly why it's not being picked up by the Recon Purge engine.


                                I will figure out why the BMC_PRODUCT BE CI is not MAD=1, and play around with it.



                                • 13. Re: Purge MarkAsDeleted Yes , not always happening
                                  Marek Ceizel

                                  Hi Jeff Sikorski,


                                  How did you solve this at the end. I have similar situation. So I set the Reconciliation Log to debug and look what I got Inside:


                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:44.7330 */ Starting run of purge activity <PurgeActivity1>

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:44.7340 */ Source datasets <

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:44.7340 */ Value : BMC Asset

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:44.7340 */ >

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:44.7350 */ Target dataset <BMC Asset>

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:46.7590 */ Class <BMC.CORE:BMC_HostedSystemComponents>

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:46.7600 */ Skipping weak relationship class: BMC.CORE:BMC_HostedSystemComponents

                                  <INFO    > <TID: 0000036108> /* Di. Jul 07 2020 21:15:46.8270 */ Activity completed: PurgeActivity1


                                  The funny thing is, HostedSystemComponents is exactly the class which I want to purge. Now I'm interested if:

                                  - is there some way to purge even weak relationships (which are MaD) ?

                                  - can I delete them using Delete activity using right qualification or that will harm something .... ?



                                  thanks a lot



                                  Carey Walker - Carey .... any idea please ?

                                  • 14. Re: Purge MarkAsDeleted Yes , not always happening
                                    Stefan Hall

                                    Weak relationships are only tracked together with the CI.  Is the weak CI also MAD yes?


                                    BTW I haven't heard from Carry here for a very long time.

                                    1 2 Previous Next