1 2 Previous Next 17 Replies Latest reply on Oct 1, 2013 5:28 AM by Bjorn Calabuig

    DMT 8.1: Validate and Promote Steps

    Bjorn Calabuig
      Share This:

      Hi List,

      I've created a Transformation in order to load data into Organization CMDB class.

       

      To achieve this I made a copy of CI-CMDB transformation and did some modifications to populate BMC.CORE:BMC_Organization and AST:Organization forms.

       

      Load, runs smoothly. But Validation step doesn't seem to work.

       

      Validation stays all time in In Progress status; only if I check on "Enable wait", the step seems to work and all the loaded registers appear as "Validated" on AST:LoadAttributes form.

       

      Promotion, doesn't work at all, always "In Progress" but it never seems to end.

       

      I've checked out all the steps made for creating this new transformation (Atrium Integration Job and related variables, Spreadsheet, etc.).I've also reviewed Application Preferences for DMT (Create only=Template Value, Job Wait=Template Value).

       

      Maybe the difference between the original transformation and the new one relies on those Application Preferences.I'm not sure where this Template Values should be defined and what is expected from them. But the same Application Preferences seem to work as expected on the original transformation!

       

      Any help would be appreciated.

       

      Kind Regards,

      Björn.

       

      ITMS 8.1

        • 1. Re: DMT 8.1: Validate and Promote Steps
          Sachin Tappe

          Please make sure that Reconciliation Engine is running. Also, you can check the status of the last run of RE Job - "BMC Asset Management CI DATA LOAD. If you see a failure there then that could be the root cause of why your Data Management job is stuck in Promote.

          • 2. Re: Re: DMT 8.1: Validate and Promote Steps
            Bjorn Calabuig

            Hi,

            No improvements.

             

            First Issue

            First of all, I've changed DMT Application Preferences to (Create only=Template Value, Job Wait=Auto Validate-Auto Promote). I don't really like to change them, because days ago I loaded Business Services, using CI-CMDB original transformation, and everything went fine.

             

            Although, with these settings, I'm still having problems trying to load Organizations (into BMC_Organization class):

             

            - Load and Validation steps seem to finish OK

            - Promote step still In Progress

             

            I've an Excel with only one register. After execution:

            - BMC.CORE:BMC_Organization shows it OK

            - AST:Organization form NO

            - Accessing to CAI:EventsInterface I can see this errors:

             

            326: "The field is required and cannot be empty.", "AST:Attributes : ReconciliationIdentity";

            52: "The field is essencial for the system and cannot be modified.", "1";

             

            Second Issue

            Although the aforementioned error, I'm trying unfortunatelly to advance ignoring Validation and Promotion steps.

            - I execute the transformation from within Spoon client

            - After execution, the register appears on BMC.CORE:BMC_Organization

            - I force reconcilliation using Atrium Core, manually executing the job "CI DATA LOAD"

            - After execution:

                - a register appears on BMC.CORE:BMC_Organization (Dataset=BMC.ASSET)

              - a register appears onAST:Organization

             

            Here, I'm facing the second issue. Some ITSM attributes still appear empty:

            - Installation Date +

            - Return Date +

            - CI ID +

            - AdditionalDetails

             

            I've checked out all mapping and workflow and it seems to be OK.

             

            But, I'm every time more convinced that both problems are related to the Promote Step.

             

            I'm aware that, since ITSM 8.0 version, there is "sometimes" a problem with Promote Step (filter: ASI:DLP:CreateAttribute) due to the fact that some fields/attributes information is under CMDB and some others is under ITSM. Using DMT, promotion failed due to this filter. Following some good advices, I've splitted in two the aforementioned filter:

             

            - filter ASI:DLP:CreateAttribute_Create (Push Fields "if no requests match" -> Create, "if any request match" -> No action

            - filter ASI:DLP:CreateAttribute_Modify (Push Fields "if no requests match" -> No action, "if any request match" -> Modify First

             

            Also,on this second filter, I've erased submitter's mapping.

             

            After modifying also the related Filter Guide, I've made some testing with NO result.

             

            Maybe someone is experiencing similar issues.

             

            Kind Regards,

            Björn.

            • 3. Re: Re: DMT 8.1: Validate and Promote Steps
              Sachin Tappe

              Hi,

              Please check escalations are running ("Disable Escalation" should be Unchecked).

              The reason why you got the error is because the RecondId is not set for the AST:Attribute form.

               

              Additional Details column had an issue in 8.1 version and BMC has fixed the related defect in 8.1 Patch1.

               

              Thanks,

              Sachin

              • 4. Re: Re: Re: DMT 8.1: Validate and Promote Steps
                Bjorn Calabuig

                Hi Sachin

                 

                Please check escalations are running ("Disable Escalation" should be Unchecked).

                - Is unchecked. As said, I've made many tests weeks ago (importing Business Services, for example) with no errors.

                 

                The reason why you got the error is because the RecondId is not set for the AST:Attribute form.

                - I'm importing data using CI-CMDB transformation. Do you mean that the WF that executes on AST:LoadAttributes to AST:Attirbutes should have something missing?

                 

                Additional Details column had an issue in 8.1 version and BMC has fixed the related defect in 8.1 Patch1.

                - We're running ITSM 8.1 Build 001


                Kind Regards,

                Björn.

                • 5. Re: Re: Re: Re: DMT 8.1: Validate and Promote Steps
                  Bjorn Calabuig

                  Hi All,

                  I'm stil having problems with Promote step:

                   

                  Step "Load CI":

                  Spoon runs Job and performs the load on BMC_CORE and AST: LoadAttributes.


                  Once the Load step completes, looking at "Atrium Core / Atrium Core Console / Reconciliation" we'll be able to see thata new Reconciliation process has started for our Job "CI Asset Management BMC LOAD DATA".


                  At this point, two things happen:

                  • Identification Activity: generation of a unique ID (ReconID) for each new CI
                  • Merge Activity: merging data from BMC.ITSM.CI.DATA dataset to BMC.ASSET

                   

                  Step "Validate CI":

                  Once CI’s have been identified, the step “Validate CI” starts automatically.

                  All entries on BMC.CORE:BMC.xxx (Monitor, Printer, etc.) have a valid ReconID.


                  Step "Promote CI":

                  The step “Promote CI” usually quite fast and pushes information from AST:LoadAttributes to AST:Attributes.


                  Although, this step never ends, data has been transferred to AST:LoadAttributes.

                  None of the entries on this form has ReconID (ReconID is null).

                   

                  The cause can be seen in CAI: Events:

                  326: Required field cannot be empty AST:Attributes->ReconID

                  52: The field is essential for the system and cannot be modified-> "1"


                  Where to look from here on to solve this problem?


                  Kind Regards,

                  Björn.

                  • 6. Re: Re: Re: Re: Re: DMT 8.1: Validate and Promote Steps
                    Bjorn Calabuig

                    Hi All,

                    Today I've been able to load data into BMC_Organization using DMT, following this steps:

                     

                    Steps:

                    1. Create my own transformation (just including a section for BMC_Organization)
                    2. Create Job: Enter Job Name and Company. Save changes.
                    3. Select Create-> BMC Using Templates-> CI-CMDB. This creates the Job as BMC wants and needs.
                    4. Enter the Load CI step. At this point, add our Job in the field: "Atrium Integrator Name". Complete other necessary customizations as attaching Excel, DATASET type variables, etc. Save changes.
                    5. Change Job status to Build.
                    6. Run the Job.

                     

                    I started with 1 register. Load-Validate-Promote ended OK. (not the same result when I tried to load above 300 Organizations; I'have had to split into 3 files... but that's not so important at this moment).

                     

                    So, as I think it's supposed to be, I've only one entry on AST:LoadAttributes:

                    Im1.jpg

                     

                    ReconID looks fine, also "AST Attributes" like Asset ID (which I map to CI I+), for example:

                     

                    Im3.jpg

                    Well, with this big success , I'm trying now to load Physical Locations, but I'm getting the same error again:

                     

                    326: Required field cannot be empty AST:Attributes->ReconID

                    52: The field is essential for the system and cannot be modified-> "1"

                     

                    Of course, Promote never ends!!

                     

                    Taking a look into AST:LoadAttributes, I found something very interesting:

                    Im2.jpg

                    5 registers have been created! Why?

                    The third one should have been the only one generated, 'cause it has all data (and seems correct):

                    im4.jpg

                    Im5.jpg

                    If we take a look at the OOTB workflow, Promote cannot end because the filter ASI:DLP:CreateAttribute checks for:

                     

                    $ReconciliationIdentity$ = 'ReconciliationIdentity'

                     

                    OK, with this condition, the workflow will try to create a new entry on AST:Attributes taking the first register he find on the list (AST:LoadAttributes). As far as now, this is correct, but... when submtting, we'll get the aforementioned error just because, for the first entry shown, the ReconID is NULL!!

                     

                    In my desperation, I've changed the condition:

                    $ReconciliationIdentity$ = 'ReconciliationIdentity' AND $ReconciliationIdentity$ != $NULL$

                     

                    I was hopping to solve the problem with this workaround but no improvements.

                     

                    I really need help.

                     

                    Kind Regards,

                    Björn.

                    1 of 1 people found this helpful
                    • 7. Re: Re: Re: Re: Re: DMT 8.1: Validate and Promote Steps

                      Hi Björn,

                       

                      I had the same error where Promote step did not finish when trying to load roughly 400 People records. I had to split the file into chucks of 100 to get the files processed. There is a defect open for this at BMC and the workaround is to split your input files.

                       

                      The defect has something to do with duplicate records being produced, this may be the reason for reconciliation to fail.

                       

                      Regards,

                      Arto

                      • 8. Re: Re: Re: Re: Re: Re: DMT 8.1: Validate and Promote Steps
                        Bjorn Calabuig

                        Hi Arto,

                         

                        I'm not able to see the knowledge base entry. Seems like the defect has beeen solved right now?

                         

                        Anyway, I'm wondering why this happens while trying to import into Physical Location and not when trying to import into Organizations. I'm going to follow again the log (only 140.0000 lines), yesterday I found that the problem seems to begin at loading step...

                         

                        I'm going also to make tests with Monitors, Printers,... I'm really surprised nobody has problems using DMT 8.1 trying to load into "not standard" classes (i.e. classes not included in CI-CMDB).

                         

                        Moi moi.

                         

                        Kind Regards,

                        Björn.

                        • 9. Re: Re: Re: Re: Re: DMT 8.1: Validate and Promote Steps

                          Hej Björn, the KA is not approved for general distribution so removed the link from my reply.

                           

                          The reason for no problems being reported might be that people are using AI Spoon jobs instead of DMT templates to push CI data into CMDB. I have no experience with using DMT templates for CI data.

                           

                          Regards,

                          Arto

                          • 10. Re: DMT 8.1: Validate and Promote Steps
                            Bjorn Calabuig

                            Arto,

                             

                            I'm using also Spoon. And DMT calls Spoon transformations.

                             

                            With Spoon I created a transformation for Organizations, using CI-CMDB as a guide (if you open this transformation, you'll be able to see that some info is mapped against a CI Class and some info against AST:LoadAttributes). I also created a Job for Organizations, that just calls my transformation.

                             

                            But, from within Spoon, you'll only be able to check that your transformation inserts registers into BMC.CORE (in our case, BMC.CORE:BMC_Organization). Nothing will be inserted into AST:LoadAttributes nor into AST:Attributes.

                             

                            So, you'll have to do this using DMT (creating, at least, a new Atrium Integration Job and a variable for DATASETID), again using CI-CMDB as "a mirror".

                             

                            That's all. I tried also to create a new Job template, searching CI-CMDB and clicking on "Copy to Template"... and associating my new Atrium Integration Job to this template... When I executed this Job, Load went well, but Validate not, I had to remove Waiting Flag in order to move forward. Validate ended but Promote never, always In Progress (again, ReconID, the most of the times, was empty).

                             

                            Since a new Job template presented this problems, for executing my jobs, I decided to use the original CI-CMDB. But, on Load step, instead of using CI-CMDB Atrium Integration Job, I use mines.

                             

                            This is the situation I'm right now: Load and Validate end OK, but Promote not. But this is not a general behaviour: as said, for Organizations, the three steps end OK but for Physical Locations, Promote not (ReconID is empty).

                             

                            I'm still hopping I find out something diving into the log.

                             

                            Kind Regards,

                            Björn.

                            • 11. Re: Re: Re: Re: Re: DMT 8.1: Validate and Promote Steps

                              Ok, good luck with the logs. Hope you find the answer quickly!

                              • 13. Re: DMT 8.1: Validate and Promote Steps
                                Carl Wilson

                                Hi,

                                I know there was a bug in the 8.0 DMT where if you allocated more than 1/2 threads the to the DMT process, the system would hang at the promote stage - the flip side of this was to reduce the number of records you are importing or set the threading to a min value (which takes longer, but does complete).

                                Appears this may still be hanging around ....

                                 

                                Cheers

                                Carl

                                 

                                http://www.missingpiecessoftware.com/

                                • 14. Re: Re: DMT 8.1: Validate and Promote Steps
                                  Bjorn Calabuig

                                  Hi All

                                  I use a very standard transformation:

                                  Im1.jpg

                                  I've to say I solved the problem.

                                  Taking a look on the log:

                                  Im0a.jpg

                                  The filter AST:DLP:CreateAttribute raises Error 326.

                                  But the error comes from above, let's take a look at the filter DMT:TMG:Thread_StagingForm_Push`!:

                                  Im0b.jpg

                                  On the top of the image, we can see that filter passes qualification and enters on VALIDATE (over AST:LoadAttributes).

                                   

                                  When filter ASI:DLP:SetReconID_FromBase is checked and passes qualification, ReconID should be set asking for instanceid on AST:BaseElement. For some reason that I'm going to explain later, no instanceid is found, thus, ReconID Is null, thus AST:DLP:CreateAttribute fails.

                                   

                                  In a prior post I was surprised that 5 registers were created for just one Physical Location (on AST:LoadAttributes), the 3rd one being the only one correct, with ReconID, etc. If we take the instanceid from this 3rd register and check its existence on AST:BaseElement, we found the corresponding entry. If we try to find registers on AST:BaseElement for any of the other 4 entries, the system won't find anything.

                                   

                                  As I also said, the log seemed to lead to an error not on Validate or Promote step, but on Load step indeed.

                                   

                                  I began to look with more interest the filters included in ASI:DLP:ValidateCIAttributes filter guide and stopped on the first one, the filter ASI:DLP:ValidateCIError_OnLoad. This filter has this Run If:

                                   

                                  ('CMDB Error Code' != $NULL$) OR ('CMDB Error Message' != $NULL$)

                                   

                                  I discovered that in my transformation, these two fields where not present:

                                  Im2.jpg

                                  I don't know why I removed them (I supose due to the fact that when verifying my transformation in Spoon, I was getting an error/warning).

                                  So, I decided to add them again to all hops where this fields were needed.

                                   

                                  And executed my job (DMT). And Promote ends!

                                   

                                  I'm going to make more tests, but I think the problem (this one ) is solved.

                                   

                                  Kind Regards,

                                  Björn.

                                  1 of 1 people found this helpful
                                  1 2 Previous Next