10 Replies Latest reply on Feb 11, 2016 4:20 AM by Dmitriy Orshanskiy

    problem using OBJREL and CSV Imports to create hierarchies

      Share This:

      I've been working on using an ETL (Generic Object Relationship CSV parser) to import data from our CMDB, and am hitting a real stumbling block with having the objects appear in the proper domain, in the proper hiearchy. I've done a good bit of searching on these great forums as well as reading the docs, and I'm still stumped.

       

      Here's an example of the input file I'm using:

      TSFROMTSTOTSCHANGETYPEENTCATNMENTNMDS_ENTNMENTCATNMPARENTDS_ENTNMPARENTRELATIONTYPE
      2015-09-25 00:00:002000-01-01 00:00:009999-12-31 00:00:00ASSERTAPPSample_BizSample_BizCONTAINS
      2015-09-25 00:00:002000-01-01 00:00:009999-12-31 00:00:00ASSERTAPPSample_AppSample_AppCONTAINS
      2015-09-25 00:00:002000-01-01 00:00:009999-12-31 00:00:00ASSERTSYSSample_ServerSample_ServerCONTAINS
      2015-09-25 00:00:002000-01-01 00:00:009999-12-31 00:00:00ASSERTAPPSample_AppSample_AppAPPSample_BizCONTAINS
      2015-09-25 00:00:002000-01-01 00:00:009999-12-31 00:00:00ASSERTSYSSample_ServerSample_ServerAPPSample_AppCONTAINS

      This should create the Sample_Biz, then the Sample_App, then the Sample_Server, and then put the App into the Biz, and the Server into the App.

           

      My problem is that when I run the ETL, these objects are getting created but put into "Newly Discovered" rather than the 'All Domain' hierarchy under the domain i've specified in the ETL. 


      How can I get these objects to appear in the proper Domain without having to manually move each of them?


      I'm looking for this relationship:

      Sample_Biz -> Sample_App -> Sample_Server


      Thanks for any ideas!

        • 1. Re: problem using OBJREL and CSV Imports to create hierarchies

          Craig,

           

          Check out the sample OBJREL below.  I see a couple issues with your example:

           

          1. There is no _ROOTAPP_ association which probably explains why the entities are still in "Newly Discovered".  When you create the ETL, there is an "Object Relationship" section which allows you to specify which domain the data gets imported into in the Workspace.  The domain that is specified in this section becomes "_ROOTAPP_".
          2. No need to use FROMTS and TOTS in this case
          3. ENTCATNMPARENT is the parent entity type (APP, WKLD, SYS).  CONTAINS is not valid.

           

          Hope this helps!

           

          TS;CHANGETYPE;ENTCATNM;ENTNM;DS_ENTNM;ENTCATNMPARENT;DS_ENTNMPARENT;ENTTYPENM

          2015-05-01 00:00:00;ASSERT;APP;BusinessServiceA;BusinessServiceA;;;svc

          2015-05-01 00:00:00;ASSERT;APP;BusinessServiceA;BusinessServiceA;APP;_ROOTAPP_;

          2015-05-01 00:00:00;ASSERT;SYS;appserver201;appserver201.mydomain.com;APP;BusinessServiceA

          2015-05-01 00:00:00;ASSERT;SYS;appserver202;appserver202.mydomain.com;APP;BusinessServiceA

          2015-05-01 00:00:00;ASSERT;SYS;appserver203;appserver203.mydomain.com;APP;BusinessServiceA

          2015-05-01 00:00:00;ASSERT;APP;BusinessServiceB;BusinessServiceB;;;svc

          2015-05-01 00:00:00;ASSERT;APP;BusinessServiceB;BusinessServiceB;APP;_ROOTAPP_;

          2015-05-01 00:00:00;ASSERT;SYS;appserver204;appserver204.mydomain.com;APP;BusinessServiceB

          2015-05-01 00:00:00;ASSERT;SYS;appserver205;appserver205.mydomain.com;APP;BusinessServiceB

          2015-05-01 00:00:00;ASSERT;SYS;appserver206;appserver206.mydomain.com;APP;BusinessServiceB

          • 2. Re: problem using OBJREL and CSV Imports to create hierarchies

            Thanks Brad, this helps a lot!

            A couple of puzzling things:


            1) When I use your code, the two business services are created in the right Domain which is great, but not the servers (appserver201, etc). Any idea on why that might be?


            2) About an hour after I ran the ETL to create the new objects, they disappeared from my Domain. When I go into the Hierarchy view and select "Show Historical Relatoins", the two Business Services show up, their "to" timestamp is set to roughly an hour after I created them. This has happened to me with other objects I've created, which is why I was originally trying to use FROMTS and TOTS... I don't really want the object relationships I'm creating to ever expire.


            3) Is the ENTTYPENM field completely optional, or do you have to use it when specifying a top level object in the Domain?


            Thanks as always!


            Craig

            • 3. Re: problem using OBJREL and CSV Imports to create hierarchies
              1) When I use your code, the two business services are created in the right Domain which is great, but not the servers (appserver201, etc). Any idea on why that might be?


              My fault for not pointing this out, but my example assumes that the server entities already exist.  I omitted the lines that create the actual entities (since they were created by another ETL) and am simply creating the relationships

               

              2) About an hour after I ran the ETL to create the new objects, they disappeared from my Domain. When I go into the Hierarchy view and select "Show Historical Relatoins", the two Business Services show up, their "to" timestamp is set to roughly an hour after I created them. This has happened to me with other objects I've created, which is why I was originally trying to use FROMTS and TOTS... I don't really want the object relationships I'm creating to ever expire.

               

              Not sure what is causing this.  I may have to defer to someone else in the community on this one.

               

              3) Is the ENTTYPENM field completely optional, or do you have to use it when specifying a top level object in the Domain?

               

              I believe ENTTYPENM is optional (I would need to test to be sure) but if nothing is specified, it should use "gen" for Generic.  The "svc" ENTTYPENM in my example ensures the top level entity is a Business Service instead of a Generic Domain.

              • 5. Re: problem using OBJREL and CSV Imports to create hierarchies
                Renato Bonomini

                Oh Brad, how could you forget to instantiate the entities, shame on you

                 

                Things to remember that Gaspare Provenzano taught us when OBJREL was created

                • always declare entities - if not, systems that don't exist won't be created
                • declare entities before relationships
                • always associate each top level domain to ROOTAPP >> this is mistake #1

                 

                On the last bullet, 99% of the time people say "it ran correctly but I don't see my data" this is because the top level domains are not associated to ROOTAPP

                • 6. Re: problem using OBJREL and CSV Imports to create hierarchies

                  Ah, OK - I should have caught that about the server entities not being created. Thanks! Will have to keep looking into why my newly created objects are "expiring" after an hour.

                  • 8. Re: problem using OBJREL and CSV Imports to create hierarchies

                    The meaning of the relationship assertion with the "TS" field is that the relationship should start at that time and never end. I am puzzled why using the TS field is causing the relationship to expire after an hour in your case. You should open a Customer Support issue.

                     

                    The FROMTS, TOTS columns were introduced in 9.5 for a different use case where one is importing historical data. They are not really recommended.

                    • 9. Re: problem using OBJREL and CSV Imports to create hierarchies

                      Thank you Sudheer, this is very helpful to know that FROMTS and TOTS are not generally recommended!

                       

                      I began setting them when my test import was not working and the documentation for the OBJREL dataset listed FROMTS and TOTS as mandatory. Dataset reference - BMC TrueSight Capacity Optimization 10.3 - BMC Documentation

                       

                      I've created a new ETL linked to a new Domain, and now objects are sticking around and not expiring- so perhaps one of my many failed attempts was causing some problem.

                       

                      I'll steer clear of FROMTS and TOTS from now on!

                       

                      Thanks,

                       

                      Craig

                      • 10. Re: problem using OBJREL and CSV Imports to create hierarchies

                        Hello all

                         

                        I have problem with building third level tree, OBJREL not associating virtual machines to cluster.

                        Can anyone help where is I mistook?

                         

                        For example. I need to build:

                         

                        1lvl. Support (contains cluster)

                        2lvl. --  appsrv104;appsrv104.domain.com (VM Cluster, contains 2 VM)

                        3lvl. -- -- appsrv105.domain.com;SYS;appsrv104 (VM)

                        3lvl. -- -- appsrv106;appsrv106.domain.com (VM)

                         

                        My csv-file example:

                         

                        TS;CHANGETYPE;ENTCATNM;ENTNM;DS_ENTNM;ENTCATNMPARENT;DS_ENTNMPARENT;ENTTYPENM;RELATIONTYPE

                        2016-01-01 00:00:00;ASSERT;APP;Support;Support;;;mf;CONTAINS

                        2016-01-01 00:00:00;ASSERT;SYS;appsrv104;appsrv104.domain.com;;;vhc;VHC_CONTAINS_GM

                        2016-01-01 00:00:00;ASSERT;SYS;appsrv105;appsrv105.domain.com;;;gm;

                        2016-01-01 00:00:00;ASSERT;SYS;appsrv106;appsrv106.domain.com;;;gm;

                        2016-01-01 00:00:00;ASSERT;APP;Support;Support;APP;_ROOTAPP_;CONTAINS

                        2016-01-01 00:00:00;ASSERT;SYS;appsrv104;appsrv104.domain.com;APP;Support;vhc;VHC_CONTAINS_GM

                        2016-01-01 00:00:00;ASSERT;SYS;appsrv105;appsrv105.domain.com;SYS;appsrv104;;gm;

                        2016-01-01 00:00:00;ASSERT;SYS;appsrv106;appsrv106.domain.com;SYS;appsrv104;;gm;

                        ~