1 2 Previous Next 17 Replies Latest reply on Sep 12, 2018 3:10 AM by Alvaro Paronuzzi

    Wildcard characters use in dynamic enrichment data tables

    Alvaro Paronuzzi

      Hello everyone,

       

      is it possible to use wildcard characters in dynamic enrichment data tables in order to make the management of these tables easier?

       

      Example:

       

      E:\Program Files\BMC Software\TrueSight\pw\server\etc\pncell_tsim\kb\data\mydatatable.baroc

       

      MYTABLE;

          mc_host =  remedy1.lan;

          mc_object = Remedy Server;

      END

       

      MYTABLE;

          mc_host =  remedy2.lan;

          mc_object = Remedy Server;

      END

       

      MYTABLE;

          mc_host =  remedy3.lan;

          mc_object = Remedy Server;

      END

       

      ....

       

      In this case every event with mc_host equal to remedy1.lan, remedy2.lan or remedy3.lan (and so on..) is currently enriched with mc_object = Remedy Server (I'm doing this using an mrl rule which reads the data table which is actually a little more complex than the one I put in the example..).

       

      What I would like to use is a single entry in my baroc file like the one below:

       

      MYTABLE;

          mc_host =  remedy*.lan;

          mc_object = Remedy Server;

      END

       

       

      Is it possible to do that in TSIM/BPPM and have it working in the same way described above?

       

      Thank you in advance for any help you can provide.

       

      Alvaro Paronuzzi

        • 1. Re: Wildcard characters use in dynamic enrichment data tables
          Alvaro Paronuzzi

          Probably this table is used for doing something similar to what I'm trying to do, but from the documentation I could not understand how to use it (or replicate it for my use case)

           

          Thank you in advance for you help.

          Al

          • 2. Re: Wildcard characters use in dynamic enrichment data tables
            Stewart Crighton

            The answer is yes.  However; it depends on how you right your MRL.

             

            In your original post you are referring to a Data Table.  In the update you are referring to a Match Table.  Either one could be used depending on what it is you want to accomplish.

             

            If you put what your rule is doing, I could assist with the modifications to use either scenario above.

             

            The basics will look something like this though:

             

            new rule_name :

               EVENT($EV) where [ $EV.status == OPEN ]

                using

                 {

                   MY_TABLE($MT)

                     where

                       [ $EV.mc_host contains $MT.mc_host ]

                  }

            ...

             

            or...

             

            This will need to be in your code:

             

                    find_match_entry( "MY_MATCH_TABLE", lookup_table_tag,

                                      $EV.mc_host,

                                      $MATCH );

            • 3. Re: Wildcard characters use in dynamic enrichment data tables
              Brendan Murray

              Hi Al,

               

              As Stewart says in his post, the answer to your question depends on what kind of data table we are talking about. In TSIM there are regular data tables that are just sub-classes of the DATA class, but there are also special data tables known as 'match tables'. These are sub-classes of the BEM_MATCH_TABLE class and they have unique features in terms of how records are matched. TSIM's Dynamic Enrichment Policies use match tables, not standard data tables. Among other things, match tables allow you to use the * character in the data as a wildcard. Note that * is the only wildcard character allowed and it has limitations which I will explain later.

               

              Here are the possible ways to address your requirement:

               

              1. USING REGULAR DATA TABLES AND MRL

               

              If you are using regular data tables, I don't believe there is any support for using wildcard characters in the data table. The data in the table is assumed be literal so any special characters, such as *, are interpreted literally, rather than as wildcards. Your table structure, table content and MRL code would all have to be designed to support your specific use cases. For example, if you need to match events where the hostname starts with remedy and has a domain of 'lan', I would have two fields in my table: hostname and domain. A sample record would look like this:

               

              MYTABLE;

                  mt_hostname =  remedy;

                  mt_domain = lan;

                  mc_object = Remedy Server;

              END

               

              Your MRL code would perform this test (my syntax may not be exact):

               

              $EV.mc_host has_prefix $DT.mt_hostname AND $EV.mc_host has_suffix $DT.mt_domain

               

               

              USING DYNAMIC EVENT MANAGEMENT POLICIES WITH MATCH TABLES

               

              As I mention above, TSIM's 'dynamic' policy types use match tables, not standard data tables. Match tables are sub-classes of BEM_MATCH_TABLE. One of the properties of match tables is that they allow you to put an * in the table data itself. This is because the 'input match' fields in a match table are interpreted as patterns, not literal data (see the links to the documentation below). However, there are limits to how the wildcard can be used. The allowed uses are:

               

              Table Content     Match Behavior

              '*'                          Any string

              '<pattern>'           Exact pattern

              '<pattern>*'          Strings with prefix pattern

              '*<pattern>'          Strings with suffix pattern

              '*<pattern>*'          Strings containing pattern

               

              Note that the angle brackets (< and >) are required by the cell. The .CSV files you use to load the data into the cell do not need to have the angle brackets. In your screen shot above, you are looking at the cell syntax. The 'Import' function in the Event Management Policies UI will convert the .CSV data into the cell format automatically. You can see the corresponding .CSV syntax in the source file for the table in your screen shot here:

               

              %BMC_PROACTIVENET_HOME%\admin\etc\samples\BPM_Category_Table.csv

               

              These links will take you to the documentation on BEM MATCH TABLES and their pattern matching behavior.

               

              You can see from the table above that the asterisks (*) in the match table patterns must be at the beginning of the pattern, the end of the pattern or both. This means remedy*.lan would not be a valid pattern in a match table. To address this requirement, you would need two dynamic enrichment policies, one to test the hostname and a second to test the domain.

               

              Policy 1 would have this data in the table (note that I am using the syntax allowed in the source .CSV file, i.e. no angle brackets):

               

              Input Match     Output

              *remedy           remedy

               

              The output value would go in a user-defined slot that is tested by the second policy, along with the the value of the domain.

               

              Input Match     Output

              remedy,*lan     'Remedy Server'

               

              The match fields in your second policy would be, in this order: your_user-defined slot, mc_host. So, your first policy establishes that the hostname begins with 'remedy' and updates the event using a custom slot to indicate this. Your second policy must match the entire custom slot and the hostname in the event must end with the 'lan' domain.

               

               

              USING MRL WITH MATCH TABLES

               

              If you prefer to use MRL rather than Event Management Policies, MRL provides a set of primitives/functions designed to use match tables.

               

              I hope this helps.

               

              Regards,

               

              Brendan

              3 of 3 people found this helpful
              • 5. Re: Wildcard characters use in dynamic enrichment data tables
                Garland Smith

                Wow!  Great writeup!  This is awesome!  If this is not already available in the product documentation, it would be great to add!  I’d love to see more practical examples in the documentation.

                 

                 

                 

                I watch the communities posts like a hawk, always looking to gain some insight or useful tidbits.  Your posts are among the most useful and informative!

                 

                 

                 

                Garland Smith

                • 6. Re: Wildcard characters use in dynamic enrichment data tables
                  Alvaro Paronuzzi

                  Brendan, Stewart,

                   

                  thank you for your help!

                   

                  I can make my request clearer now:

                  1. USING REGULAR DATA TABLES AND MRL

                  This is the common use of data tables that I'm doing. I understand form Brendan's response that the entries on these tables are literal, so I cannot use wildcards here.

                   

                  Match tables are probably what I need to achieve my goal, which is for example setting "Remedy Server" inside the mc_object slot to all the events having mc_host = remedy*.lan  (e.g. remedy1.lan, remedy2.lan, etc.)

                   

                  As I am more familiar with MRL, I think this will be the option to follow:

                   

                  USING MRL WITH MATCH TABLES

                   

                   

                  I will take a look to the documentation you shared and surely come back shortly with another couple of questions.

                   

                  Thank you,

                  Al

                  • 7. Re: Wildcard characters use in dynamic enrichment data tables
                    Alvaro Paronuzzi

                    The use case seems a little bit more complex and, as per my understanding of the documentation, managing it with match tables + policies (or mrl rules)may be painful.

                     

                    Let's see the full picture:

                    I have a few event slot values and their combination should lead to URL that needs to be added into a slot.

                     

                    Example:

                     

                    Company, Service, mc_host, mc_object, mc_parameter, url

                    Mycompany, myservice, g*bk*.lan, B*, *,http://www.pearljam.com

                    Mycompany2, myservice2, myhost, myobject, *G*,http://www.aerosmith.com

                     

                    What I need to do is adding the correct URL to the event according to the event slot values mentioned in the example above.

                     

                    According to your experience, do you think it can be achieved using policies (or mrl) + match tables?

                     

                    As per my current experience I could do this enrichment using a static data table with all literal values ("listed"), but the high number of potential matching cases suggested the customer to try using wildcards and so I'm trying to see if it's feasable or not because I am not very familiar with match tables and their use.

                     

                    Thank you,

                    Al

                    • 8. Re: Wildcard characters use in dynamic enrichment data tables
                      Stewart Crighton

                      So what you are looking for is a bit more advanced than a generic prefix.  There are multiple wild cards nested in your search strings.

                       

                      Personally I would switch over to a match_regex search.  However; BMC support will point out that there is an overhead to using regex.  So be cautious for performance issues.

                       

                      One final note... since we are on the topic of Data Tables vs Match Tables...  Please note that match tables have a really cool feature in MRL called apply_match_entry where the output_expressions field is used  to create strings.  This field can contain basic MRL that outputs a string.  That string can be used later in the MRL.  I use this frequently to dynamically modify slots based on data in a table.  I am not sure that can be really used here at this time.  However; it might be useful in the future for you.

                       

                       

                      Please note that the following is just a quick free hand typing of how I would approach this.  I did not compile or test any of this.

                       

                      Using just a standard Data Table....

                       

                      Rules:

                      # Toss all the data needed into a single field.  This will reduce the number of regex searches later

                      refine url_search_pattern : EVENT($EV)
                      where [ $EV.status not_equals CLOSED ]
                      {
                        $EV.url = sprintf("%s|%s|%s|%s|%s",[$EV.Company,$EV.Service,$EV.mc_host,$EV.mc_object,$EV.mc_parameter]);

                      }
                      END

                       

                      # Find the entries, and use the data found to update the url slot.

                      refine update_url : EVENT($EV)
                      where [ $EV.status not_equals CLOSED ]
                      using
                      {
                        CUST_URL_TABLE($CUT)
                        {
                          match_regex($EV.url,$CUT.regex,$CUT.options)
                        }
                      }
                      {
                        $EV.url = $CUT.url;
                      }
                      END

                       

                      Classes entries:

                      MC_DATA_CLASS : CUST_URL_TABLE ISA DATA DEFINES
                      {
                              url      : STRING;
                              regex    : STRING;
                              options  : STRING;
                      };
                      END

                       

                      Sample Data:

                      CUST_URL_TABLE;    
                              url='http://www.pearljam.com';
                              regex='Mycompany|myservice|g.*bk.*.lan|B.*|.*';
                              options='';
                      END
                      CUST_URL_TABLE;    
                              url='http://www.aerosmith.com';
                              regex='Mycompany|myservice|myhost|myobject|.*G.*';
                              options='';
                      END

                       

                       

                      The following would be using a Match Table.  However; it really only works on your second sample data.  This is because Match tables use a very basic search of is exactly, has_prefix, has_suffix, contains, total wild card....

                       

                      Rule:

                      refine update_url : EVENT($EV)
                      where [ $EV.status not_equals CLOSED ]
                      {
                         if find_match_entry(
                            CUST_URL_MATCH_TABLE,
                            'FindURL',
                            [ $EV.Company, $EV.Service, $EV.mc_host, $EV.mc_object, $EV.mc_parameter ],
                            $MATCH )
                         then
                         {
                            opadd($EV, '', 'Updating URL using record: ' || $MATCH.data_handle ) ;
                            $EV.url = $MATCH.url ;
                         };
                      }
                      END

                       

                      Classes Entries:

                      MC_DATA_CLASS : CUST_MATCH_TABLE ISA BEM_MATCH_TABLE
                      DEFINES
                      {
                         notes : STRING ;
                         input_match_reference : LIST_OF STRING ;
                      };
                      END

                      MC_DATA_CLASS : CUST_URL_MATCH_TABLE ISA CUST_MATCH_TABLE
                      DEFINES
                      {
                         tag : default = 'FindURL' ;
                         input_match : default = [ '<Company>', '<Service>', '<mc_host>', '<mc_object>', '<mc_parameter>' ] ;
                         url : STRING ;
                         input_match_reference : default = [ '<Company>', '<Service>', '<mc_host>', '<mc_object>', '<mc_parameter>' ] ;
                      };
                      END

                       

                      Sample Data:

                      CUST_URL_MATCH_TABLE;
                              input_match=['<Mycompany>', '<myservice>', '<myobject>', '*<G>*'];
                              url='http://www.aerosmith.com'
                      END

                      1 of 1 people found this helpful
                      • 9. Re: Wildcard characters use in dynamic enrichment data tables
                        Alvaro Paronuzzi

                        I adapted the first example to my use case and I feel I'm pretty close now....but I still cannot go beyond the last compilation error saying:

                         

                        BMC-IMC120089E: BMC-IMC031001E: Error at line 30, column 4: BMC-IMC031141E: Expected class name

                        BMC-IMC120018E: Parse error for rule 1 in file C:/BMCSOF~1/PROACT~1/pw/server/etc/pncell_bppm-prepro/kb/rules/sklAddProcedureToServiceEv.mrl

                         

                        Here's the mrl adapted from your example (the compilation error is returned on the highlighted line):

                         

                        # Toss all the data needed into a single field.  This will reduce the number of regex searches later

                         

                        refine skl_url_search_pattern : SERVICE_EV($EV)

                        where [ $EV.status not_equals CLOSED ]

                        {

                          $EV.skl_sdv_procedureurl = sprintf("%s|%s|%s",[$EV.itsm_company,$EV.mc_service,$EV.mc_object]);

                         

                        }

                        END

                         

                         

                         

                        # Find the entries, and use the data found to update the url slot.

                         

                        refine skl_update_url : SERVICE_EV($EV)

                        where [$EV.status not_equals CLOSED]

                        using

                        {

                          SKL_PROCEDUREURL_TABLE($TABLE)

                          {

                            match_regex($EV.skl_sdv_procedureurl,$TABLE.regex,$TABLE.options)

                          }

                        }

                        {

                          $EV.skl_sdv_procedureurl = $TABLE.url;

                        }

                        END

                         

                         

                        The SKL_PROCEDUREURL_TABLE data class is defined as:

                         

                        MC_DATA_CLASS:

                            SKL_PROCEDUREURL_TABLE ISA SKL_DATA DEFINES

                            {

                                url      : STRING;

                                regex    : STRING;

                                options  : STRING;

                            };

                        END

                         

                         

                        And the SKL_DATA data class definition (already existing) is:

                         

                        MC_DATA_CLASS :

                            SKL_DATA ISA DATA ;

                        END

                         

                         

                        I currently cannot see what I'm missing. Thank you in advance for any help you can provide.

                         

                        Al

                        • 10. Re: Wildcard characters use in dynamic enrichment data tables
                          Stewart Crighton

                          Greetings –

                           

                          So, let’s get the obvious stuff out of the way….

                           

                          You created the definition of that table in a file under the classes folder?

                          Did you add that file to the .load file in the classes folder?

                           

                          I would then look at the rule itself…  might just be something odd with the missing where statement:

                           

                           

                          refine skl_update_url : SERVICE_EV($EV)

                           

                          where

                           

                          using

                           

                          {

                           

                            SKL_PROCEDUREURL_TABLE($TABLE)

                           

                            where

                           

                          }

                           

                          {

                           

                            $EV.skl_sdv_procedureurl = $TABLE.url;

                           

                          }

                           

                          END

                          • 11. Re: Wildcard characters use in dynamic enrichment data tables
                            Alvaro Paronuzzi

                            I added a where condition but unfortunately the error returned is the same.

                             

                            I'm not 100% of the use of match_regex primitive:

                            but my understanding is that $STR is the event slot value composed by the previous rule (using sprintf) and $REGEX is the record in the table containing wildcards...but what is the output of this primitive?

                             

                            I have one more question: according to Brendan's reply:

                             

                            If you are using regular data tables, I don't believe there is any support for using wildcard characters in the data table. The data in the table is assumed be literal so any special characters, such as *, are interpreted literally, rather than as wildcards.

                             

                            may a record like this one not be used as expected (using * as wildcard)?:

                             

                            SKL_PROCEDUREURL_TABLE;   

                                    url='https://mydomain.com:8443/Procedure2';

                                    regex='COMPANY|Service|.*B.*';

                                    options='';

                            END

                             

                            Thank you,

                            Al

                            • 12. Re: Wildcard characters use in dynamic enrichment data tables
                              Stewart Crighton

                              So I just put this onto a test cell.  It works as expected.

                               

                              Double check that the classes .load is updated correctly.

                               

                              The first question is a fair one…  I do not know is the basic answer… but it works.

                               

                              For the second question.  The string in the table is a literal string that happens to include * as part of it.

                               

                              How that literal string is used though is the key.  Since we are using it in a match_regex that changes it to be more dynamic.

                              • 13. Re: Wildcard characters use in dynamic enrichment data tables
                                Alvaro Paronuzzi

                                Thank you for your test.

                                This should guarantee that the class is correctly loaded and the records have been loaded as well, right?

                                I really don't understand what this compiler wants from me..

                                 

                                Could you please share the version of the rule that you've been able to compile successfully? It would make the debug quicker..

                                 

                                Thank you,

                                Al

                                • 14. Re: Wildcard characters use in dynamic enrichment data tables
                                  Stewart Crighton

                                  Of course…  I made minor modifications, as I do not have your special event class…..

                                   

                                   

                                  1. Toss all the data needed into a single field.  This will reduce the number of regex searches later

                                   

                                  refine skl_url_search_pattern : EVENT($EV)

                                  where

                                  {

                                    $EV.msg = sprintf("%s|%s|%s",[$EV.itsm_company,$EV.mc_service,$EV.mc_object]); # e.g. SKYLOGIC|Tooway on KA-SAT|B08

                                   

                                  }

                                  END

                                   

                                   

                                  1. Find the entries, and use the data found to update the url slot.

                                  refine skl_update_url : EVENT($EV)

                                  where

                                  using

                                  {

                                    SKL_PROCEDUREURL_TABLE($TABLE)

                                    where [

                                      match_regex($EV.msg,$TABLE.regex,$TABLE.options)

                                    ]

                                  }

                                  {

                                    $EV.msg = $TABLE.url;

                                  }

                                  END

                                   

                                   

                                   

                                  MC_DATA_CLASS :

                                      SKL_DATA ISA DATA ;

                                  END

                                   

                                  MC_DATA_CLASS:

                                      SKL_PROCEDUREURL_TABLE ISA SKL_DATA DEFINES

                                      ;

                                  END

                                  1 of 1 people found this helpful
                                  1 2 Previous Next