12 Replies Latest reply on Nov 27, 2019 9:10 AM by Alvaro Paronuzzi

    Unexpected event correlation

    Alvaro Paronuzzi
      Share This:

      Hello MRL experts,

       

      I'm seeing an unexpected behavior with the following MRL rule:

       

      correlate sklAutoBulkCreationDM:

      SKL_EV($NEW)

      where [

      $NEW.CLASS == DATAMINER_EV

      AND

      $NEW.status == OPEN # Not related to any event

      AND

      $NEW.skl_sdv_procedureurl == 'dadada'

      AND

      $NEW.skl_platform_owner == TEST

      AND

      $NEW.skl_hide_autobulk != YES

      ]

      with SKL_EV($OLD)

      where [

      $OLD.CLASS == DATAMINER_EV

      AND

      $OLD.status == OPEN

      AND

      $OLD.skl_host == $NEW.skl_host

      AND

      $OLD.skl_sdv_procedureurl == $NEW.skl_sdv_procedureurl

      AND

      $OLD.skl_platform_owner == $NEW.skl_platform_owner

      AND

      $OLD.skl_hide_autobulk != YES  #Not hidden because of an existing AUTOBULK

      ]

      within 1 m #M parameter of the requirements

      when $NEW.event_handle #Static

      {

            concat (['[AutoBulk] ',$OLD.mc_host,' is ',$OLD.severity], $MSG);

       

            generate_event (AUTOBULK_EV, [ mc_host = $OLD.mc_host, mc_object = 'multiple', mc_parameter = 'multiple', severity = $OLD.severity, mc_priority = PRIORITY_2, msg = $MSG, skl_sdv_procedureurl = $OLD.skl_sdv_procedureurl, skl_sdv_service_group = $OLD.skl_sdv_service_group, skl_sdv = 'Yes', skl_assigned_group = $OLD.skl_assigned_group, skl_owner_group = $OLD.skl_owner_group, mc_service = $OLD.mc_service, skl_sdv_event_visibility = $OLD.skl_sdv_event_visibility, mc_tool='TrueSight (event.lan)',skl_platform_owner=$OLD.skl_platform_owner, mc_host_class=$OLD.mc_host_class, mc_tool_suggestion='multiple Impacted Services', itsm_company=$OLD.itsm_company, itsm_category=$OLD.itsm_category, itsm_type=$OLD.itsm_type, itsm_item=$OLD.itsm_item, itsm_operational_category1=$OLD.itsm_operational_category1, itsm_operational_category2=$OLD.itsm_operational_category2, itsm_operational_category3=$OLD.itsm_operational_category3]);

           

            #trace operation

            concat(['EV#',$NEW.event_handle,' was automatically related to EV#',$OLD.event_handle,' and triggered the creation of a new AutoBulk event'],$DETAIL);

            opadd($NEW,'',$DETAIL,'');

           

            unset_cause; # After the creation of the AUTOBULK_EV the relationship between the 2 events is no longer needed

      }

      END

       

      The expected behavior is to see an AUTOBULK_EV created when the second DATAMINER_EV arrives and gets correlated to the first DATAMINER_EV if the second one arrived within 1 minutes from the arrival of the first one.

       

      What I'm seeing instead is...

       

      Event 2 correlated to Event 1 creating Event 3 (AUTOBULK_EV)

      +

      Event 1 correlated to Event 2 creating Event 4 (AUTOBULK_EV)

       

      So I have the creation of an additional AUTOBULK_EV, which is not expected. I believe the correlation between Event 1 and Event 2 (the second correlation) should not happend.

       

      In the test environment I have never seen this unexpected behavior and the rule used in the production environment is the same...  :-s

       

      Could anyone see the issue which is causing this unexpected behavior?

      Unfortunately I don't have the issue traced in cell logs because, if I enable the cell tracing, the issue disappears, so it's a quite weird investigation...

       

      Product version: TSIM 10.7 patch 3.

       

      Thank you in advance for your help.

       

      Kind Regards,

      Alvaro Paronuzzi

        • 1. Re: Unexpected event correlation
          Philippe Plomteux

          Alvaro,Not 100% sure about the reason of this behaviour...But if you are not really interested in keeping the events correlated...why don't you use a simpler rule of type "new" ?Easier to write, read and debug for this rather simple use case ?Just 2 cents.Philippe

           

           

            Le mer., oct. 23, 2019 à 15:26, Alvaro Paronuzzi<communities_update@bmc.com> a écrit :   

           

          BMC Communities

           

          |

          Unexpected event correlation

           

          created by Alvaro Paronuzzi in TrueSight Infrastructure Mgmt - View the full discussion

           

          Hello MRL experts,

           

           

           

          I'm seeing an unexpected behavior with the following MRL rule:

           

           

           

          correlate sklAutoBulkCreationDM:

           

          SKL_EV($NEW)

           

          where [

           

          $NEW.CLASS == DATAMINER_EV

           

          AND

           

          $NEW.status == OPEN # Not related to any event

           

          AND

           

          $NEW.skl_sdv_procedureurl == 'dadada'

           

          AND

           

          $NEW.skl_platform_owner == TEST

           

          AND

           

          $NEW.skl_hide_autobulk != YES

           

          ]

           

          with SKL_EV($OLD)

           

          where [

           

          $OLD.CLASS == DATAMINER_EV

           

          AND

           

          $OLD.status == OPEN

           

          AND

           

          $OLD.skl_host == $NEW.skl_host

           

          AND

           

          $OLD.skl_sdv_procedureurl == $NEW.skl_sdv_procedureurl

           

          AND

           

          $OLD.skl_platform_owner == $NEW.skl_platform_owner

           

          AND

           

          $OLD.skl_hide_autobulk != YES  #Not hidden because of an existing AUTOBULK

           

          ]

           

          within 1 m #M parameter of the requirements

           

          when $NEW.event_handle #Static

           

          {

           

                concat (['[AutoBulk] ',$OLD.mc_host,' is ',$OLD.severity], $MSG);

           

           

           

                generate_event (AUTOBULK_EV, );

           

               

           

                #trace operation

           

                concat(,$DETAIL);

           

                opadd($NEW,'',$DETAIL,'');

           

               

           

                unset_cause; # After the creation of the AUTOBULK_EV the relationship between the 2 events is no longer needed

           

          }

           

          END

           

           

           

          The expected behavior is to see an AUTOBULK_EV created when the second DATAMINER_EV arrives and gets correlated to the first DATAMINER_EV if the second one arrived within 1 minutes from the arrival of the first one.

           

           

           

          What I'm seeing instead is...

           

           

           

          Event 2 correlated to Event 1 creating Event 3 (AUTOBULK_EV)

           

          +

           

          Event 1 correlated to Event 2 creating Event 4 (AUTOBULK_EV)

           

           

           

          So I have the creation of an additional AUTOBULK_EV, which is not expected. I believe the correlation between Event 1 and Event 2 (the second correlation) should not happend.

           

           

           

          In the test environment I have never seen this unexpected behavior and the rule used in the production environment is the same...  :-s

           

           

           

          Could anyone see the issue which is causing this unexpected behavior?

           

          Unfortunately I don't have the issue traced in cell logs because, if I enable the cell tracing, the issue disappears, so it's a quite weird investigation...

           

           

           

          Product version: TSIM 10.7 patch 3.

           

           

           

          Thank you in advance for your help.

           

           

           

          Kind Regards,

           

          Alvaro Paronuzzi

           

          Give back to the community: Login and mark as Helpful or Correct.

          Start a new discussion in TrueSight Infrastructure Mgmt by email or at BMC Communities

          Following TrueSight Infrastructure Mgmt in these streams: Email Watches

           

           

           

           

           

           

          Thank you for your participation in BMC Communities.

          • 2. Re: Unexpected event correlation
            Alvaro Paronuzzi

            Hi Philippe,

            Thank you for your response.

            I'm keeping the relationship just as long as the AUTOBULK_EV event is created.

            I am missing how to create the new event depending on the slot values of the two events arrived within a defined timeframe within the new phase. I'm probably missing what should be used instead of the "correlate / with" couple inside the new phase.

            Thank you in advance for your help.

             

            Alvaro

            • 3. Re: Unexpected event correlation
              Philippe Plomteux

              Am I oversimplifying if I propose the following ?

              The action block will execute when the two events have arrived within a minute of each other.

               

              new sklAutoBulkCreationDM:

              SKL_EV($NEW)

              where [ $NEW.CLASS == DATAMINER_EV AND

              $NEW.status == OPEN AND

              $NEW.skl_sdv_procedureurl == 'dadada' AND

              $NEW.skl_platform_owner == TEST AND

              $NEW.skl_hide_autobulk != YES

              ]

              updates SKL_EV($OLD)

              where [

              $OLD.CLASS == DATAMINER_EV AND

              $OLD.status == OPEN AND

              $OLD.skl_host == $NEW.skl_host AND

              $OLD.skl_sdv_procedureurl == $NEW.skl_sdv_procedureurl AND

              $OLD.skl_platform_owner == $NEW.skl_platform_owner AND

              $OLD.skl_hide_autobulk != YES  #Not hidden because of an existing AUTOBULK

              ]

              within 1 m #M parameter of the requirements

              {

                    concat (['[AutoBulk] ',$OLD.mc_host,' is ',$OLD.severity], $MSG);

                    generate_event (AUTOBULK_EV, [ mc_host = $OLD.mc_host, mc_object = 'multiple', mc_parameter = 'multiple', severity = $OLD.severity, mc_priority = PRIORITY_2, msg = $MSG, skl_sdv_procedureurl = $OLD.skl_sdv_procedureurl, skl_sdv_service_group = $OLD.skl_sdv_service_group, skl_sdv = 'Yes', skl_assigned_group = $OLD.skl_assigned_group, skl_owner_group = $OLD.skl_owner_group, mc_service = $OLD.mc_service, skl_sdv_event_visibility = $OLD.skl_sdv_event_visibility, mc_tool='TrueSight (event.lan)',skl_platform_owner=$OLD.skl_platform_owner, mc_host_class=$OLD.mc_host_class, mc_tool_suggestion='multiple Impacted Services', itsm_company=$OLD.itsm_company, itsm_category=$OLD.itsm_category, itsm_type=$OLD.itsm_type, itsm_item=$OLD.itsm_item, itsm_operational_category1=$OLD.itsm_operational_category1, itsm_operational_category2=$OLD.itsm_operational_category2, itsm_operational_category3=$OLD.itsm_operational_category3]);

                    #trace operation

                    concat(['EV#',$NEW.event_handle,' was automatically related to EV#',$OLD.event_handle,' and triggered the creation of a new AutoBulk event'],$DETAIL);

                    opadd($NEW,'',$DETAIL,'');

               

              }

              END

              1 of 1 people found this helpful
              • 4. Re: Unexpected event correlation
                Alvaro Paronuzzi

                Thank you, Philippe.

                I will surely give a try with the rule moved to the new phase but my concern is that this unexpected behavior may occur also in that phase.

                As the "duplicated" relationship between the two events is not expected to be established (EV#2 correlates with EV#1 and EV#1 correlates with EV#2), I'm afraid the same may happen with the update (EV#1 updates EV#2 and EV#2 updates EV#1).

                The behavior is unexpected and cannot reproduced "on demand". In addition, I've never seen it in the test environment, neither after the release of this rule in the production environment when we temporary enabled the forwarding of the same events to the prod cell and the test cell.

                It may seem some sort of performance issue, but as far as I'm not able to reproduce the issue when the cell tracing is turned on it's hard to really understands what's happening and why the unexpected behavior is occurring.

                 

                Thank you,

                Alvaro

                • 5. Re: Unexpected event correlation
                  Philippe Plomteux

                  You cannot rule out a product issue of course, but I would in any case rate a "new" rule safer to run/build/understand and perhaps more "relevant" in this particular context.

                  KR

                  • 6. Re: Unexpected event correlation
                    Alvaro Paronuzzi

                    Hi Philippe,

                    I moved the rule to the "new" phase and I'm no longer seen the issue. On the other hand, unfortunately I'm seeing that, after the rule was moved from the Correlate phase to the New phase, the within clause seems to be ignored.

                    Here's an example of the use case:

                     

                    EV#1 arrives.

                    After more than 1 minute (which is the value in the within clause) EV#2 arrives and the creation of the AUTOBULK_EV event is triggered.

                     

                    I expect the AUTOBULK_EV event to be created only if EV#2 arrives within 1 minute from the arrival of EV#1. Is my expectation correct? If so, why is it happening also if the second events arrives after 2 or 3 minutes?

                     

                    Thank you in advance for your help.

                    Alvaro

                    • 7. Re: Unexpected event correlation
                      Philippe Plomteux

                      Alvaro,

                       

                      Can you share your latest code (the new rule in question) ?

                       

                      I have not tested the within clause for a while but it should indeed limit the search time window to whatever you provide there.

                       

                      Alternatively, you could set a condition in the ECF of the old event…Something like [ $EV.mc_arrival_time - $OLD.mc_arrival_time < 60 ]

                       

                      KR

                       

                      Philippe

                       

                       

                       

                       

                       

                      De : Alvaro Paronuzzi 

                      Envoyé : mardi 26 novembre 2019 15:36

                      À : Philippe Plomteux <pplomteux@yahoo.com>

                      Objet : Re:  - Unexpected event correlation New message on BMC Communities

                       

                       

                       

                       

                       

                       

                      <https://communities.bmc.com/> BMC Communities

                       

                       

                       

                      Unexpected event correlation

                       

                       

                      reply from Alvaro Paronuzzi <https://communities.bmc.com/people/aparonuzzi?et=watches.email.thread>  in TrueSight Infrastructure Mgmt - View the full discussion <https://communities.bmc.com/message/850112?et=watches.email.thread#850112>

                      • 8. Re: Unexpected event correlation
                        Alvaro Paronuzzi

                        new sklAutoBulkCreationDM:

                        SKL_EV($NEW)

                        where [

                        $NEW.CLASS == DATAMINER_EV

                        AND

                        $NEW.status == OPEN

                        AND

                        $NEW.skl_sdv_procedureurl == 'Escalation.doc'

                        AND

                        $NEW.skl_platform_owner == PLATFORM_OWNER

                        AND

                        $NEW.skl_hide_autobulk != YES

                        ]

                        updates SKL_EV($OLD)

                        where [

                        $OLD.CLASS == DATAMINER_EV

                        AND

                        $OLD.status == OPEN

                        AND

                        $OLD.skl_host == $NEW.skl_host

                        AND

                        $OLD.skl_sdv_procedureurl == $NEW.skl_sdv_procedureurl

                        AND

                        $OLD.skl_platform_owner == $NEW.skl_platform_owner

                        AND

                        $OLD.skl_hide_autobulk != YES

                        ]

                        within 1 m

                        {

                              concat (['[AutoBulk] ',$OLD.mc_host,' is ',$OLD.severity], $MSG);

                         

                              generate_event (AUTOBULK_EV, [ mc_host = $OLD.mc_host, mc_object = 'multiple', mc_parameter = 'multiple', severity = $OLD.severity, mc_priority = PRIORITY_2, msg = $MSG, skl_sdv_procedureurl = $OLD.skl_sdv_procedureurl, skl_sdv_service_group = $OLD.skl_sdv_service_group, skl_sdv = 'Yes', skl_assigned_group = $OLD.skl_assigned_group, skl_owner_group = $OLD.skl_owner_group, mc_service = $OLD.mc_service, skl_sdv_event_visibility = $OLD.skl_sdv_event_visibility, mc_tool='TrueSight (event.val.lan)',skl_platform_owner=$OLD.skl_platform_owner, mc_host_class=$OLD.mc_host_class, mc_tool_suggestion='multiple Impacted Services', itsm_company=$OLD.itsm_company, itsm_category=$OLD.itsm_category, itsm_type=$OLD.itsm_type, itsm_item=$OLD.itsm_item, itsm_operational_category1=$OLD.itsm_operational_category1, itsm_operational_category2=$OLD.itsm_operational_category2, itsm_operational_category3=$OLD.itsm_operational_category3, skl_team_owner=$OLD.skl_team_owner]);

                             

                              #trace operation

                              concat(['EV#',$NEW.event_handle,' and EV#',$OLD.event_handle,' triggered the creation of a new AutoBulk event'],$DETAIL);

                              opadd($NEW,'',$DETAIL,'');

                        }

                        END

                        • 9. Re: Unexpected event correlation
                          Philippe Plomteux

                          Looks fine to me…Just to minimize risks of anything going wrong, could you replace the “1 m” by “60” ?

                           

                           

                           

                          De : Alvaro Paronuzzi 

                          Envoyé : mardi 26 novembre 2019 16:08

                          À : Philippe Plomteux <pplomteux@yahoo.com>

                          Objet : Re:  - Unexpected event correlation New message on BMC Communities

                           

                           

                           

                           

                           

                           

                          <https://communities.bmc.com/> BMC Communities

                           

                           

                           

                          Unexpected event correlation

                           

                           

                          reply from Alvaro Paronuzzi <https://communities.bmc.com/people/aparonuzzi?et=watches.email.thread>  in TrueSight Infrastructure Mgmt - View the full discussion <https://communities.bmc.com/message/850132?et=watches.email.thread#850132>

                          • 10. Re: Unexpected event correlation
                            Alvaro Paronuzzi

                            Philippe,

                            could you replace the “1 m” by “60” ?

                            I applied the change but unfortunately I'm still seeing the issue.

                            I think raising a case to the Support would be the most appropriate thing at this point.

                             

                            Alvaro

                            • 11. Re: Unexpected event correlation
                              Philippe Plomteux

                              OK…In the meantime you can still set a condition (like I indicated) :

                               

                              ($EV.mc_arrival_time - $OLD.mc_arrival_time < 60)

                               

                               

                               

                               

                               

                              De : Alvaro Paronuzzi 

                              Envoyé : mercredi 27 novembre 2019 10:38

                              À : Philippe Plomteux <pplomteux@yahoo.com>

                              Objet : Re:  - Unexpected event correlation New message on BMC Communities

                               

                               

                               

                               

                               

                               

                              <https://communities.bmc.com/> BMC Communities

                               

                               

                               

                              Unexpected event correlation

                               

                               

                              reply from Alvaro Paronuzzi <https://communities.bmc.com/people/aparonuzzi?et=watches.email.thread>  in TrueSight Infrastructure Mgmt - View the full discussion <https://communities.bmc.com/message/850325?et=watches.email.thread#850325>

                              2 of 2 people found this helpful
                              • 12. Re: Unexpected event correlation
                                Alvaro Paronuzzi

                                Hi Philippe,

                                ($EV.mc_arrival_time - $OLD.mc_arrival_time < 60)

                                The rule is working as expected when using this condition instead of the within clause.

                                Thank you for your precious help!

                                 

                                Alvaro