4 Replies Latest reply on Dec 17, 2019 10:47 AM by Brendan Murray

    MRL to correlate memory and swap alerts.

    Sistemas Securitas Direct
      Share This:

      Hello friends,

       

      Help with a mrl please!! I need to correlate the unix KM memory and swap alerts and generate an additional event when both metrics are alerted. The MRL contains 3 parts: create new event (work fine), close event (work fine) and severity changes (not working).

       

      The code of close event and the severity changes are similar... I do not understand why one works and the other does not, the only thing that occurs to me is that the changes of state do not go through the mrls?. Any help would be welcome, what I need is that when the origin events change their severity, the adhoc event is also modified.

       

       

      ###################################################################################

      #  12/12/2019 Linux alerts, Correlate mem and swap

      ###################################################################################

       

      new correlate_LinuxMEMandSWAP_MAJOR : EVENT ($NEW)

      updates ALL EVENT ($OLD)

       

      where [

            $NEW.status == OPEN AND $OLD.status != CLOSED AND ($OLD.severity != INFO AND $NEW.severity != INFO) AND

            (($NEW.mc_object_class == MEMORY AND $OLD.mc_object_class == SWAP) OR ($NEW.mc_object_class == SWAP AND $OLD.mc_object_class == MEMORY)) AND

            $OLD.mc_host == $NEW.mc_host AND ($OLD.severity == MINOR OR $NEW.severity == MINOR)

            ]

                              {

                              generate_event (ALARM, [mc_host = $OLD.mc_host,

                              mc_origin = $OLD.mc_origin,

                              date = $NEW.date,

                              mc_parameter = Correlated_MEM_SWAP,

                              msg = '[ALERTA CORRELADA] Consumo de memoria y swap elevados',

                              mc_object_class = Correlated_MEM_SWAP,

      mc_object = Correlated_MEM_SWAP,

                              status = OPEN,

                              severity = MAJOR ]) ;

                                                                      ntadd($NEW,"Generando Alerta correlada MAJOR por alto consumo de memoria y swap");

                                                                      ntadd($OLD,"Generando Alerta correlada MAJOR por alto consumo de memoria y swap");

                  }

      END

       

      ######################################################################################

      #  DJM 12/12/2019 Closing correlate alert

      ######################################################################################

       

       

      execute Close_correlate_LinuxMEMandSWAP : EVENT($OLD)

              where [$OLD.mc_object_class within ['MEMORY','SWAP'] AND $OLD.severity != INFO]

      using { EVENT($CUSTOM) where [$CUSTOM.mc_object_class equals Correlated_MEM_SWAP AND $CUSTOM.mc_parameter equals Correlated_MEM_SWAP AND $CUSTOM.status != CLOSED AND $CUSTOM.msg contains 'CORRELADA' AND $OLD.mc_host == $CUSTOM.mc_host] }

                      when $OLD.status == CLOSED

                      {

                      $CUSTOM.status = CLOSED;

                                      ntadd($CUSTOM,"Closing correlate alert");

                      }

      END

       

       

      ######################################################################################

      #  DJM 12/12/2019 severity changes

      ######################################################################################

       

      execute Severity_MinorToMajor_correlate_LinuxMEMandSWAP : EVENT($OLD)

              where [$OLD.mc_object_class within ['MEMORY','SWAP'] AND $OLD.severity == MINOR]

      using { EVENT($CUSTOM) where [$CUSTOM.mc_object_class equals Correlated_MEM_SWAP AND $CUSTOM.mc_parameter equals Correlated_MEM_SWAP AND $CUSTOM.status != CLOSED AND $CUSTOM.msg contains 'CORRELADA' AND $OLD.mc_host == $CUSTOM.mc_host] }

                      when $OLD.severity == MAJOR

                      {

                      $CUSTOM.severity = CRITICAL;

                                      ntadd($CUSTOM,"Change severity Minor to Major");

                      }

      END

       

      execute Severity_MajorToMinor_correlate_LinuxMEMandSWAP : EVENT($OLD)

              where [$OLD.mc_object_class within ['MEMORY','SWAP'] AND $OLD.severity == MAJOR]

      using { EVENT($CUSTOM) where [$CUSTOM.mc_object_class equals Correlated_MEM_SWAP AND $CUSTOM.mc_parameter equals Correlated_MEM_SWAP AND $CUSTOM.status != CLOSED AND $CUSTOM.msg contains 'CORRELADA' AND $OLD.mc_host == $CUSTOM.mc_host] }

                      when $OLD.severity == MINOR

                      {

                      $CUSTOM.severity = MAJOR;

                                      ntadd($CUSTOM,"Change severity Major to Minor");

                      }

      END

       

      Thanks and regards!!

        • 1. Re: MRL to correlate memory and swap alerts.
          Brendan Murray

          Hi SSD,

           

          The main difference between your close rule and your severity change rules is that the where clause in your close rule does not refer to the same slot as the when clause. I am pretty sure the problem with your severity change rules is that the where clause conflicts with the when clause. The result is the when clause is never executed.

           

          For your Severity_MinorToMajor_correlate_LinuxMEMandSWAP rule, the where clause is this:

           

          where [$OLD.mc_object_class within ['MEMORY','SWAP'] AND $OLD.severity == MINOR]

           

          This means the body of the rule will only execute if the severity of the triggering event is MINOR.

           

          However, your when clause uses the same slot:

           

          when $OLD.severity == MAJOR

           

          Your rule will execute when the severity slot changes. If it changes to MAJOR, the ECF (event class and where clause) for the rule will be evaluated and it will fail because the slot value is now MAJOR and your rule only works if the severity is MINOR. The event must match the ECF of the rule before the when clause will be evaluated. From the documentation: "The When clause in rule phases is reevaluated whenever a value changes for a slot, if the ECF condition is met."

           

          Try changing your where clauses so that they don't conflict with your when clauses.

           

          Please let us know if this solves your problem.

           

          Regards,

           

          Brendan

          1 of 1 people found this helpful
          • 2. Re: MRL to correlate memory and swap alerts.
            Brendan Murray

            Hi SSD,

             

            While I was looking at your code, I noticed a few other things you may want to change:

             

            1. The ECFs of your rules reference the base EVENT class:

             

            new correlate_LinuxMEMandSWAP_MAJOR : EVENT ($NEW)

             

            It is best to avoid this, if possible, as it means the cell has to evaluate your rule against every event. This can slow down event processing. For best performance, try to restrict your rule to a specific event class. For example, your rules are about Linux Memory and Swap. I assume you are using the PATROL agent to monitor these things, so your rule will only ever match either PATROL events or ALARM events, depending on whether the events are coming directly from the agent or from the BPPM/TSIM analytics engine. For example, if the events are coming directly from the agent, you should use the PATROL_EV event class, not EVENT.

             

            e.g. new correlate_LinuxMEMandSWAP_MAJOR : PATROL_EV ($NEW)

             

            2. In your new rule, you generate an event of the ALARM class:

             

            generate_event (ALARM, [mc_host = $OLD.mc_host, ...

             

            The ALARM class is used for events generated by the BPPM/TSIM analytics engine. I would recommend not using this class. I am not sure it will cause any problems, but it might cause confusion, especially if you need to open a Support case with BMC.

             

            3. Further to item 2, I would strongly recommend you create a custom event class for your generated events. It will simplify your execute rules and also make them more efficient. The reason is that it is much faster for the cell to evaluate the class of an event than it is to evaluate its slots. Your current logic generates the event like this:

             

            generate_event (ALARM, [mc_host = $OLD.mc_host,

                                    mc_origin = $OLD.mc_origin,

                                    date = $NEW.date,

                                    mc_parameter = Correlated_MEM_SWAP,

                                    msg = '[ALERTA CORRELADA] Consumo de memoria y swap elevados',

                                    mc_object_class = Correlated_MEM_SWAP,

                                    mc_object = Correlated_MEM_SWAP,

                                    status = OPEN,

                                    severity = MAJOR ]) ;

             

            And the using clause of your close and severity change rules then has to evaluate the entire EVENT class with this very complicated where clause:

             

            where [$CUSTOM.mc_object_class equals Correlated_MEM_SWAP AND $CUSTOM.mc_parameter equals Correlated_MEM_SWAP AND $CUSTOM.status != CLOSED AND $CUSTOM.msg contains 'CORRELADA' AND $OLD.mc_host == $CUSTOM.mc_host]

             

            First, I would like to understand why you check all these slots. You are generating these events. Your logic is what puts the values in these slots. You should already know that if $CUSTOM.mc_object_class equals Correlated_MEM_SWAP then $CUSTOM.mc_parameter will also equal Correlated_MEM_SWAP. I don't understand why you need to check this.

             

            In your new rule, the generate_event should look something like this:

             

            generate_event (CORRELATED_MEM_SWAP, [mc_host = $OLD.mc_host,

                                    mc_origin = $OLD.mc_origin,

                                    date = $NEW.date,

                                    msg = '[ALERTA CORRELADA] Consumo de memoria y swap elevados',

                                    severity = MAJOR ]) ;

             

            It is not necessary to specify 'status=OPEN' for your generated event. When you generate an event it automatically has a status of OPEN.

             

            The using clause in your other rules should then look something like this:

             

            using { CORRELATED_MEM_SWAP($CUSTOM) where [$CUSTOM.status != CLOSED AND $OLD.mc_host == $CUSTOM.mc_host]

             

            This greatly simplifies your logic and should improve the efficiency of your rules.

             

            To use this approach, you will need to define a new event class for your custom event. You do this in the classes directory of the cell knowledge base. The simplest form of adding a new class looks like this:

             

            MC_EV_CLASS :

               CORRELATED_MEM_SWAP ISA EVENT ;

            END

             

            This example simply creates the new class and does not add any slots. It is effectively a copy of the base EVENT class. If you think you need additional slots, you can add those to your class definition. Look at the other class definitions in the classes directory to see how to do that. It is not hard.

             

            4. Be careful about using updates ALL, especially against the base EVENT class. This is an expensive call. When used with the base EVENT class, It requires the cell to evaluate every EVENT in the event repository. That could be thousands of events. This has the potential to slow down the cell significantly. Sometimes it may be necessary, but most of the time it is not. Only use updates ALL if you expect that more than one event will match your where clause and only use it against the most specific event class possible. If you know that that only one event will match the where clause of your updates block, then use updates, not updates ALL.

             

            In your case, I am not sure whether it possible that more than one event will match your updates block. If you must use updates ALL, at least use a more specific class. For example, you could use the PATROL_EV event class since these are events coming from the Linux KM (I assume).

             

            Also, your generate_event primitive is inside your updates ALL block, even though it is not actually updating anything. I am not sure, but I suspect that if more than one event does match your updates ALL condition you will generate an event for every event that matches your updates block. I am pretty sure that's not the behavior you intend.

             

            5. Related to item 4, the where clause of your updates ALL is extremely complicated:

             

            where [

                  $NEW.status == OPEN AND $OLD.status != CLOSED AND ($OLD.severity != INFO AND $NEW.severity != INFO) AND

                  (($NEW.mc_object_class == MEMORY AND $OLD.mc_object_class == SWAP) OR ($NEW.mc_object_class == SWAP AND $OLD.mc_object_class == MEMORY)) AND

                  $OLD.mc_host == $NEW.mc_host AND ($OLD.severity == MINOR OR $NEW.severity == MINOR)

             

            This will add even more performance overhead to an already expensive updates ALL. I suggest you simplify the where clause two ways:

             

            a. Move any conditions that reference the new event only into the ECF for the rule itself, not the updates ALL. The updates ALL has to evaluate these conditions for every event in the cell. Moving them to the where clause for the rule will cause them to be evaluated only once. If they don't match, the cell will skip the updates ALL completely, thereby reducing the load on the cell.

             

            For example:

             

            new correlate_LinuxMEMandSWAP_MAJOR : PATROL_EV ($NEW)

                 where [$NEW.status == OPEN AND $NEW.severity != INFO]

             

            b. Do not try to do too many things in a single rule. You end up with very complicated ECF conditions such as in your updates ALL that are very inefficient. Your correlate_LinuxMEMandSWAP_MAJOR rule should actually be two rules:

             

            correlate_LinuxMemAndSwap

            correlate_LinuxSwapAndMem

             

            (I took the "MAJOR" out of the rule names. In general, I would not embed severity names in the rule names.)

             

            The rules would start like this:

             

            new correlate_LinuxMemAndSwap : PATROL_EV($NEW)

                 where [$NEW.status == OPEN AND $NEW.severity != INFO AND $NEW.mc_object_class == MEMORY]

             

            updates ALL PATROL_EV($OLD)

            where [

                  $OLD.mc_host == $NEW.mc_host AND $OLD.status != CLOSED AND $OLD.severity != INFO AND $OLD.mc_object_class == SWAP

                   AND ($OLD.severity == MINOR OR $NEW.severity == MINOR)

                  ]

            ...

             

            new correlate_LinuxSwapAndMem : PATROL_EV($NEW)

                 where [$NEW.status == OPEN AND $NEW.severity != INFO AND $NEW.mc_object_class == SWAP]

             

            updates ALL PATROL_EV($OLD)

            where [

                  $OLD.mc_host == $NEW.mc_host AND $OLD.status != CLOSED AND $OLD.severity != INFO AND $OLD.mc_object_class == MEMORY

                   AND ($OLD.severity == MINOR OR $NEW.severity == MINOR)

                  ]

            ...

             

            The rest of the rules would contain the generate_event code I recommend above.

             

            Breaking your logic into two rules this way will make it faster and much easier to read.

             

            Regards,

             

            Brendan

            2 of 2 people found this helpful
            • 3. Re: MRL to correlate memory and swap alerts.
              Sistemas Securitas Direct

              Great! Just what I needed!! It works!

              Is the first time that I use the when clausule. I am very grateful for the recommendations you have given me, thank you very much Brendan!

              • 4. Re: MRL to correlate memory and swap alerts.
                Brendan Murray

                You're most welcome. I'm glad to hear you got it working.

                 

                Regards,

                 

                Brendan