HowTo : Email Notifications Like It isn't the 80's - Part 1: Intelligent Events

Version 1

    I have zero background in anything prior to ProactiveNet v9.5 so when we were looking to set up Email Notifications, there was confusing information on how to send a silly email alert. 


    Part 1 of this document focuses on Event Rules based on Intelligent Events.  Part 2 will focus on External Events.


    The below information describes our "best practice" for doing this.  ProactiveNet's email alert system still leaves a lot to be desired as I am a massive fan of usability, but you can at least get some clean looking notifications coming out of their clunky interface.


    We use ZERO Notification Alerts as configured through the 1980's style Admin Console.  ALL of our email notifications are done through the Operations Console -> Options -> Administration Tab -> Event Rules.


    Before we progress, there needs to be an understanding of the types of events ProactiveNet will alert and perform actions on: External Events and Intelligent Events.


    External Events:

    • Anything the Patrol Agent sends you.  If you copied the event to Notepad, the header of the event would say PATROL_EV.  Because the event isn't generated by ProactiveNet, it is External.  This are auto-generated by Patrol or based on Agent Threshold CMA Policies.  As a side note: We never configure agent thresholds.  Patrol just doesn't have the brains that ProactiveNet has.  Some would disagree with this but again: we aren't in the 80's.

    • ProactiveNet Self-Monitoring Events (yeah I know, it's counter-intuitive).  If you copied the event to Notepad, the header of the event would say PPM_SM_EV.  Things like an Patrol agent disconnecting from ProactiveNet will generate this event.
    • Anything else such as SNMP traps, external cells, etc.  Things that aren't generated by ProactiveNet.

    Intelligent Events:

    • Any event generated by a Server Threshold configuration.  These are the smart guys (thus intelligent).  If you copied the event to Notepad, the header of the event would say ALARM.  This is where you get alerts from baselines, signatures,etc.  Some incredible power over alerts that agent thresholds don't remotely offer.  This is the ONLY type of threshold we use.


    Now that we have defined our terms, we can now get into how we have configured the Event Rule based on the type of Event that ProactiveNet displays.


    Getting into Event Rules, you need to know about some quirks and recommendations the configurations.  As stated above, it isn't remotely the best, but we made it work.  Here are a list of things to note:


    1. If you aren't extensively using Groups in ProactiveNet (a HowTo document is in the works) then using Event Rules and pretty much any part of ProactiveNet will be useless.  Every application should be grouped and is what smart Event Rules depend on.
    2. Despite the description text ONLY  use Basic Rules for External Events.  You lose a ton of granularity potential when you use a basic rule for intelligent events.
    3. Subsequently: ONLY use Advanced Rules  for Intelligent Events.  Intelligent Events are smart, don't put them in a dumb box.
    4. Have ONE event rule for when an event opened and a SEPARATE rule for when that event gets closed.  The only exception is for external events that don't auto-close (such as Windows Event Log events or Log File Monitoring events).  You might like to know when the event is closed, but typically there is not need.
    5. Create a Schedule for each application group.  There scheduling interface is wonky but it has potential.  We use the schedule for maintenance intervals or known outages because the Blackout CMA Policies don't work very well.
    6. Create Email Groups depending on the application group.
    7. You cannot create multiple templates for Event Rules.  This is silly and should be changed, but for now, put all of your templates in a word document for easy copy/paste.
    8. Use a consistent naming convention ALWAYS.  If you don't it will be very hard to track five years down the road.
    9. Finally, there will be a lot of event rules depending on the number of applications.  At a minimum, each application will have 4 event rules configured: an opened and close for performance (intelligent) events and an open and close for agent disconnect events.  For our 100+ applications we have 400+ Event Rules.



    Intelligent (ALARM) Event Rules - typically performance events, but includes Services.


    • Open Event Rule administration and click Add.
    • The event rule name should be the same convention for EVERY event rule or you will lose track.  Our naming convention is:
      1. <GroupName><AppName> Performance Event Opened
      2. <GroupName><AppName> Performance Event Closed.
        1. Example1: Corporate IT Exchange Performance Event Opened
        2. Example2: Middleware WebLogic Performance Event Closed.
    • If you don't already have one for that application support group, click the radio button for Schedule then click add.  If they have a standard maintenance schedule, enter that here.  If not, just select all days.  The interface is weird (and the exclusions are event weirder: why only have monthly exclusions and not daily or weekly??).

      • Select the radio button for Advanced Event Rule.
      • Select the radio button for Include Selected Groups and select the group you are want to alert on.

    • Click Next.
    • We now get to configure the performance monitors that we will alert on (such as CPU, Disk, I/O, etc).
    • There are two radio buttons for the type of monitor type or monitor instance you want to alert on.  If you have your server thresholds configured properly, you should only need to top radio button:
      • Select Monitor Type and Attribute: This will be the one used most often.  It allows you to select the Monitor Type such as Logical Disk then allows you to select the Attribute such as Free Space or Busy Disk Time.
      • Select Monitor Instance and Attribute: This gets ultra granular.  It is great the power is there, but is mostly unnecessary.  In a nutshell, this allows you to do something like alert off of a specific drive on a specific server.  If your groups and thresholds are configured correctly you won't need this but it might be a useful troubleshooting tool.
    • With the top radio button selected, click the Add button.  You are now presented with the Monitors and their sub-attributes depending on what you are alerting off of.
      • A note on selecting these: Think about what you are alerting on NOW and potentially in the future.  You DON'T want to select monitors that you will never create a server threshold on such as TCP Connections, but you don't want to have to edit a bunch of Event Rules if you missed something down the road.
      • So far for all of our Windows Systems, we add Logical Disk: Free Space, Logical Disk: Free Megabytes, Paging File: Usage, Processor: Processor Time, Windows Process: Number of Processes Monitored, Windows Process: Processor Time and Windows Service: Status.
      • You have to add each monitor at a time (but you can select multiple attributes within the monitor) so you have to click Add for each.
    • Select the monitor then the attribute(s) then click Add.

    • Do this for all monitors until your list is populated like below:

    • We now need set the severity that we will actually alert on.  In our environment (embarrassingly) we still have server space outages so we set the severity to Major.  But the others are Critical.  Again, the importance in setting your server thresholds is a huge part in limiting your email alerts.  So now it will look like this:


    • We now need to set the Event State Filter and because this event is for when the event first opens, we select only Open.  When creating the Closed event, select only the Closed checkbox.

    • Click Next.
    • We are now in the what to do when the event triggers sectionI am only covering the email part of it because we haven't done the action part but the power is there and it is something we want to implement.
    • Enter an SMTP email address in the From field.  We just have some dummy name that isn't an actual SMTP address but people know where it is from.  If you want people to be able to reply to the email then put in a valid SMTP address.
    • You can ignore the Message Format because the email group that you configure will pre-select the format.
    • If you don't already have a group set up, select the radio button for Email Group then click New.
      • Like all things in ProactiveNet, create a standard naming convention for your email groups.  They aren't pulled from Exchange or anything so you can come up with your own naming convention that is easy to spot and select from a drop-down list. 
      • Enter the SMTP address for the group.  You can enter multiple on separate lines if you wish.  But remember: Everyone linked to that SMTP address will get an email.  Make sure it is limited membership group!!!!!
      • Select the drop-down for the Email Type and select Full Page HTML/GIF.
        • You can select anything you like, but we aren't in the 80's so HTML looks better.
      • Click Add.  You will now see the area for Selected Emails populated with your SMTP address(es).

    • Click Done.  The new group will now show up in the Email Group List.  You now need to Add it to the Actions to Perform section:

    • We now need to customize how email will be formatted.  As stated above, you can't create email format templates.  They have ONE for Global settings, but seriously, it is ugly and doesn't fit with making actual human readable emails.
      • Click Event Text Customization
      • The window has three sections for the email format: Full Page ASCII, One Line ASCII or HTML.  We are working in the HTML section because that is how our Email Group for this application is configured.
      • Because it is HTML we have some very basic HTML formatting options.  I only use the <b>old tag as you will see.
      • Following is how we have formatted our Performance Event Subject and body of the email using the variables listed on the right side.  You can customize it to your hearts content.  After trying many variations, the below was the most readable.  Also note, the text will actually change depending on if it is opened or closed.
      • Opened:
        • Subject Line: $SEV Event Opened for the “$INSTANCE_NAME” $MO_TYPE on $DEVICE.
        • Body Text:

    <b>Device Name</b>: $DEVICE

    <b>Device IP</b>: $MC_HOST_ADDRESS

    <b>Monitor Type</b>: $MO_TYPE - $ATTR_NAME

    <b>Instance</b>: $INSTANCE_NAME

    <b>Current Value</b>: $LAST

    <b>Measurement Units</b>: $UNITS


    <b>Time Opened</b>: $TIME


    If this is an <b>unplanned system state</b>, please open an incident ticket and notify stakeholders.

      • Closed:
        • Subject Line: The $SEV Event is Closed for the “$INSTANCE_NAME“ $MO_TYPE on $DEVICE.
        • Body Text:

    <b>Device Name</b>: $DEVICE

    <b>Device IP</b>: $MC_HOST_ADDRESS

    <b>Monitor Type</b>: $MO_TYPE - $ATTR_NAME

    <b>Instance</b>: $INSTANCE_NAME

    <b>Current Value</b>: $LAST

    <b>Measurement Units</b>: $UNITS


    <b>Time Closed</b>: $TIME


    The above instance is no longer experiencing a $SEV event.  Please validate system functionality.

        • This gives us emails that look like this:

    • Make sure the check-box for Include More info is selected as that gives you the pretty graph.
    • Click Apply and you will see a silly message about Local settings overwriting global settings even though global settings aren't even used.
    • Finally click Done at the bottom of the page and you now have a spiffy Intelligent Event configured.
    • Now create a similar event changing key text to Closed and you are good to go.
    • See the next HowTo document on configuration External Events. (coming soon)