Track-It! Troubleshooting 101: Work Order Notifications

Version 1
    Share:|

    We do a lot of troubleshooting notifications out of Track-It!, so I thought I'd take some time to outline some basic steps you can do if you think you're not getting all of the notifications that you should be.

     

    -First, always bear in mind that Track-It! will not send a notification to a tech if that tech does something to his own work order.  If you open a work order, edit a work order, or close a work order, and that work order has you as the Tech Assigned, you will not be notified.  However, if you open, edit, or close a work order that belongs to another tech, then that tech should most likely be notified, assuming your Event Policies say to send under these conditions.

     

    Now, when you suspect that notifications may not be going out, look at the Audit Trail for a work order that should have sent notifications.  Any time Track-It! is supposed to send a notification, it should put a blue hyperlink in the Audit Trail that shows the notification.

     

    If there's no hyperlink where you expect to see a notification, then that comes down to three possibilities:

     

    1) Event Policies. Open that work order and see what Event Policy is governing it.  Then go to Admin Console > Lookup Tables > Help Desk > Event Policies.  Open that Event Policy, and click the Configure Notifications button.  Check to see if this Event Policy is in fact configured to send to the indicated person, for the desired event.

     

    2) Missing email address. Double click the Tech or the Requestor (whichever one did not get the notification but should have) directly from within that work order.  You should see the Edit Tech or Edit User dialog.  Make sure there's an email address.  If there's not an email address, then Track-It! has no idea where to send any notifications intended for this person.

     

    3) Database timeouts or other errors.  This one is a little tougher to troubleshoot.  Open c:\Windows\temp on the server, and sort the files by Date Modified.  Among the most recently modified files you should see a Track-It-TIServiceManagement.log. Open that, and go to the time frame around the missing notification.  If you see timeout errors, or there are other errors around the time that the notification should have been sent, then most likely this error prevented Track-It! from queuing up that notification.

     

     

    If the Audit Trail does have the blue hyperlink but the notification was not received, this indicates that the notification was queued up.  It may have been handed to the email server, or it may not have.  In order to tell, we query the database table that holds the notifications.  For example, if work order 1234 has a blue hyperlink but I didn’t get notified, then I'd use a query like this:

     

    select NotificationMessageID, NotificationStatusID from NotificationMessage where Subject like '%1234%'

     

     

    That should return results similar to this:

     

     

     

    In this case, I can that 8 different notifications were attempted for this work order (the 8 unique Notification Message IDs in the first column of results).  Each of them has a Notification Status ID of 3.  This means that each one was successfully handed off to the SMTP server, and that SMTP server took ownership of each one.  To track these notifications any further, we’d need to query the SMTP server logs in a similar manner to see what it did with the notification.

     

    If there had been a Notification Status ID of 4, then that would indicate an error handing off to the SMTP server, or a rejection by the SMTP sever.  I’d take the Notification Message ID that indicated the error, and query another table to find out the details of the error.  So, if my results from the query above looked like this:

     

     

     

    Then I’d query the Notification Error table to see what the error message was:

     

    select ErrorMessage from NotificationError where NotificationMessageID = 1425

     

    In my case, that returns:

     

     

     

    That’s the old 5.7.1 relay denied error, so I could then use this Community post to resolve it:

     

    https://communities.bmc.com/docs/DOC-28685

     

    Finally, it’s possible that maybe a large attachment was attempted, or a medium sized attachment was attempted to several recipients, and that’s causing a backlog.  Track-It! has a finite amount of time to hand a message off to the SMTP server and get an acknowledgement from that SMTP server, or it will post an error.  In the case of larger attachments, those take quite a bit longer to hand off, and in some cases the handoff never succeeds.  When that happens, Track-It! realizes that an error occurred, so it marks that message as needing to be re-attempted on the next cycle.  This ends up causing a backlog, because Track-It! will never manage to get that original message handed off, and subsequently all of the notifications that should occur after it sit in a limbo state waiting their turn for delivery.

     

    How do you tell when this is the case?  Well, when run that query to see what the Notification Status ID is, you’ll see all 1s instead of 3s.  (A Notification Status ID of 1 indicates that the message is waiting to be handed to the SMTP server.)


     

     

     

    You can run a query like this to find out how many are backlogged:


    select count (*) from NotificationMessage where NotificationStatusID = 1

     

     

     

    In this case, there are 19 backlogged messages.  I’ve seen this number up in the thousands, because some systems send a lot of notifications, and when a backlog occurs it may be a while before it’s reported/noticed.

     

    To clear this condition, you’re going to have to trick Track-It! into thinking all of those pending notifications have been sent successfully.  This means that none of those 19 backlogged notifications in my system will get sent out.  But that’s necessary in order to break up the backlog, so that notifications resume going forward.

     

    I’d stop the Track-It! Service Management service, then run a query like this to change all of the Notification Status ID = 1 to be Notification Status ID = 3:

     

    update NotificationMessage set NotificationStatusID = 3 where NotificationStatusID =1

     

    I expect to see my 19 rows updated:


     

     

     

    The I delete the contents of the Notification Queue table:


    delete from NotificationQueue

     

    At that point, I can start the Track-It! Service Management service and notifications should resume for any changes that take place from here forward.

     

     

    So to recap:

    Check the Audit Trail for the work order to determine if Track-It! tried to send or not.  If not (no blue hyperlink) then make sure Track-It! was supposed to send (Event Policies) and knows where to send (email addresses), and that there are no database errors.

     

    If Track-It! did try to send (there is a blue hyperlink) then start checking the Notification Message table.  A Notification Status ID of 1 indicates that the message delivery to the mail server is pending, and you may have a large backlog preventing notifications; a Notification Status ID of 3 indicates that the notification was successfully handed off to the mail server and you have to use the mail server’s logs to pick up troubleshooting from here; and a Notification Status ID of 4 indicates a failure to hand off to the SMTP server or a rejection by the SMTP server.

     

     

    These aren't the only reasons notifications might stop being sent, but they are the most common.  They are also relatively easy to troubleshoot.  The next time you suspect that notifications aren't going out the way they should, give these steps a try.  And, if you find anything that needs clarification or could make this easier, let me know and I'll update the document.



    -Keith