A few things you can check to begin with...
- What is the polling interval defined on the EmailDaemon.properties form?? Change the polling interval to MailboxPollingUnitIsMinutes=false
- Clear the log form of the email system (AR System Email Error Log). If there are any stuck entries, clear all of the entries if they are not required.
- If the old email messages are archived then clear the "Sent" Messages from AR Email Messages form.
- Are you using distributed lists for group notifications..? you can check the email exchange servers performance too.
We have already followed the steps advised by you. But no help.
What we have observed is SYS:NPC:IM1-TriggerGroupNTG escalation which is invoked to process group notification for all the users present in the group is taking too much time to process.
This could be because the number of users configured per support group is around 300+.
Please note on a daily basis more than 1.5 lacs notifications are triggered from Remedy system & this count as well as the number of users per group is expected to increasing.
Polling interval of SYS:NPC:IM1-TriggerGroupNTG escalation is 1 min.
If that is the case how about trying by defining another Thread or so for the Escalation Queue... if the i understand correctly first the escalation is modifying the record in the sys process control and it takes that record to the group form and loops around that group and notifies each member by pushing the record to the nte notifier form...
so the escalation thread execution starts from modifying the record and till it pushes the record in the nte notifier it would be alive .... meanwhile the next escalation is reaching its 1 min interval and waiting for the execution of the first escalation to get over right?
so set another thread for the escalation queue and observe the performance... depending upon the number of threads available from ar to your db and the number records which are getting cumulated in the sys process control you can determine the max thread size for the escalation queue...
i would say set to 2 observe it if doesn't help much, set it to 3 and observe...
For the escalation queue, the maximum number of threads are started when the
server starts up.
I think you know the root cause of the issue you are facing, as you say it is pretty clear that there is a delay in the escalations. I think you must have already evaluated this option but, have you checked the min / max threads set for the escalation queue..? You can try increasing the max number of threads for the escalation queue (based on the CPUs available on the server) and restart the application service.
Have you checked the performance of the escalations after the ar system services are recycled..? this is because maximum number of worker threads are allocated per queue after the restart and hence there is a less possibility of escalations in escalation pools waiting forever to get triggered and executed.
If you have performed debug logging or any other workflow logging can you share the same in these posts..?
1 of 1 people found this helpful
If still facing the issue.
1] Check 'EmailDaemon.properties' and add a new property called as 'NumberOfSenderThreads = 1' if it is not present.
2]If the property is present and still facing the issue, then try and increase the number of sender threads to '3'.
To understand more about the 'EmailDaemon.properties' please refer to 'BMC Remedy Email Engine Guide'.
This will surely help your environment.
We have implemented the configuration : Private-RPC-Socket: 390603 12 20.
Still we are facing delay in notification mails.
1 of 1 people found this helpful
We have faced a similar kind of issue and here is what we went about doing. We were on ARS 7.5.4 though.
- Checked the number of records on the email messages form. if the emails are not being cleared on the Email Messages form, then there is an issue with the email engine. Please check your SQL logs for long standing queries on the AR System Email Messages form in this case.
- The second issue was the escalation not processing records on the NTE:SYS-NT process Control records form. To resolve this issue we increased the number of escalation threads. Checked the SQL queies taking too much time on the DB level. Made a change in the ar.conf related to cursor sharing.(We changed this one to Force).
These changes actually improved the performance of the entire AR system as such and we did not bump into any performance issues after that.
Here are few points to improve the performance of the Notifications in general:
1. How many records are present in the following forms and if huge number of records are present then need to either archive them or delete them.
AR System Email Messages
AR System Email Error Logs
AR System Email Attachments
NTE:SYS-NT Process Control form
2. Whether email are going out through AR System Email Messages form if yes need to work on the following escalations and run them on separate pools. Steps are given as below.
a. There are three escalations which trigger and do the notification job,
b. First go ahead and increase the escalation threads from Server Information->Ports and Queues tab,
Private-RPC-Queue: 390603 1 1
Private-RPC-Queue: 390603 4 4
This would be creating 4 threads/pools of escalation and click on apply after making this change.
c. Now open the escalation SYS:NPC:ProcessHeldNotifications and change the pool number to 2 and save the escalation, this will make sure that this escalation will be running on 2nd thread
d. Now open the escalation SYS:NPC:TriggerGroupNotifications and change the pool number to 3 and save the escalation, this will make sure that this escalation runs on 3rd thread.
e. Now open the escalation SYS:NPC:TriggerNonGroupNotifications and change the pool number to 4 and save the escalation, this will make sure that this escalation runs on 4th thread
So via this, it will make sure that all the notifications are running on separate thread and it is not going to interfere in other operations activities and performance will be good too, and rest all the other escalation (apart from notifications) will be running on default pool 1. So notifications would be normal.
3. If email engine is not able to send emails through AR System Email Messages form then need to verify the following items
a. Check the connectivity and server names specified in emaildaemon.properties and emaild.exe or emaild.sh
files whether they are same or have some difference like domain name is appended etc. Server name should be same at all places i.e. either only hostname or fully qualified domain at all locations.
b. Verify the port numbers specified and duplicate entries if any in emaildaemon.properties file. Connectivity or java issues can be identified using stderr.log or stdout.log files.
c. If java version specified does not matches the actual location then will need to correct that as well.
We have added the parameter NumberOfSenderThreads=1 in the email daemon properties file.
We monitored the system after the changes were implemented as per your suggestion and have found that there was no improvement in the delay.
As per you earlier suggestion we have increased the number of threads to 3 and monitored the delay.
We have archived data of mails which were delayed earlier, due which currently there is a delay of only 20 minutes being observed.
As per our earlier experiences the delay in system increases with increase in usage and duration of time.
Lets hope this time we dont get much delay.
It is nice to hear the improvement.
- One more suggestion keep the data of last 6 months on email messages form and purge emails earlier to that.
- This will help improve the performance and also you can refer the bkp if further required.
What should be the maximum number of records in Nte sys process control form to avoid delay?
Records should be to the minimalist count as records doesnot stay on 'NTE:SYS-NT Process Control' form,
The flow is 'NTE:SYS-NT Process Contro' form --> Notifier Log --> Notifier.
Hope things are good by now.
Please mark the thread as answered if so or let us know the update..
It seems that we have to upgrade our version from 7.0.01 to 7.6.04 as we are still facing the delay.
I have tried all the options mentioned, but could not get rid of the delay.
BTW, thanks for helping me throughout!