It can be caused by a number of issues at various levels. What are you OS, DB, versions ?
Have you checked out the log files ? I've seen a platform where the AR System threads were killed over time because of network timeouts on the DB connection, resulting in this symptom. There were error messages in arerror.log every time it happened.
If you can't see anything in the logs, then you'll have to monitor the activity of each component :
* Tomcat : dumpthreads.jsp might be enough to detect an issue, if not you can use Java monitoring tools such as VisualVM. Pay attention to the memory and the garbage collector behavior in particular.
* AR System : use the OS monitoring tools to see the memory used (swap ?), the CPU activity per thread, ... At least on AIX there are still some memory leaks issues. If you can see anormal activity on a thread, then you can use more advanced tools like pstack along with the AR System API logs.
There could be many reasons behind this as mentioned by Sylvain.
There may certain jobs running at specific time (AIE/AI/Recon/Normalization etc), or may be It is due to certain escalations running in a same pool, or may be due to certain user/Admin operations (reporting/data import etc).
You can think of load balancing configurations to minimize single server load.
Below are some reference points on performance tuning you can check:
Can you please be more specific with the dumpthreads.jsp file ? is this normally included in tomcat installation or midtier or it is something specially created? where can I get it ?
thanks a lot
BMC Support provided it to me during a case investigation. I think it's not very complicated but quite handy.
May I kindly ask you to share it with me ?