It all depends your number of users and workload on the system and available resources (number of cpu, RAM, disk space etc). If you need to reproduce and take the logs that should not impact much and if you don't know when the issue arrives then you can turn on logs for a day to capture the issue. you know your environment and decide as per your need.
Thanks for the reply, but our issue is that there is no timeline for reproducing the issue, it could be 24 hours or more.
If we turn on the logging, that slows down the system.
So we are stuck in a scenario, where we cannot get enough logs to troubleshoot the issue.
Would you share what kind of problem you guys are facing and wants to trouble shoot? Also how many transactions daily and number of users? What is your hardware setup? You can turn on for a day and see if it is affecting the performance then you can turn it off otherwise keep going till capture the culprit.
WE have an average load of 5-10k incidents per day.
We have around 20-25 users on average working in shifts.
We are facing performance issues and wanted to troubleshoot.
BMC support asks us to produce logs during the issue, however we cannot anticipate the timing of the issue.
Even if we enable the logs during the issue, BMC says it couldnt find anything in the logs pertaining to the issue.
1 of 1 people found this helpful
Please have a look on below links that might be useful-
Is there a way to enable logs logs from command line, as sometimes, when performance issue hits us, we cannot even access our UI properly to enable the logs.
3 of 3 people found this helpful
In such scenario, BMC has come up with the Always ON logging feature in 9.x version. Now talking about your version i.e. 8.x, yes keeping the logs all the time is not a feasible solution as you might end up with your storage space soon. but as an alternate you can setup some network drive that is accessible by AR Server machine.
If you want to keep the log enable all the time, then setting up the network drive and specifying the network drive path is one way that you could do.
Regarding performance debugging:-
- We don't need the logs to be enabled all the time.
- You can try to identify a pattern like when you see performance lag morning, afternoon, after lunch, evening or at night.
- If you know the performance is hitting the system. Enable the logs at that point and keep it enable for 20-30 mins.
- Try to figure out the pattern with looking into the logs like arerror.log, armonitor.log, arjavaplugin.log as these logs are always up and running.
Consider an example,
If you see that today at 3:00 PM you hit the performance issue but you were not able to get the logs. Take a look at the above log files and see what exactly the server was doing. Similarly monitor for next day at 3:00 PM time. If you see that the server again hits the performance issue, you can enable the logs and ask your users to continue working and report all sort of slowness, errors that they report for atleast 30 min of time. This will be helpful logs with the end user observations.
3 of 3 people found this helpful
FYI, I typically leave API/Filter/SQL/Escalation logs on at all times on all servers. Just make sure you have Buffer Logged Lines enabled and a reasonable Maximum Log-File Size/Maximum Backups set (to give you enough info but not run out of disk space). While I know there are busy enough systems out there where this would cause a performance problem, I am yet to run across one of those. Assuming you have sufficiently sized servers, it doesn't sounds like your user-load will be an issue.
I too have run into similar use cases like where "Betty was trying to save CRQxxxxxxx and got this error". As long as it was reported timely I can go in and find that transaction in the logs. This has also proven helpful in those times when there have been larger impacting issues and I could go back and find the root cause (sure I have been doing this since before always on logging).
Regarding finding the needing in the logging haystack... LJ LongWing Log Parser tools are invaluable! In the example of Betty above I would parse the log with 'ParseString" looking for the CRQ number and Betty's login ID, get the thread ID and then parse the log again using ParseServerSideLog. If it is a complex transaction I might then run that though RemoveFailedDisabled but usually I don't even go that far.