Are you using ProactiveNet or TrueSight and have been asked to tune the environment for performance?
Is this really necessary? – The answer is YES – Tuning is very important to ensure continued performance of growing environments.
Perhaps you have been using the product for years and your original environment remains. This may or may not work for you based on the size of your environment and growth over the years. Proper maintenance of such large environments is crucial to keeping performance up to par.
Have you seen our FAQ regarding performance and scalability? If not, I would encourage you to visit the page and review the questions most users have regarding performance and tuning.
The most common question we see in Support is How do I know my system is overloaded?
There are several ways to determine if there are performance issues and if your system exceeds capacity.
The first thing to do should be to login to the TrueSight Console/ProactiveNet Operations Console and access the Performance Diagnostics Report
Login to the Operations Console->Options->Administration->Performance Diagnostics View and this will show you the number of monitored devices, instances, attributes, KPI attributes, components, and Events
Sample Performance Diagnostic Report
Then refer to the BMC TrueSight Documentation based on the type of environment you are using. Here are a couple of references – Sizing the Presentation Server and Sizing for Data and Event Management Environments
For example, here’s the sizing guideline for the TSIM 10.7.00
There are also other values to compare based on the various components under the TSOM Suite
If your values are greater than the tested values, there are likely to be performance issues. BMC would recommend attempting to lower the load on the system perhaps filtering instances and removing any unnecessary monitoring.
The most common symptom of the load on the system exceeding capacity is that the analytical engine runs out of memory. If this occurs, OutOfMemory exceptions are logged in one of the files that are located in the <installationDirectory>\pw\pronto\logs directory:
If the OutOfMemory exception is caused by Rate or JServer processes, the log is in the ProactiveNet.log file. If the OutOfMemory exception is caused by the agent controller process, the log is in the pronet_cntl.out file.
You can increase the MaxHeap value in one of the following files, located in the <installationDirectory>\pw\custom\conf directory:
For the Rate process, pnrate.conf
For the JServer process, pnjserver.conf
For the agent controller, pnagentcntl.conf
The following list presents additional common issues that can arise when capacity is exceeded:
Gaps sporadically appear in data collection across all monitors, or artificial alarm delays occur after a threshold condition has been violated. You might also see gaps in the data shown in graphs. When this occurs, you might see pending, cache size limit exceeded, and dropping messages logged in the Infrastructure Management log file.
Gaps in data could result from memory issues or I/O bottlenecks. If you do not see OutOfMemory exceptions logged, the system probably has an I/O issue. Check the I/O status by looking at the percentage of the disk that is busy (for example, the iostat cmd on Solaris). If the system consistently shows that over 30%, the system probably has an I/O issue.
User response becomes much slower when using the web interface. If user response is slow for all interactions, the system has a problem with memory or CPU. If user response is only slow when graphing, the system probably has an I/O issue.
Check the out-of-the-box monitors that Infrastructure Management creates on the BMC TrueSight Infrastructure Management Server. If you see sustained (several minutes) CPU spikes above 80%, this usually indicates some resource issue, although not necessarily a lack of CPU.
There is a listing of the main parameters which are used to tune these environments based on your use. Please be sure to revisit these values when your application grows. Instances and attributes can add up very quickly, particularly when monitoring filesystems or large environments.
Some of the common issues descriptions we see in Support which ultimately lead to performance tuning include – jserver consuming large amounts of CPU, jserver does not restart, slow performance, poor performance in the UI, data not collecting, java_cntl high consumption and pronet_cntl process down, etc…We do see a good number of customer environments which do require tuning. Part of the process of investigating these types of issue will usually include verification of hardware sizing, minimum requirements, sizing, and the current configuration, and log files (usually a pw dump 1), and possibly a heap dump. A heap dump will help Support determine the root cause of memory issues in your environment. The type of details will depend on which product component is experiencing the issue.
If you are experiencing any of the symptoms mentioned above, we would ask that you report them to Support and we will be able to assist you in reviewing the issue, the environment, and making recommendations for improved performance. We will do our best to get your system back up and running as soon as possible. Remember to be proactive and tune your environment and monitor growth to keep ahead of the game when it comes to application resources.
Related Knowledge Articles
Here is a quick video with some TSIM performance tuning tips – Enjoy
We also have a BPPM Performance Diagnostics video for your review
TrueSight Operations Management 11.00 is coming soon! We are excited to continue work to prepare for the release of TrueSight 11.00.
Keep watch for more details and announcements about the upcoming version.
The release is right around the corner, contact your BMC Account Manager to discuss upgrade plans today!
Frequently Viewed Articles
Here is a listing of the most frequently read Knowledge Articles over the last month
We want to hear from you! Post your comments regarding what you want to see!
Looking for a previous blog posting? Find it here: BMC TrueSight Pulse Blogs