Connect with Remedy - Troubleshooting Performance Issues in Remedy Java Applications Webinar Q&A

Version 2
    Share This:

    Presentation References

     

    Java Virtual Machine Monitoring, Troubleshooting, and Profiling Tool

    http://docs.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html

     

    Java VisualVM JMX connection setup

    https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/jmx_connections.html

     

    Heap Dump Analysis using Eclipse Memory Analyzer

    http://help.eclipse.org/neon/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Fgettingstarted%2Fbasictutorial.html

     

    http://www.simplecodestuffs.com/java-heap-dump-analysis-using-eclipse-memory-analyzer-mat/

     

     

    Thread Dump Analysis Using TDA

    https://java.net/downloads/tda/tda-2.2.pdf

     

    https://dzone.com/articles/how-analyze-java-thread-dumps

     

    GC Log Analysis

    http://gceasy.io/

     

    Q&A

    ________________________________________________________________

    Q: What is meant by "Application Snapshot"?

    A: Application Snapshot is one of the type of snapshot which can be captured using VisualVM tool. It provides information about CPU, Memory (Heap, Non-Heap), Threads, and Classes loaded in Java process.
    ________________________________________________________________

    Q: Which Perf. Mon. tool do you recommend the most for slow performance but not OOM errors?

    A: You can use JVisualVM or YourKit to perform CPU profiling that would tell you where the time is being spent.
    ________________________________________________________________

    Q: I've used most of these tools. Which tools do BMC engineers use the most to troubleshoot BMC Remedy bugs/performance?

    A: BMC engineers make use of exactly these tools (Visual VM and Yourkit, Eclipse Memory Analyzer Tool) that are being called out in this presentation.
    ________________________________________________________________

    Q: We suffer from an error on the Dev Studio client (An error has occurred when activating this view Widget is disposed), would you recommend either of these tools to help diagnose this?

    A: Based on the error description you provide, I believe this is a functional / runtime issue, rather than a performance issue - dev studio workspace logs should help in this case - not these tools.
    ________________________________________________________________

    Q: Will OnDemand allow monitoring from customer local?

    A: No. Remedy OnDemand team will monitor these internally.
    ________________________________________________________________

    Q: Chetan mentioned this can be run locally to a remote server.  Assist troubleshooting and monitoring sounds feasible

    A: That is correct, remote connection is possible, however, it needs specific access to the java process (server or midtier, etc.) and would cause security concerns if access is allowed outside the OnDemand.
    ________________________________________________________________

    Q: what settings are needed to monitor AR-server using other java-monitoring tools/applications? Like for example appdynamics?

    A: You can pass a jvm.option -agentpath in the arserver.config
    ________________________________________________________________

    Q: We have had a number of issues since using Remedy AR 9.1. One of which is a problem with arsignal. Can we make use of these tools to find what is happening?

    A: No. arsignal is still a C process. These tools are specifically used for monitoring Java processes only (e.g. server, midtier, plugin server).
    ________________________________________________________________

    Q: I noticed the Initial memory pool is 1024 and Maximum Memory Pool is 2048, If we are experiencing heap issues...should Initial Memory Pool be increase to 2048?

    A: Depending on the heap issue experienced we will need to increase either the Initial Memory Pool or the Maximum Memory Pool.

    Case 1: I) Operating System was itself running low on memory when the java process encountered OutOfMemoryError, and

    ii) OutOfMemoryError is observed in java process at value less than the Maximum Memory Pool then we need to allocate Initial Memory same as Maximum Memory Pool i.e. 2048.

    This kind of allocation (setting Initial and Maximum Memory Pool same) would guarantee that OS reserves the desired memory resource required for java process upfront when the java process starts.

    Case 2: I) Operating System is having sufficient memory.

               ii) OutOfMemoryError is observed in java process for value equal or greater than Maximum Memory Pool.In this case the Maximum Memory Pool size needs to be increased.

    ________________________________________________________________

    Q: What would be the recommendation for the Max Memory Pool for Midtier?

    A: As you got it right; it depends on utilization...not just the number of users but also how much data is being pulled up. Monitor and increase it if necessary. You should refer to https://docs.bmc.com/docs/display/public/brid91/JVM+runtime+analysis for more details
    ________________________________________________________________

    Q: With the built-in JMX listener in ARS, is it possible to enable SSL/security via settings rather than via arserver.config properties file?

    A: Build-in JMX listener has security enabled - it needs AR Server Admin credentials to connect.
    SSL - no. It's not possible by just changes. arserver.config settings is the way to go in this case.
    ________________________________________________________________

    Q: Why does RMI need an Admin account to monitor Java?  Why not a READ account or something other than Administrator? 

    A: It's designed that way so that Remedy Admin doesn't have to remember one more set of login and password.
    ________________________________________________________________

    Q: Performance is also very much slower in Mid-Tier compared to the thick client. I'm guessing chunking is necessary because your codebase was designed for C and


    A: If midtier is tuned well (caching, memory, other config parameters), it should be much better than thick client. I would recommend you go through https://docs.bmc.com/docs/display/public/brid91/JVM+runtime+analysis if you are having performance issues with midtier
    ________________________________________________________________

    Q: Is this applicable for 8.1?

    A: No, AR Server 8.1 is not a Java application, however, this would apply to java processes in 8.1 such as midtier, java plugin server.
    ________________________________________________________________

    Q: I don’t see option of AJAX

    A: You mean JMX port - it's available for AR 9.0+ (i.e. java server - not C server)
    ________________________________________________________________

    Q: Is the RMI Administrator password encrypted? From what I remember it's in clear text which is an unnecessary risk for an Administrator account that is just analyzing the java process.

    A: The user name and password for connecting to JMX is stored in the database in encrypted fashion. It's the password field in User form.
    ________________________________________________________________

    Q: You can still monitor ARS 8.1 java plugins correct?

    A: Yes, if you are monitoring the Java Plugin Server
    ________________________________________________________________

    Q: TY. So you're saying if I search for 100K records in the User form in Mid-Tier, it should be faster than thick client? Cause thick client is very quick for searching.

    A: Even midtier would fetch chunked data and show you 1000 entries at a time. That I believe should be at par with what user tool did, as long as your midtier and AR server have negligible network latency.
    ________________________________________________________________

    Q: An application written in C for Windows will usually be faster than Java

    A: You are right in general as long as it deals with the windows resources. In this case, we are talking about going out to the AR server and fetching data. Midtier increases 1 step for that data, but that should be negligible if midtier and AR are co-located
    ________________________________________________________________

    Q: How can I add a second -javaagent to the arserver.config file?

    A: We have not tried with multiple javaagent in arserver.config yet.  The right way of passing second -javaagent option would be by adding a new jvm.option.x=-javaagent:<AgentJarFile>. Where x is the next jvm option. If you are unable to do so, please open a ticket with BMC for this, we will help you in adding second javaagent to arserver.config file.
    ________________________________________________________________

    Q: My heap dump button is "gray out", how do I turn on that options?

    A: If you already connected to your Java application using JVisualVM and due to some reason your java service stop running, then these buttons will be greyed out. Please check if your java application service is up and running.
    ________________________________________________________________

    Q: Is there any performance issue to always have JMX enabled on for example Mid-tier?

    A: Impact if any would be minimal just for monitoring; specific dumps (heap dump for example) can cause an impact
    ________________________________________________________________

    Q: Returning 100k records in the midtier is much slower than the thick client.  I find that this is because the midtier needs to create HTML to display 100k+ records, which uses a LOT of memory, and bogs down EVERYONE.

     

    A: Searching entries does not fetch HTML. It only fetches entries in raw format and browser javascript renders that data in the browser. We have particularly seen that Internet Explorer (IE) is the slowest while doing this. Other browsers are much better.

     

    ________________________________________________________________

    Q: We had to turn paging on, which helped ALOT

    A: Right. Thanks Kevin for pointing that out. Its right level of configuration and tuning that would help midtier performance.

    ________________________________________________________________

    Q: paging? Windows paging??

     

    A: No. Not windows paging. I mean page size in midtier i.e. pagination. It can be set per-user using user preferences so that the GetListEntry returns 5000 records, so I get 1-5000, 5001-10000, etc.

    ________________________________________________________________

    Q: page size in midtier? Where is that set exactly and what would it be set to?

    A: There are a couple of settings in midtier and dev studio. Midtier - config.properties (arsystem.FileExport_max_entries, arsystem.webreport.onscreen_max_entries, arsystem.nativereport.onscreen_max_entries) .. dev studio - table fields have Max Rows property. And Server Info setting pointed out by Kevin.

    ________________________________________________________________

    Q: And what do you mean by increasing page sizing in mid-tier helped with performance. Is this page sizing a setting in config.properties or something?

    A: In the AR System Admin Console, Server Information, Configuration: Max Entries Returned by GetList.  If you set to 5000, you get paged results quickly. You can also set in User Preferences by user if some people have specific requirements.
    ________________________________________________________________

    Q: Are there any resources on how to interpret the results/output of these dumps or is that covered later in the presentation?

    A: Basic Analysis is demonstrated later in the presentation
    ________________________________________________________________

    Q: When you issue new hotfixes or patches, do you update the ehcache framework to the latest version?

    A: No. Typically any third party version upgrade is never part of a hotfix or patch. It's upgraded only in service packs or releases.
    ________________________________________________________________

    Q: Where do I need to look it up on BMC Communites for this webinar recording

    A: https://communities.bmc.com/docs/DOC-34485 has the webinar schedule and recording links are posted here.  Previous webinars are here as well
    ________________________________________________________________

    Q: BMC: We have in the logs that ehcache has a new version out. With these arsignal issues, I think you might want to look into updating ehcache with future hotfixes/patches for Remedy 9.1

    A: Thanks for the feedback. Yes, ehcache upgrade for midtier is one of the pending items and would be taken up in one of the future releases (can’t give any commitments).
    ________________________________________________________________

    Q: I understand, Ashutosh. We might have to update ehcache to see if it helps with these arsignal issues, at least in our dev environment.

    A: ehcache in midtier and arsignal are not related. They are 2 separate processes and midtier never talks to arsignal.
    ________________________________________________________________
    Q: How can we manage thread dead lock errors?

    A: Chetan is answering this question in the presentation.
    ________________________________________________________________

    Q: We have Max Entries Returned by GetList = 10000

    A: Yes, 10000 is huge. If this is user facing server, performance impact will be significant. Out of the box setting is 2000 and should suffice for most of the customers for user-facing servers.
    ________________________________________________________________

    Q: Our GC is causing 40-70 second suspension during a bulk webservice process overnight... This log viewer may help.

    A: Yes it would. Based on the output of log viewer the areas in heap can be fine tuned.
    ________________________________________________________________

    Q: People complain lot about Overview Console slowness when they log into Remedy. We checked with Database team and the query takes hardly 3-4 seconds while from Application it takes forever.

    A: In this case we need to check the queue sizes in the ar.cfg. Also capturing multiple thread dumps of arserver during the log on into Remedy would help in analysis.
    ________________________________________________________________

    Q: Why is a Remedy Administrator account needed for Java monitoring?  What is the correlation?

    A: There is no real correlation - it's designed that way for convenience so that admin doesn’t have to remember more passwords.
    ________________________________________________________________

    Q: How we can identify the Web service call performance issue?

    A: Enable fine level logs in midtier and API+SQL+Filter logs in the server. Check all these logs to identify the area due to which web service call is taking time. It may be SQL, it may be a plugin in plugin server, it may be midtier, and it may be network. The logs should be able to tell you that. Once you find out which java process is taking time, use jvisualvm or yourkit for the profiling to get more details where the time is being spent.
    ________________________________________________________________

    Q: Looking into getting my hands on last night's GC log from my tech guy now.  This is a very helpful part of the webinar... I've been staring at the Visualizer for weeks. thanks!

    A: Thank you, also you would need to enable the GC parameters to get the GC information.
    ________________________________________________________________

    Q: We have a SmartIT performance issue, how to troubleshoot, already used jvm, any monitoring tool available?

    A: Enable SmartIT and server (API+Filter+SQL) logs to see where the time is being spent. Once you identify the java process which is taking time, you can profile that process using jvisualvm or yourkit to get more details.
    ________________________________________________________________

    Q: Almost every time you need to look at logs you need the logs to have been running at the time of an error, which implies that jmxremote should be always enables. What load does this cause?

    A: Jmx and logs are 2 different things and for different purposes. Jmx is used for monitoring non-functional aspects such as CPU/memory/performance of the system. For functional issues, it's always logs that will help. There is also feature for Always On logging in AR 9.0, which you can look at from documentation. Enabling jmx itself doesn't have much load. Taking dumps or profiling would have overhead.

    Enabling jmx itself doesn't have much load. Taking dumps or profiling would have overhead.

    ________________________________________________________________

    Q: Probably this has already been asked ... but how do I collect/activate the GC (log-) file?

    A: Please refer to presentation for details.
    ________________________________________________________________

    Q: MyIT SmartIT is taking too much time to login. After logon still few options (Approvals, Appointments) are taking too much time to load.  Any idea on how to resolve this one?

    A: It could have multiple reasons including auth time etc. and may not necessarily be a heap/thread/GC issue so it’s best that a ticket be logged with Support.
    ________________________________________________________________

    Q: So number of threads will go back to minimum value if the load is low?


    A: Yes, it would.
    ________________________________________________________________

    Q: What are the most common java issues you are seeing in 8.1.02 and 9x and what is the first thing you look at to troubleshoot it?

    A: The biggest difference in 8x stack and 9x stack is the server. In 8x, server was a C process and 9x has java server. Also the memory requirements for java server are more than a C server. Out of the box max heap size for 9x server is 6 GB, however if it's configured correctly, it about 3-4 GB may be actually used. Any signs of more than 60-70% of memory or CPU usage consistently for some time could be an initial sign for trouble. Monitor this via jvisualvm.
    ________________________________________________________________

    Q: is it CPU cores * 3 for Fast & * 5 for List thread (Max)?

    A: No, there is no empirical formula to decide fast and list threads based on CPU core, since there are many other threads (escalations, archive, activemq, etc.) in the system apart from list and fast threads. Decision on whether fast and list threads are enough can be taken based on API logs that may show queuing or based on thread dumps that may show consistent full usage of these threads.
    ________________________________________________________________