11 Replies Latest reply on Mar 24, 2013 9:47 AM by Robin Elliott

    java_pid****.hprof generated by AO

      Share:|

      Hi Experts,

       

      In my environment i have 1 CDP.Recently i found that AO is generating a lots of *.hprof file in AOHOME\AO\tomcat\logs folder and its occupy almost total disk drive. i found almost 5 .hprof file and it consumes almost 9 GB space.

       

      I know its because of JVM heap memory leak java generate this type of file.But i could not find the exact reason for this memory leak,

       

      i found this CDP JVM heap configuration in AOHOME\AO\bin\BMC Atrium Orchestrator CDP.lax

        

      lax.nl.java.option.additional=-Xms2048m -Xmx2048m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=192m  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="d:\\Program Files\\BMC Software\\AO\\tomcat\\logs" -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file="d:\\Program Files\\BMC Software\\AO\\tomcat\\conf\\logging.properties" -Dcatalina.base="d:\\Program Files\\BMC Software\\AO\\tomcat" -Dcatalina.home="d:\\Program Files\\BMC Software\\AO\\tomcat" -Djava.io.tmpdir="d:\\Program Files\\BMC Software\\AO\\tomcat\\temp"

       

      Let me know what should i do now ?

       

      Please help me as soon as possible.

       

      Regards

      Kalyan Halder

        • 1. Re: java_pid****.hprof generated by AO
          Jayesh Panchal

          Hi Kalyan,

           

          You are right.you are running short of heap memory or its a memory leak.

           

          Java is providing us a "jvisualvm" tool by which you can monitor the usage of java heap space.

           

          This tool you can find in jdk\bin directory. There are many other tools also.

           

          You can just load the .hprof file and check which classes has taken the major heap memory.

           

          I think in AO there are lots of web-services which open the session data and keep it open after the completion of the operation.

           

          One thing you can do that monitor your heap space usage using jvisualvm tool. if heap space is not enough then you can increase the heap size.

           

          If that doesn't help then contact BMC support because they can provide you the exact hotfix for the issue.

           

          Regards,

          Jayesh

          • 2. Re: java_pid****.hprof generated by AO

            Couple of questions:

            1) It appears the -Xms and -Xmx settings have been modified from the defaults of

            -Xms1024m -Xmx1280m

             

            In general, it is not a Java best practice to set both -Xms and -Xmx to the same values.

             

            2) What version of BAO is this?

             

            3) What all is running on this CDP?

             

             

            It is possible that the CDP experiances spikes of concurrent activity that maxes out the heap. In production like environments it is a best practice to start with  -Xms2048 -Xmx4096

            • 3. Re: java_pid****.hprof generated by AO
              Ranganath Samudrala

              1. What version of BAO?

              2. What kind of jobs - rules, scheduled or web service (ORCA)?

              3. How many jobs are running concurrently in the CDP?

              4. Do you have  workflow logic that uses recurssion - a process calling itself?

              5. Do you process huge XMLs in your transformations?

               

              2G for BAO is alright as long one can answer to the questions 3, 4 and 5 satisfactorily so that you can run the CDP with that much memory.

               

              More often than not OOM errors are caused by badly written workflows that do not manage recurssion properly.

               

              Ranga

              • 4. Re: java_pid****.hprof generated by AO

                Jake,

                 

                Yes we modified the -Xms and -Xmx value.

                 

                1. BAO version - 7.6.02.04

                2. There are all total 10 Module is up among them 4 are BMC module and 6 are customized module

                • AMP-AD-BMC-Remedy-ARS
                • AutoPilot-AD-Utilities
                • AutoPilot-OA-Change_Management
                • AutoPilot-OA-Common_Utilities

                 

                Customized processes mainly interacting with ARS,SQL,SMTP,BBSA,Oracle.

                 

                Regards,

                Kalyan

                • 5. Re: java_pid****.hprof generated by AO

                  Ranga,

                   

                  1. BAO version 7.6.02.04
                  2. There are all total 10 Module is up among them 4 are BMC module and 6 are customized module 
                  • AMP-AD-BMC-Remedy-ARS
                  • AutoPilot-AD-Utilities
                  • AutoPilot-OA-Change_Management
                  • AutoPilot-OA-Common_Utilities

                   

                  Customized processes mainly interacting with ARS,SQL,SMTP,BBSA,Oracle.

                    3.   Concurrently there are 10 Main jobs is running but internally they are calling another porcesses as well

                    4.   Yes we have an small recurrence process which will check for an particular file present in an directory or not the only it will process further. and its coded well and it will finish depend on condition never goes infinity.

                    5.   Yes we are processing huge XML in transformation.

                   

                  Regards,

                  Kalyan

                  • 6. Re: java_pid****.hprof generated by AO
                    Ranganath Samudrala

                    Have you looked at processes.log file and seen any evidence of recurssion going haywire?

                     

                    Let us say you have 1K of data passed into a call-process and this call-process invokes itself again and again (recurssion), this 1K of data is cloned again and again and could cause OOM errors eventually.

                     

                    In the processes.log file, you should see evidence of this around the time the OOM error occurred, if this indeed is a problem - a line that is huge - could take up half of more of a 4M file.

                     

                    If you take out the XML and save it in a file, how big is that file - 1K, 10K or something else? Are these XMLs used in the recursive logic?

                     

                     

                    I know for sure that there is no recursion in the OOTB modules.

                     

                    Ranga

                    • 7. Re: java_pid****.hprof generated by AO

                      Kalyan,

                       

                      I am assuming that you modified the lax file and removed the following two parameters for immediate relief from not filling up your disk drive with .hprof files.

                      • -XX:+HeapDumpOnOutOfMemoryError
                      • -XX:HeapDumpPath="d:\\Program Files\\BMC Software\\AO\\tomcat\\logs"

                       

                      I would suggest that if you are regularly seeing this problem and/or its causing you production problems I would raise a ticket with BMC support.  They will likely want a reasonable set of grid/process.logs that show your peak workload and potentially capture the problem occurring.  They have access to some analysis and profile tools that will aid in both you and BMC better understanding your environment and running workflows.

                       

                      With the old 32 bit OS/JVMs you could not set the Xmx setting above 2GB, but with the advent of 64bit OS/JVMs you can.  Your peak Job workload, mix of workflows, data passed between workflows to children, adherence to best practices etc... will all determine how much memory should be added to the Heap.  At the largest end of the spectrum we have 1 or 2 customers with settings of 12 to 16 GB for the Xmx, but they are accounting for complex/large/deep workflows at peak volumes within BAO where floods of up to 2000 server down alerts hit concurrently.  For mid use type customers anything from 2 to 6 GB can be set as the Xmx but its best to follow the rule that the Xmx value falls in the range of only 50-60% of memory actually allocated to the server (If the Server memory is 8GB, Xmx could be 4GB)...

                       

                      Regards,

                       

                      Robin.

                      • 8. Re: java_pid****.hprof generated by AO

                        Robin,

                         

                        Thanks for your suggestion.

                         

                        But still we didn't remove those two parameter from .lax file.

                        • -XX:+HeapDumpOnOutOfMemoryError 
                        • -XX:HeapDumpPath="d:\\Program Files\\BMC Software\\AO\\tomcat\\logs"

                         

                        We found some process is trying to retrieve data from a file (size of almost 40 MB) and becasue if this AO JVM is falling down.Acctually AO File Adapter is unable to process a large file.

                         

                        Regards,

                        Kalyan Halder

                        • 9. Re: java_pid****.hprof generated by AO
                          Ranganath Samudrala

                          Good you found the source of the problem. What kind of a file is this? Are you using data in this file to perform any XPATH/XSLT transformation? If not, you should change the workflow so that file is not pulled into memory.

                           

                          Ranga

                          • 10. Re: java_pid****.hprof generated by AO

                            Ranga,

                             

                            First of all the file type is XML. and yes we are doing some transformation on File Adapter Response. But in our case File Adapter is unable to retrieve the data from that large XML file.No response from File Adapter at all.

                             

                             

                            Regards,

                            Kalyan Halder

                            • 11. Re: java_pid****.hprof generated by AO

                              Extremely large XML files do consume a lot of memory within BAO.  Testing in my environment, I was able to execute several reads on such a file if the -Xmx setting has 3-4GB assigned, but under load of other processes running you would likely still need a higher value of 8GB if this large file read occurs frequently (every minute or so).

                               

                              When dealing with such large objects you want to make sure you limit the passing of these Context values into sub workflows as much as possible or making copies of them into other context items.

                               

                              I am never much for telling people to drop an Adapter Activity into their own workflows, but in this case I would suggest it, if you have to read this file.  Skip the call to the AD-Utilities file read, and grab the same Adapter Request XML out of the Dev Studio processes.log and pass it directly to the Adapter Activity.  That will save your environment some memory overhead in reading such a file in a child workflow.  You may also want to immediately transform the returned adapter response and parse it down to just the XML information the rest of your workflow needs.

                               

                              Regards,

                               

                              Robin.