1 2 Previous Next 21 Replies Latest reply on Sep 13, 2019 1:47 AM by Enric Soria

    Tomcat Reaching the High memory utilization >>> FP 12.1.9

    abdallah hamaydeh
      Share This:

      Gents,

       

      We are facing that Tomcat 7 Reaching full of server memory. we did put max 15192MB and server have 32G, if we increase the max to 20G tomcat will keep going increase till 20G then system start hanging.

       

      we are using the following:

      Java Option:

      -Dfile.encoding=UTF-8

      -XX:+UseCompressedOops

      -XX:MaxMetaspaceSize=6192m

      -XX:+UseConcMarkSweepGC

      -XX:+CMSClassUnloadingEnabled

       

      Java version:

      Jre1.8.0_60  

       

      if we replace Java with jdk-8u144-windows-x64 it will make any difference?

       

       

      Thanks in advance!

        • 1. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
          abdallah hamaydeh

          Gents,

           

          After investigation the issue we found Tomcat open sessions without close it, so he kept adding in-memory Cumulatively.

          this is the only reasonable explanation, so how can we make sure that tomcat kill (close) as if we restarted

          • 2. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
            Eric Liszt

            Hi Abdallah,

             

            For most of our customers, we recommend that they configure Tomcat to automatically re-start in the evening during off-hours or on the weekend during maintenance windows.    See this knowledge article for information on how to make this happen for your system: How to automate Tomcat Service reboot ?

            1 of 1 people found this helpful
            • 3. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
              Stefan Hall

              Hi Eric,

              Really, restart should be the solution?

              Wouldn't it make more sense at some point to find the cause and solve it sustainably?

              • 4. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                S R

                We see the exact same thing, our Tomcat builds over time.  For us it really is apparent for publishing, each publish pushes the memory higher each time.  We've resorted to scheduling a reboot every Sunday.

                1 of 1 people found this helpful
                • 5. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                  abdallah hamaydeh

                  Thanks Mr.Eric,

                   

                  Restart is good for long term, but unfunctionally tomcat memory goes high several time during the working hours, we can't stop the service when there is more than 60 user using the system 4-5 times daily paces to resolve FP hanging.

                   

                  Case 00453007 Tomcat

                  and this very critical Case 00674324

                  2 of 2 people found this helpful
                  • 6. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                    Brian McKenzie

                    Sounds like tweaking the garbage collection settings in Tomcat will probably help. To much to go into detail here but, I would think support should be able to provide guidance.

                    1 of 1 people found this helpful
                    • 7. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                      Romuald Bois

                      Hello,

                       

                      The most commonly misconfigured settings on a production Tomcat server can be found in the attributes of the connector element, which is defined in the Tomcat server's server.xml file.

                      By default, the maxThreads attribute is set to a humble 200, which is fine for a single core machine, but can be multiplied linearly given the number of processors on a multicore production machine.

                      On a machine with four processors or more, setting this value to anything between 800 and 1000 will not cause a problem.

                      And if the configured number ends up far exceeding the number of threads required, the thread pool will naturally scale back this number when the server load is low.

                       

                      The other most often misconfigured, or for that matter, outright missing setting on a production Tomcat server has to do with compression.

                      Any communication between the client and server that is primarily text, be it HTML, XML or simply Unicode can regularly be compressed up to 90% using a simple and standard GZIP algorithm.

                      This can have a massive impact on reducing network traffic, allowing responses to be sent back to the client much faster, while at the same time allowing for more network bandwidth to be available for other network heavy applications.

                       

                      A sample connector entry in Tomcat's server.xml file that takes advantage of these settings would look like this:

                       

                       

                           <Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"

                                     connectionTimeout="20000"

                                     redirectPort="8443"

                         URIEncoding="UTF-8"

                         maxActive="20"

                         maxIdle="10"

                         initialSize="10"

                         maxWait="30"

                         maxThreads="800"

                         keepAliveTimeout="5000"

                         maxKeepAliveRequests="1000"

                         acceptorThreadCount="2"

                         acceptCount="100"

                         maxConnections="8192"

                         minSpareThreads="80"

                         maxHttpHeaderSize="65536"

                         compression="on"

                         compressionMinSize="2048"

                         noCompressionUserAgents="gozilla, traviata"

                         compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/x-javascript,application/javascript,application/xml,application/extjson,x-font/otf,application/x-font-woff,x-font/ttf,x-font/eot"

                                     />

                       

                       

                       

                          <!-- Define an AJP 1.3 Connector on port 8009 -->

                          <Connector port="8009" protocol="AJP/1.3" acceptorThreadCount="2" acceptCount="100" maxThreads="800" packetSize="65536" maxConnections="8192" minSpareThreads="80" keepAliveTimeout="5000" tomcatAuthentication="false" URIEncoding="UTF-8" />

                       

                       

                      JAVA OPTIONS TO ADD IN TOMCAT SERVICE:

                       

                      -Djava.awt.headless=true

                      -XX:NewSize=400m

                      -XX:MaxNewSize=400m

                      -Dfile.Encoding=UTF-8

                      -XX:+DisableExplicitGC

                      -XX:CompileThreshold=8000

                      -XX:+AggressiveOpts

                      -XX:+UseCompressedOops

                      -XX:+UseG1GC

                      -XX:+UseNUMA

                       

                       

                      Also possible :

                      -XX:MaxMetaspaceSize=6192m

                      -XX:+UseConcMarkSweepGC

                      -XX:+CMSClassUnloadingEnabled

                       

                       

                      WHAT DO THE PERFORMANCE SETTINGS RECOMMENDED ABOVE ACTUALLY DO? HERE'S AN EXPLANATION:

                       

                       

                      Setting Description:

                       

                      -XX:-DisableExplicitGC

                      This option disables calls to the function System.gc(), which is often run explicitly by many classes. When it runs, it triggers a full garbage collection process, which consumes a lot of execution time and results in inefficient heap usage. When it's disabled, the JVM still performs garbage collection nonetheless, whenever necessary.

                       

                      -XX:CompileThreshold

                      This option sets the number of method invocations/branches before compiling. The JVM usually waits for a method to be executed a certain number of times before it's compiled. Not compiling every method helps startup time and reduces RAM footprint. This option allows you to control that threshold.

                       

                      -Djava.awt.headless

                      This option, when set to true, prevents graphics rendering code from assuming that a graphics console exists, avoiding any chance of encountering an obscure Java bug related to X-servers and graphics rendering.

                       

                      -XX:+UseCompressedOops

                      The JVM has a special setting that forces using 32bits index internally (read: pointers-like). That allows to address 4 294 967 295 objects * 8 bytes address => 32 GB of memory. (NOT to be confused with the 4GB address space for REAL pointers).

                      It reduces the overall memory consumption with a potential positive impact on all caching levels.

                       

                      -XX:+UseNUMA

                      Non-uniform memory access (NUMA) is a computer memory design used in multiprocessing, the memory access time depends on the memory location relative to the processor.

                      Modern systems have extremely complex memory architectures with multiple layers of memory and caches, either private and shared, across cores and CPU.

                      Quite obviously accessing a data in the L2 cache in the current processor is A LOT faster than having to go all the way to a memory stick from another socket.

                      The NUMA-aware flag tells the JVM to optimize memory allocations for the underlying hardware topology.

                      The performance boost can be substantial.

                      2 of 2 people found this helpful
                      • 8. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                        abdallah hamaydeh

                        Thanks Mr. Romuald Bois

                         

                        I will try this first thing in the morning before the customer starts working.

                        • 9. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                          Romuald Bois

                          Hi,

                           

                          You can also find this interesting about tuning Tomcat in production environments :

                          Tuning JVM Garbage Collection for Production Deployments

                          1 of 1 people found this helpful
                          • 10. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                            abdallah hamaydeh

                            thanks a lot, Mr.Romuald Bois,

                             

                            So far so good till now

                            1- FP start up faster (memory was 1000MB) before it takes (memory around 3000MB)

                            2- now Memory stabile on 5000MB and we have 16-20 agent working on the system.

                             

                            Hope this will be the final solution that I will be

                            1 of 1 people found this helpful
                            • 11. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                              Stefan Hall

                              Hi abdallah hamaydeh,

                              Sounds good to me.

                              Here were some tips, Can you briefly summarize your changes for us.

                              • 12. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                                abdallah hamaydeh

                                Sure I will for all.

                                 

                                But let us wait till tomorrow to make sure that no surprises because will be more than 60 agents and huge end-user numbers access on the same time on Footprints

                                 

                                My head's under water but I'm breathing fine.

                                 

                                I mean thanks...

                                • 13. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                                  Romuald Bois

                                  Hi again,

                                   

                                  I have found that adding these variables to the JAVA options in Tomcat service seem to make  a positive difference in Garbage Collection handling :

                                   

                                  -XX:MaxGCPauseMillis=200

                                  -XX:ParallelGCThreads=20

                                  -XX:ConcGCThreads=5

                                  -XX:InitiatingHeapOccupancyPercent=70

                                  2 of 2 people found this helpful
                                  • 14. Re: Tomcat Reaching the High memory utilization >>> FP 12.1.9
                                    abdallah hamaydeh

                                    Dear Mr. Romuald,

                                     

                                    Unfortunately, Tomcat still reaching maximum utilization memory and I have created new case support Case Number 00679842

                                    1 2 Previous Next