3 Replies Latest reply on Mar 11, 2016 7:16 AM by Lubos Strejcek

    Deadlock in Tomcat 7 running Mid Tier 8.1.02

    Michael Galat
      Share This:

      Hi All -

       

      Has anyone run into this?  Occasionally I see the mid tier stop responding.  When I remote over to the server, I will find that the Tomcat7 process is completely idle - 0 CPU usage.   Looking in Java VisualVM, I see a message stating "Deadlock Detected".  I took a thread dump, loaded it into Notepad++ and searched for :

       

      - Thread |- locked |- waiting on |- waiting to lock

       

      What I found was many threads waiting for java.util.concurrent.locks.ReentrantLock$NonfairSync, owned and locked by t@89 and thread t@89 is waiting to lock java.util.HashSet which is owned by t@94.  Thread t@94 has java.util.HashSet locked, and is waiting to lock  java.util.concurrent.locks.ReentrantLock$NonfairSync, which is locked by t@89.  Here is an excerpt, showing the two threads:

       

      ...

      "http-bio-80-exec-40" - Thread t@94

      - waiting to lock <a4dc6e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by "http-bio-80-exec-36" t@89

      - locked <12274693> (a java.util.HashSet)

      - locked <74ab4012> (a org.apache.tomcat.util.net.SocketWrapper)

      - locked <4163ece6> (a java.util.concurrent.ThreadPoolExecutor$Worker)

      ...

      "http-bio-80-exec-36" - Thread t@89

      - waiting to lock <12274693> (a java.util.HashSet) owned by "http-bio-80-exec-40" t@94

      - locked <4155f93b> (a org.apache.tomcat.util.net.SocketWrapper)

      - locked <3d5b653a> (a java.util.concurrent.ThreadPoolExecutor$Worker)

      - locked <a4dc6e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

      ...

       

      Here are the complete stack dumps -

       


      "http-bio-80-exec-40" - Thread t@94
         java.lang.Thread.State: WAITING
      at sun.misc.Unsafe.park(Native Method)
      - waiting to lock <a4dc6e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by "http-bio-80-exec-36" t@89
      at java.util.concurrent.locks.LockSupport.park(Unknown Source)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
      at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
      at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
      at com.remedy.arsys.goat.AttachmentData$AttachmentDataCache.disposing(AttachmentData.java:1292)
      at com.remedy.arsys.stubs.SessionData.dispose(SessionData.java:373)
      - locked <12274693> (a java.util.HashSet)
      at com.remedy.arsys.session.LoginServlet.cleanStaleSessions(LoginServlet.java:517)
      at com.remedy.arsys.session.LoginServlet.doThePost(LoginServlet.java:197)
      at com.remedy.arsys.session.LoginServlet.doRequest(LoginServlet.java:72)
      at com.remedy.arsys.stubs.GoatHttpServlet.postInternal(GoatHttpServlet.java:97)
      at com.remedy.arsys.stubs.GoatHttpServlet.doPost(GoatHttpServlet.java:61)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
      - locked <74ab4012> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

         Locked ownable synchronizers:
      - locked <4163ece6> (a java.util.concurrent.ThreadPoolExecutor$Worker)
         
          "http-bio-80-exec-36" - Thread t@89
         java.lang.Thread.State: BLOCKED
      at com.remedy.arsys.stubs.SessionData.removeEventListener(SessionData.java:690)
      - waiting to lock <12274693> (a java.util.HashSet) owned by "http-bio-80-exec-40" t@94
      at com.remedy.arsys.goat.AttachmentData$AttachmentDataCache.removeAllUnused(AttachmentData.java:1463)
      at com.remedy.arsys.goat.AttachmentData$AttachmentDataCache.access$1300(AttachmentData.java:1252)
      at com.remedy.arsys.goat.AttachmentData.removeAllUnused(AttachmentData.java:1014)
      at com.remedy.arsys.stubs.AttachServlet.doDownload(AttachServlet.java:298)
      at com.remedy.arsys.stubs.AttachServlet.doRequest(AttachServlet.java:70)
      at com.remedy.arsys.stubs.GoatServlet.postInternal(GoatServlet.java:122)
      at com.remedy.arsys.stubs.GoatHttpServlet.doGet(GoatHttpServlet.java:56)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
      - locked <4155f93b> (a org.apache.tomcat.util.net.SocketWrapper)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

         Locked ownable synchronizers:
      - locked <3d5b653a> (a java.util.concurrent.ThreadPoolExecutor$Worker)

      - locked <a4dc6e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

       

      Just looking at it, my guess is that the Mid Tier software is not handling the locking of java.util.HashSet in a good manner.  This is the second time in a few months that I have seen this java.util.HashSet implicated in a deadlock.  Has anyone else seen this?  Are there any fixes for this?

       

      So far, the only recourse we have is to stop and restart Tomcat.

       

      Thanks,

      Mike