1 Reply Latest reply on Oct 31, 2016 4:02 PM by Davin Lindner-Green

    SSO token cache blocking midtier threads

    Davin Lindner-Green
      Share This:

      Hello all,

       

      I don't know if this belongs in the Atrium SSO community or the Midtier. I have a situation where my Remedy midtier server frequently locks up, becomes unresponsive, for several minutes. Midtier issue right? Perhaps not...see below...

       

      Through the Java JMX Console (JVisualVM) I have observed all of the request processing threads blocking at the same time on an object called "com.bmc.atrium.sso.agents.web.jee.SsoTokenCache"

       

      I don't know the internals of the ATSSO agent on the midtier but just from the class name I can extrapolate that there is some single threaded code in there somewhere doing something with an Atrium SSO token cache and for some reason, at certain times, most or all AJP processing threads are blocked on it.

       

      Anyone have any idea about this? Suggestion for configuring it better? Excerpt of thread dump is below.

       

      Thanks!

      Davin

       

      Atrium SSO 9.0.1 (Windows 2012)

      Remedy Midtier 8.1.2 (Windows 2012, IIS, Tomcat 7, Java 8)

       

      Thread dump (this is the thread that all the other ajp-nio-8009-exec-* threads are blocking on):

       

      "ajp-nio-8009-exec-125" - Thread t@20636
         java.lang.Thread.State: BLOCKED
      at com.bmc.atrium.sso.sdk.impl.Notification$Notify.add(Unknown Source)
      - waiting to lock <321f7316> (a java.util.ArrayList) owned by "pool-1-thread-1" t@78
      at com.bmc.atrium.sso.sdk.impl.Notification.addListener(Unknown Source)
      at com.bmc.atrium.sso.sdk.impl.SSOTokenImpl.addSSOTokenListener(Unknown Source)
      - locked <5b3336ab> (a com.bmc.atrium.sso.sdk.impl.SSOTokenImpl)
      at com.bmc.atrium.sso.agents.web.jee.SsoTokenCache.getNewToken(Unknown Source)
      at com.bmc.atrium.sso.agents.web.jee.SsoTokenCache.get(Unknown Source)
      - locked <4080219e> (a com.bmc.atrium.sso.agents.web.jee.SsoTokenCache)
      at com.bmc.atrium.sso.agents.web.jee.Authenticate.getToken(Unknown Source)
      at com.bmc.atrium.sso.agents.web.jee.Authenticate.process(Unknown Source)
      at com.bmc.atrium.sso.agents.web.jee.JEEFilter.doFilter(Unknown Source)
      at com.bmc.atrium.sso.agents.web.SSOFilter.doFilter(Unknown Source)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
      at org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:175)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1700)
      - locked <7dd29a9f> (a org.apache.tomcat.util.net.NioChannel)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Unknown Source)

         Locked ownable synchronizers:
      - locked <41a43824> (a java.util.concurrent.ThreadPoolExecutor$Worker)

        • 1. Re: SSO token cache blocking midtier threads
          Davin Lindner-Green

          I'm following up with a resolution to this problem that I posted a year ago, in case anyone else can benefit from it. Working with support I received a set of extra parameters to add to the Midtier Java environment (in Tomcat for example, use the tomcat7w.exe utility, go to the Java tab, and add these to the end of the JVM arguments):

           

          -Datsso.ttl.token.cache.entry.ms=1800000

          -Datsso.token.cleaner.period.minutes=90

          -Datsso.network.timeout=30000

           

          I believe the ttl.token.cache is the time that an SSO session token is allowed to reside in the MT's local cache of SSO tokens, before needing to be validated by communication between the MT and ATSSO server. This is in milliseconds to the value above is 30 minutes, an increase over the default (I don't know the default). Less communication between MT and ATSSO may avoid some problems.

           

          The token.cleaner parameter is a process that checks for expired tokens on the MT side, and by default runs every 10 minutes. This increases it to 90 minutes.

           

          The network.timeout is in milliseconds, I don't know the default but the 30,000 would increase it to 30 seconds.

           

          Support said that the root cause of the problem was an unstable network connection (communication between MT and Atrium SSO being interrupted). So these settings make the process more resilient in the face of potential dropped connections or handshakes between the two. So presumably you could increate or decrease them to try and get better results.

           

          Thanks

          Davin