Utilizing JSSO with ARS 9.x encounters repeated server hangs

Version 2
    Share:|

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Remedy AR System Server


    COMPONENT:

    AR System Server


    APPLIES TO:

    ARS 9.x



    PROBLEM:

    ARS 9.x environment utilizing 3rd party JSSO plugin

    Eventually server starts to hang

    Java thread dump shows FAST threads ( pool-Fast-thread ) being blocked on authentication activity.  With heavy user activity all FAST threads eventually start being blocked as users attempt to login.

    Here is a snippet of the *.tdump file generated using jvisualvm.

    "pool-Fast-thread-29" #1373 prio=5 os_prio=0 tid=0x0000000001571800 nid=0x67c7 waiting for monitor entry [0x00007f8441539000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at com.bmc.arsys.server.domain.service.impl.AccessControlServiceImpl.authenticate(AccessControlServiceImpl.java:372)
        - waiting to lock <0x0000000705107c20> (a java.lang.String)
        at com.bmc.arsys.server.internal.aspects.AccessControlInterceptor.monitorFacadeMethod(AccessControlInterceptor.java:151)
        at com.bmc.arsys.server.facade.impl.AccessControlFacadeImpl.verifyUser_aroundBody8(AccessControlFacadeImpl.java:63)
        at com.bmc.arsys.server.facade.impl.AccessControlFacadeImpl$AjcClosure9.run(AccessControlFacadeImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.bmc.arsys.server.internal.aspects.ApiTracer.monitor(ApiTracer.java:179)
        at com.bmc.arsys.server.facade.impl.AccessControlFacadeImpl.verifyUser(AccessControlFacadeImpl.java:63)
    --
    "pool-Fast-thread-28" #1372 prio=5 os_prio=0 tid=0x0000000001b55000 nid=0x67c6 waiting for monitor entry [0x00007f844173a000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at com.bmc.arsys.server.domain.service.impl.AccessControlServiceImpl.authenticate(AccessControlServiceImpl.java:372)
        - waiting to lock <0x0000000705107c20> (a java.lang.String)
        at com.bmc.arsys.server.internal.aspects.AccessControlInterceptor.monitorFacadeMethod(AccessControlInterceptor.java:151)
        at com.bmc.arsys.server.facade.impl.AccessControlFacadeImpl.verifyUser_aroundBody8(AccessControlFacadeImpl.java:63)
        at com.bmc.arsys.server.facade.impl.AccessControlFacadeImpl$AjcClosure9.run(AccessControlFacadeImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.bmc.arsys.server.internal.aspects.ApiTracer.monitor(ApiTracer.java:179)
        at com.bmc.arsys.server.facade.impl.AccessControlFacadeImpl.verifyUser(AccessControlFacadeImpl.java:63)

     


    CAUSE:

    Insufficient threads cause bottleneck therefore FAST threads hang


    SOLUTION:

    This requires setting both the Plugin-AREA-Threads setting for arplugin and the numCoreThreads for the java plugin server 9999 to avoid the hanging condition.   Both parameters need to be set.  
    When both parameters are defined you will see multiple threads entries for JSSO in arplugin.log file reported.

    Recommended the following settings:

    ar.conf
    ======
    Plugin-AREA-Threads: 12    12

    pluginsvr_config.xml
    ================
    <numCoreThreads>30</numCoreThreads>

    Note:  Making the changes via the Central Config form.

    For numCoreThreads select the com.bmc.arsys.pluginserver component running on port 9999.  Modify the existing parameter raising from the default value of 5 to 30.
    For the Plugin-AREA-Threads parameter select the com.bmc.arsys.server component.  Add the entry if it does not exist.

    Recommend making these changes across all servers if in a server group.

    It is also advisable contacting the Vendor for JSSO plugin and upgrade to the latest version.


    Article Number:

    000118855


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles