ERROR (8505): Error sending message through Java Messaging Service; Server group unable to create new native thread

Version 1
    Share:|

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Remedy AR System Server


    APPLIES TO:

    9.1.03 and above



    PROBLEM:

    After restarting the system it was observed that the system appeared stable and functioning for a number of hours. And then they receive the following:

    =============

    Wed Aug 23 08:32:56.284 2017 Extension loaded: com.bmc.arsys.assignmentEngine 9.1.3.SNAPSHOT BMC.ARS.ASSIGNMENTENGINE
    Wed Aug 23 19:45:04.034 2017 Exception in thread "ActiveMQ BrokerService[broker_0] Task-321"
    Wed Aug 23 19:45:04.035 2017 java.lang.OutOfMemoryError: unable to create new native thread
    Wed Aug 23 19:45:04.036 2017 at java.lang.Thread.start0(Native Method)
    Wed Aug 23 19:45:04.036 2017 at java.lang.Thread.start(Thread.java:714)
    ....
    Wed Aug 23 19:45:04.037 2017 at java.lang.Thread.run(Thread.java:745)

    Wed Aug 23 19:45:04.051 2017 Error while broadcasting messages to servergroup. Message[UserReleasedMessageData[ messageType=UserReleased userName=webservices cacheId=1276 super.toString()=ServerGroupMessageData [messageType=UserReleased, callType=null, super.toString()=AbstractMessageData [serverNameOfMessageOrigin=sv2lxremasuat02, delayOutsideMsgDelivery=false]]]]ERROR (8505): Error sending message through Java Messaging Service; Server group unable to create new native thread

    Wed Aug 23 19:48:04.349 2017 Exception in thread "ActiveMQ BrokerService[broker_0] Task-320"
    Wed Aug 23 19:48:04.352 2017 java.lang.OutOfMemoryError: unable to create new native thread
    Wed Aug 23 19:48:04.354 2017 at java.lang.Thread.start0(Native Method)


    =============
     


    SOLUTION:


    ===================================================================
    Apply hotfix  9.1.03 201708040632  which includes and enhancement fix - SW00530401
    ===================================================================


    1.Change the following parameter in arserverd.conf   - explicity in bytes

    From :
    jvm.minimum.heap.size=8g
    jvm.maximum.heap.size=8g

    To :
    jvm.minimum.heap.size=8589934592    (explicity in bytes)
    jvm.maximum.heap.size=8589934592

     
    2. Mapped the JVM Options parameter of both of the server by removing the duplicates and maintaining the sequence of options :

    jvm.option.1 = -javaagent:../lib/spring-instrument-4.1.9.RELEASE.jar
    jvm.option.2 = -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
    jvm.option.3 = -Dlogback.configurationFile=file:../conf/logback_server.xml
    jvm.option.4 = -Xss2M
    jvm.option.5 = -XX:MetaspaceSize=384M
    jvm.option.6 = -XX:MaxMetaspaceSize=512M
    jvm.option.7 = -Dcom.sun.management.jmxremote
    jvm.option.8 = -Dcom.bmc.arsys.boot.flavor=server
    jvm.option.9 = -XX:OnOutOfMemoryError=./kill-server
    jvm.option.10 = -Djetty.home=../jetty
    jvm.option.11 = -Dorg.eclipse.<servername>.http.jetty.autostart=false
    jvm.option.12 = -XX:ErrorFile=file:../db/arserverjvmcrash_PID%p.log
    jvm.option.13 = -XX:+HeapDumpOnOutOfMemoryError
    jvm.option.14 = -XX:HeapDumpPath=../Logs
    jvm.option.15 = -XX:+UseCompressedOops
    jvm.option.16 = -XX:+UseConcMarkSweepGC
    jvm.option.17 = -XX:+UseParNewGC
    jvm.option.18 = -XX:NewRatio=2
    jvm.option.19 = -XX:+DisableExplicitGC

    3 .From Remedy Administration Console -> Centralized Configuration -> com.bmc.arsys.server -> 

    Broker-Max-Memory-Usage  1024
    Broker-Max-Temp-Storage  16384
    Broker-Send-Failed-No-Space-After-Timeout  30
    Broker-Temp-Storage-Directory     /remedy/bmc/ARSystem/db/activemq-data/broker_0/tmp_storage



    4.  Change the ulimit –n to 50000 

    ======================================
    You must allow the following ports on each server:
    ======================================

    1099    RMI port required by ActiveMQ (not configurable).
    61617    JMS broker port for ActiveMQ (Default-messaging-port).
    40001    Port on which the RMI Registry listens for ehcache (Peer-listener-port).
    40002    Port on which the remote objects bound in the registry receive calls for ehcache (Peer-Remote-Object-Port).
    10001    Shutdown port (not configurable).
    61500    Port for JMX connections (Jmx-port).


    https://docs.bmc.com/docs/ars91/setting-ports-and-rpc-numbers-609073765.html


    Article Number:

    000142298


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles