Tomcat Clustering in MidTier

Version 8
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Remedy AR System Server


    COMPONENT:

    AR System Mid Tier


    APPLIES TO:

    Tomcat-Clustering;deploy the mid tier in a Tomcat cluster;jvmRoute;clusterable ;distributable;      ;        ;          ;MessageDispatch15Interceptor;MessageDispatchInterceptor;ClusterListener;228.0.0.4;            ;distributable;multicasting;multicast;Mcast              



    PROBLEM:

     Tomcat Clustering configuration and related issues


    SOLUTION:

    NOTE: Before you configure the cluster, ensure that all the severs in cluster are in same time zone and are in sync.

       
    • AR mid tier web application is clusterable since version 7.6.04. however, the clustering optimization is implemented in version 9.0 and higher
       Reference : https://docs.bmc.com/docs/brid1808/configuring-the-tomcat-cluster-for-mid-tier-817165486.html
    To deploy the mid tier in a Tomcat cluster, you need to perform below 3 steps.  
       
    1. Configure a tomcat cluster
    2.  
    3. Configure the deployment of the mid tier to be distributable, and 
    4.  
    5. Configure the load-balancer to change routing on detection of a node failure.
       --------------------------------------------------------------------------------------------------------------------------
    A.
    Configure a tomcat cluster
    Configure and validate clustering reference link: https://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html)  
       
    1. Stop Tomcat.
    2.  
    3. Open the arsys.xml file located in the TomcatInstallationFolder\conf\Catalina\localhost folder in a text editor, and replace the <Manager> tag with the following XML:
      
    <Manager className="org.apache.catalina.ha.session.DeltaManager"  expireSessionsOnShutdown="false" notifyListenersOnReplication="true" />
      
       
    1. Open the server.xml file located in the TomcatInstallationFolder\conf folder in a text editor and make the following changes:     
           
      • Locate the <Engine> tag and add the jvmRoute attribute. 
        Use Node1 for the first node, Node2 for the second, and so on.
      •    

      • <Engine name="Catalina" defaultHost="localhost" jvmRoute="Node1">
      •    
      •  
      •    
      • Copy the <Cluster> tag information from the attached file and paste it after the <Engine> tag in your server.xml file. You can download the sample server.xml file. 
      •   
       NOTE:
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    In tomcat 7.x its experienced <Cluster> tag information from the attached file and paste it under the tag child of<Host> as opposed to <Engine> 
    Refer Link: https://serverfault.com/questions/298886/farm-deployer-not-working-in-tomcat-cluster 
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    MessageDispatch15Interceptor gives error as Tomcat 8.5 catalina-tribes.jar no longer includes the MessageDispatch15Interceptor class.  
    Change the tag from <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch 15 Interceptor"/> 
    to 
    <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/> 

    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    If you are using Tomcat version 8.0.38 and above, you must remove the following XML tag from the server.xml file.
    <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    There are two ways to add node to a Tomcat clusters, Multicast and Static Membership. 

    Multicast: When this is used all the tomcats will start listening and transmitting on a multicas IP address and port and built it's own topology, as long as all nodes include the same Multicast IP and Port they will see each other if they are reachable within the network. 

    For versions of Tomcat prior to 8.0.38, you can use the   server.xml file as is without making any changes.  
    <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
      
    <Manager className="org.apache.catalina.ha.session.DeltaManager"
      
                       expireSessionsOnShutdown="false"
      
                       notifyListenersOnReplication="true"/>
      
    <Channel className="org.apache.catalina.tribes.group.GroupChannel">
      
              <Membership className="org.apache.catalina.tribes.
      
    membership.McastService"
      
                            address="<ipAddress>" 
      
                            port="45570"
      
                            frequency="500"
      
                            dropTime="3000"/>
      
            <Receiver className="org.apache.catalina.tribes.transport.
      
    nio.NioReceiver"
      
                          address="auto"
      
                          port="4000"
      
                          autoBind="100"
      
                          selectorTimeout="5000"
      
                          maxThreads="6"/>
      
            <Sender className="org.apache.catalina.tribes.transport.
      
    ReplicationTransmitter">
      
                  <Transport className="org.apache.catalina.tribes.transport.
      
    nio.PooledParallelSender"/>
      
            </Sender>
      
    <Interceptor className="org.apache.catalina.tribes.group.
      
    interceptors.TcpFailureDetector"/>
      
          <Interceptor className="org.apache.catalina.tribes.group.
      
    interceptors.MessageDispatchInterceptor"/>
      
    </Channel>
      
     <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
      
    filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt|.*\.jsp|.*\.swf|.*BackChannel/*|.*./resources/.*|.*./sharedresources/.*|.*./plugins/.*|.*./pluginsignal/.*|.*./imagepool/.*"
      
    statistics="false" />
      
    <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
      
     <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
      
         tempDir="/tmp/war-temp/"
      
         deployDir="/tmp/war-deploy/"
      
         watchDir="/tmp/war-listen/"
      
         watchEnabled="false"/>
      
    <ClusterListener className="org.apache.catalina.ha.session.
      
    JvmRouteSessionIDBinderListener"/>
      
    <ClusterListener className="org.apache.catalina.ha.session.
      
    ClusterSessionListener"/>
      
    </Cluster>
      
    Note 
    The default value for the   <ipAddress> variable is 228.0.0.4. This IP address and port number defines a unique cluster. However, for each member of a cluster, ensure that they have the same multicast IP address and port number. 
    Enter the multicast IP address of the host on which you want the session replication to take place in the   Address value instead of   auto. For example, in the   <Receiver> tag, modify   address="auto" to   address="172.x.x.x". For more information, see   FAQ about cluster configuration

    Static Membership: If you want to avoid multicast traffic over your network you can use the following cluster setup 

    <Cluster 
            channelSendOptions="8" 
            channelStartOptions="3" 
            className="org.apache.catalina.ha.tcp.SimpleTcpCluster"> 
        <Manager 
            className="org.apache.catalina.ha.session.DeltaManager" 
            expireSessionsOnShutdown="false" 
            notifyListenersOnReplication="true" 
        /> 
        <Channel className="org.apache.catalina.tribes.group.GroupChannel"> 
            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> 
                <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" /> 
            </Sender> 
         <Receiver 
                address="172.20.15.115" 
                autoBind="0" 
                className="org.apache.catalina.tribes.transport.nio.NioReceiver" 
                maxThreads="6" 
                port="4100" 
                selectorTimeout="5000" 
            /> 
         <!-- <Interceptor className="com.dm.tomcat.interceptor.DisableMulticastInterceptor" /> --> 
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor" staticOnly="true"/> 
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" /> 
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor"> 
                <Member 
                    className="org.apache.catalina.tribes.membership.StaticMember" 
                    port="4100" 
                    host="172.20.24.188" 
                    uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1}" 
                /> 
            </Interceptor> 
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>  
         </Channel> 
         <Valve 
            className="org.apache.catalina.ha.tcp.ReplicationValve" 
            filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" 
        /> 
        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" /> 
      </Cluster> 

    You can see there's a <Member> tag in this member you can add any other tomcat that is member of the cluster, by using the port and remote IP address and a unique ID, the local tomcat will detailed over the <Receiver> tag, you can list as many remote members as needed, for each member you need to add this tags with their unique values. 

    For example, Simulating you are adding two additional remote tomcats, one with IP addres 10.0.0.1 and other with 10.0.0.2
    Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor"> 
                <Member 
                    className="org.apache.catalina.tribes.membership.StaticMember" 
                    port="4100" 
                    host="10.0.0.1" 
                    uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1}" 
                /> 
                <Member 
                    className="org.apache.catalina.tribes.membership.StaticMember" 
                    port="4100" 
                    host="10.0.0.2" 
                    uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2}" 
                /> 
      </Interceptor> 


    (Optional) Add two new file handlers   5cluster.org.apache.juli.FileHandler and   6cluster.org.apache.juli.FileHandler in the handlers section to the   logging.properties file.  
    handlers = 1catalina.org.apache.juli.FileHandler, \
      
               2localhost.org.apache.juli.FileHandler, \
      
               3manager.org.apache.juli.FileHandler, \
      
               5cluster.org.apache.juli.FileHandler, \
      
                6cluster.org.apache.juli.FileHandler, \ 
      
                java.util.logging.ConsoleHandler
      
       
    1. (Optional) To increase the cluster logging, change the log level to FINE for the following file handlers in the logging.properties file located in the TomcatInstallationFolder/conf folder:
      5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler
      
    5cluster.org.apache.juli.FileHandler.level = INFO
      
    5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
      
    5cluster.org.apache.juli.FileHandler.prefix = cluster.
      
     
      
    6cluster.org.apache.juli.FileHandler.level = INFO
      
    6cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
      
    6cluster.org.apache.juli.FileHandler.prefix = ha.
      
     
      
    org.apache.catalina.tribes.MESSAGES.level = INFO
      
    org.apache.catalina.tribes.MESSAGES.handlers = 5cluster.org.apache.juli.FileHandler
      
     
      
    org.apache.catalina.tribes.level = INFO
      
    org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler
      
     
      
     
      
    org.apache.catalina.ha.level = INFO
      
    org.apache.catalina.ha.handlers = 6cluster.org.apache.juli.FileHandler --------------------------------------------------------------------------------------------------------------------------B.Configure the deployment of the mid tier to be distributable There are 2 methods:
      
    <>A.add the XML attribute/value distributable="true" to the deployment descriptor of the web application
      
    Open <tomcat dir>/conf/Catalina/localhost/arsys.xml
      
     
      
     <Context docBase="D:\Program Files\BMC Software\ARSystem\midtier" distributable="true" path="/arsys" reloadable="false" sessionCookiePathUsesTrailingSlash="false" />
      
     
      
    <>B.Add the XML tag <distributable />  as a child tag of the XML tag <web-app> in the file <mid tier dir>/WEB-INF/web.xml 
      
     
      
     
      
       
    1. Restart Tomcat.
    2.  
    3. Verify whether the node is added to the cluster:     
           
      • Open the ha.date.log file located at /opt/apache/tomcat7.0/logs for Linux and <C:\Program Files\Apache Software Foundation\Tomcat7.0\logs> for Windows.
      •    
      • Search for memberAdded entry. If you have added four mid tiers to the cluster, you should see three memberAddedentries and their IP addresses in the file for the other three members. Ensure that the IP address does not represent a localhost.
      •   
    ---------------------------------  load-balancer to change routing-------------------------------------------- 
    C.Configure the load-balancer to change routing on detection of a node failure

    a. Once you have a Tomcat cluster running with the mid tier web app deployed as distributable, we need to configure the load-balancer to distribute the user load across the cluster nodes
    Consult your load-balancer vendor documentation for the health monitoring of each node of the cluster and redistribute the HTTP requests on an unavailable node to an available node.
    In the BMC Performance Lab, the monitoring heartbeat is set at 15 seconds interval with 2 retries on a failure. This introduces a roughly 20 seconds indeterminate window, the interval whence an HTTP request might be routed to a failed node. You can configure the heartbeat at a higher rate depending on your requirement. However, note that more frequent heartbeats introduces extra overhead.
    Note 
    To   add an extra mid tier to the cluster, perform the same configurations on the extra (n+1) mid tier. 


    b.  Web Loadbalancer has a important role here,  
    -Make sure a node (mid-tier) is alive (failing to do this will route requests to dead nodes) 
    -Make sure a single request is sent to the same midtier unless node is dead (Even though it is a cluster the implementation doesn’t make Mid-tier a REST Full service). That is A sticky bit session persistence is still required . 



    --------------------------------------------  Questions on clustering----------------------------------------------------------- 
    D.How Does a Tomcat Midtier clustering fail-over react under fail-over (reference KBA 000163497)
                            
                                                       

    Customer has two midtier and two Arserver , with individual LB at web and ars layer. Customer is initially connected to tomcat-midtier 1 . Both Tomcat 1 and Tomcat 2 are configured in a Cluster. 


    Use Case 1 :
               
               
    1. Let’s say customer is filling out the form A , and when hitting the save button ; Tomcat service-Midtier1 goes down.
    2.          
    3. At this point the request is already sent . Will the Clustering ensure that the request is forwarded through the Midtier 2 ?
    4.          
    5. Or does it require a page refresh ?
    6.         
     
    Use Case 2 :
               
               
    1. Let’s say customer is filling out the details and the tomcat 1 goes down ; At this time the User has not Hit the Save button .
    2.          
    3. And Customer hits the Save button . Will clustering ensure that the request is automatically routed to the next Midtier 2 ?
    4.         
                                
    Answer                                                   
    Use Case 1:          
               
    1. Let’s say customer is filling out the form A, and when hitting the save button ; Tomcat service-Midtier1 goes down.
    2.          
    3. At this point the request is already sent. Will the Clustering ensure that the request is forwarded through the Midtier 2?
    4.          
    5. Or does it require a page refresh?
    6.         
    Answer: 
     
    Since the request to Submit has failed (due to MidTier 1 crashing during the request), the end user would receive an error message for that request (the specific error may vary depending on the exact state of things at the time). The load balancer (again, if configured correctly) would route the traffic to MidTier 2 on the next request. So, after the “error” the user should be able to just Submit again, and MidTier 2 would service the request. No refresh would be required.
     
    Use Case 2:          
               
    1. Let’s say customer is filling out the details and the tomcat 1 goes down ; At this time the User has not Hit the Save button .
    2.          
    3. And Customer hits the Save button . Will clustering ensure that the request is automatically routed to the next Midtier 2 ?
    4.         
    Answer:
     
    The request should be routed thru MidTier 2 (again, assuming the env is configured correctly and the load balancer is aware to route traffic to MidTier 2). This request should work. 
     
      
    ------------------------------------------- Troubleshooting  -------------------------------------
    E.
    Troubleshooting  
       
    1. After adding clustering if tomcat keep crashing with below error :
       org.apache.catalina.tribes.membership.McastServiceImpl.start Unable to join multicast group, make sure your system has multicasting enabled.
    org.apache.catalina.ha.tcp.SimpleTcpCluster.startInternal Unable to start cluster.
    java.net.SocketException: error setting options; No faulty members identified.
    org.apache.catalina.startup.Catalina.start The required Server component failed to start so Tomcat is unable to start.
    org.apache.catalina.tribes.ChannelException: java.net.SocketException: error setting options; No faulty members identified.

    Solution : Then Enable the multicasting  à open command line and run command à Run command à netsh interface ip show joins
    It will Displays IP multicast groups that have been joined on one or more interfaces. If an interface is not specified, multicast groups for all interfaces are shown
       
      2. After clustering tomcat not starting .
       Solution: The first location you should take a look at is catalina.out, it will tell you what might be wrong.
    If you can't make it works with multicast, just try static membership (I think it will simpler).
    Below is my config:
    Put the  <Cluster> node inside the <Host> element:

    3.  error in catalina logs of tomcat 7 "SEVERE: FarmWarDeployer can only work as host cluster subelement! "
    Solution:
    https://serverfault.com/questions/298886/farm-deployer-not-working-in-tomcat-cluster

     


    Article Number:

    000167460


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles