Health check from Remedy midtier perspective

Version 6
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Remedy AR System Server


    COMPONENT:

    AR System Mid Tier


    APPLIES TO:

    Midtier versions 8x and later



    QUESTION:

    How do I ensure the midtier component is configured correctly for better performance


    ANSWER:

    Health check areas
    1. Compatibility
     Advice: Check the compatibility matrix for the midtier version.  If midtier version is from a couple of years ago, it may not have been tested with newer platform tools, eg: OS, Java, Tomcat.
           Example compatibility check:   Midtier 9.0 is compatible with:

              Third Party Prerequisite: Java 7 and above  
            Enabling Tech: Apache Tomcat 7 and above  
            Release Date:04/30/2015 
      
        
          Known incompatibility:  Java 8 is not compatible with Midtier 7.6.04  
             
    2. Follow the Midtier Deployment knowledge article that pertains to the version of midtier in use:  
      Midtier 7.6.04 through 9.0           https://communities.bmc.com/docs/DOC-18162 
    Midtier 9.1 and later                    https://selfservice.bmc.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000h9kqCAA&type=FAQ
      
    On Windows OS, the JVM setting can be reviewed from the Tomcat Config panel - Java tab.   The Tomcat Config panel is accessed by executing the <Tomcat_install>\bin\tomcat#w.exe where # designates the Tomcat release..  
      For other OS, review the JAVA_OPTS settings in one of  the startup scripts. 
      
     
      JVM settings have the most impact
      
      With these settings, Heap size will be used and when reaching 70-90% of usage Garbage collector will be invoked and remove unused objects from it.  
      
       
    • Set heap size the same for min and max.  Note:  These may be higher if the OS and Java are 64 bit,and  the OS physical RAM is greater than HeapSize+3GB.
      
                            -Xms3072m -Xmx3072m          
      
       
    • Set  the ratio between memory segments
      
      -XX:NewRatio=2    
      
       
    • For *nix systems set font paths|
      
        
      
                 -Djava.awt.headless=true 
      
                 -Dsun.java2d.fontpath=JREPATH/lib/fonts  
      
       
    •   Set garbage collector settings like this
      
                      -XX:+UseConcMarkSweepGC  
                    -XX:+UseParNewGC  
      
       
    •  In case a problem happens, make sure JVM will log that information . The recommended settings are: 
      
                      -XX:+HeapDumpOnOutOfMemoryError   
                    -XX:ErrorFile=PATH/java_hs_err.log  
                    -XX:HeapDumpPath=PATH  
      
       
    •  CompressedOops . On or Off ?
      
            -XX:+UseCompressedOops  (enables it) 
      
      Some knowledge articles (including the midtier deployment knowledge article would recommend to enable it. It can be enabled, but be aware that we have seen problems with Java 6.  If the JVM crashes and produces an hs_err_pid file, we can identify if this option was causing it, and then disable it.  There are no definitive measurements of performance gain/loss with or without this flag enabled. 
      
      Option A: Play it safe and disable it.              -XX:-UseCompressedOops  
    Option B: Enable it and in case it crashes disable it using Option A 
      
       
    •  Provide space for Classes storage
      
                      For java 7 or lower:  -XX:MaxPermSize=256m  
                    For java 8 or better: -XX:MaxMetaspaceSize=512m 
      
       
    • Regarding monitoring tools 
      
      To monitor java performance of the server from a remote desktop, add 
      
                        -Dcom.sun.management.jmxremote  
                      -Dcom.sun.management.jmxremote.port=19999  
                      -Dcom.sun.management.jmxremote.ssl=false  
                      -Dcom.sun.management.jmxremote.authenticate=false  
                      
                    If jVisualVM can be used, gather a snapshot to have a better understanding of your JVM performance.  
      
       
    • For Solaris only:  
      
       Add -d64   
      
      This forces the JVM to run in 64 bit mode. Without it, Solaris will run the JVM in 32 bit mode and heap will be limited to 2 GB.  Allocating more than 2 GB heap in 32 bit mode will cause an error during runtime 
                     
    3.  Load balancer and Network configuration 
              For load balancer configuration, it’s the customers responsibility to verify that the load balancer is setup according to the following recommendations.
             
    The required settings are: 
     Load balancers in front of midtiers (between browsers and midtiers)   
       
    •  Session affinity must be turned On using cookie-based persistence. This applies to midtiers installed in a Tomcat clustered environment as well as standalone.
    •  
    •  Any session timeout set at the Loadbalancer level must be longer than the session timeout set in the midtier config tool.   If session cookies are used, then this requirement is mute.
    •  
    •  Enable the OneConnect profile for F5 loadbalancers.   This means that traffic is balanced at the HTTP request level and not the TCP Connection level.
                
       Load balancer between Midtier and AR server  
       
    • Session affinity should be turned Off
    •  
    • Under midtier connection settings, "Enable lifespan checkbox" should be ticked and the Connection Lifespan set to 15 minutes
    •  
    • Default values for Connection Pool settings are ok for 90% of cases
    •  
    • There will be a connection pool of 80 connections per AR server/ AR server Group. This pool would suffice to serve up to 4 or 5 times the number of users (320 to 400 human users)        
    •  
    • NOTE:  If the high watermark under connection pool goes above 40 connections (50%), collect midtier performance logs and analyze if any operation is taking longer than 10 seconds.   Long running api/sql calls may be the root cause of the high watermark and can be linked to AR server logs. 
    •  
    • OneConnect profile should NOT be used in the load balancer that resides between the midtiers and the AR Servers.
      
       
    1.  For more information, please see this link https://support.f5.com/kb/en-us/solutions/public/7000/200/sol7208.html
    2.  
    3.  Avoid using a OneConnect profile for non-HTTP virtual servers that process more complex transactions, such as FTP or RTSP.   Doing so may result in traffic disruption and session failure. Even for simple non-HTTP protocols, an iRule may be required to manage connection reuse.
      
       
    • Http profile should not be used in this load balancer. It can prevent the AR server to be added to the midtier config.jsp stating RPC errors
      
      Network health check
                    Make sure network is not dropping packets 
                    Make sure network MTU size is adequate   4.  Midtier hotfixes 
            For the most recent version, you can find the latest recommended hotfix here:    ftp://ftp.bmc.com/pub/ARRecommendedFixes/
             
    5.  Have a single AR server configured under the Midtier config tool - AR server settings 
            Having multiple AR servers configured under AR server settings will increase the amount of cache used. Extra AR servers can be added to help troubleshoot certain scenarios to help identify AR Server problems.  Once the issue is solved (eg. Schedule normalization engine), remove any extra AR servers from midtier configuration tools 
         
    6. Preload 
            Preload is a technique that will reduce the time to render the form for the first time. It works by gathering a list of active links from the AR server(s), then traverses the forms linked to them and builds partial cache. Once a user accesses a form that was preloaded, the form will finish its cache process (compile HTML will still be displayed on midtier logs).  
    In some cases forms or views cannot be identified by this process and then preload would not partially load those forms.  
            
    How to identify if a form is fully loaded into cache? Access config.jsp page, login and then modify the url to go to config_cache_adv.jsp.  Review the View Building Information (sample displayed following) found at the bottom of the config_cache_adv.jsp page.                                    
    View Building Information   
     
                                                                                                                                                                             
    SERVERFORM NAMEVIEW NAMEINLINELOCALETIMEZONEUSER NAMEGROUP-COMBOHIT COUNT
    clm-aus-019258AR System Customizable Home PageDefault Administrator Viewtrueen_GBEtc/GMT+12AllenAdmin1
    clm-pun-033729SHR:OverviewConsoleOverview Homepage Contenttrueen_USEtc/GMT+12AllenAdmin2
    All the views for these particular users would be in the cache.  
    Once a form is in this table (it is also in ViewStats.dat file), any process that rebuilds the cache (sync cache, or restarting mid-tier)  will fetch all these views.  Access to any of these views by users with same permissions will only see Compile JS. 

    Mid-tier will always collect information in ViewStats.dat files regardless of cache preferences or preload enabled or not.  
    After some time has passed and ViewStats.dat reflects the forms actually in use;  preload or prefetchConfig.xml would only become a burden at restart times and should be disabled. 
    If the viewStats.dat has been removed, enableing preload or prefetchConfig.xml may help  

            
    7.  Resource Check Intervals 
            For production midtiers, the cache should not be set to sync automatically. There are 2 ways to achieve this: 
            a. Disable Perform Check : This will still cache objects but won't ping AR server at regular intervals. However, this will disable  " sync cache" button that may be useful later. 
            b. If you still need sync cache, but want to check cache manually :  increase the Resource Check intervals from 86400 (one day) to 86400000 (1000 days) 

    WARNING:  Setting the Check Intervals to 0 will disable midtier cache and would require midtier to retrieve Form, permissions and Active links from AR server on each form request.  This would have impact performance.             
    8.  Enable Cache persistance 
            This will save cache to disk when the JSP engine is shutdown or restarted.  On startup, the cache will be retrieved from disk and only refresh any content that has been updated. 
             
    9. When do I need to flush the cache? 
            If content is promoted to production midtiers, start by syncing the cache. Midtier cache logs will show when this operation starts and finishes. If any content is not synced, try flushing the cache. If that fails, perform a hard cache flush for the whole cache to be rebuilt.  (See step 10).  Consider that hard cache flush will delete viewStats.dat and that file reflects the views more frequently used in your system. Without it rebuilding cache will be lengthier and depending on your configuration it may cache forms that are not really useful to users. 
             If you only have a handful of forms and the midtier is version 9x, use the http://<midtier>:<port>/arsys/shared/config_cache_cleanup.jsp page to clear specific forms from the cache.   Note:  form and active links will be cleared. 
            
     10. How to perform hard cache flush 
            Stop Tomcat (Jsp Engine) Service. 
            Clear out the contents of the 'work', 'temp' & 'logs' folders inside Tomcat X.X folder 
            Clear out the contents of the 'cache', 'cachetemp', 'attstore' and 'logs' folders present inside the Mid-Tier folder. 
            For versions 8.1.02 or earlier, Delete 'viewStats.dat' and 'viewStats.dat.bak' from <Mid-Tier folder>\WEB-INF\Classes  if present. 
            For versions 9.0 or later, delete the content of the <Mid-Tierfolder>\WEB-INF\classess\viewServerStat directory 
            Start the Tomcat (Jsp Engine) Service. 
            Clear browser cache/cookies/etc completely.  

             
    11.  Which logs should I have enabled in production: 
            During normal operation (no issues observed) 
                Go to Log Settings and enable the following categories 
                    Session Management, Performance,  Servlet 
                Make sure  
                    Log Level is set to: Info 
                    and log format: Detail text 
                    Maximum Log File Size (kb):  10240 
                    Maximum Number of Log Files : 10 
            If a generic problem is observed  
                Go to Log Settings and enable the following categories 
                    Cache, Session Management, Performance,  Servlet 
                    Consider filtering the logs by User 
                Make sure  
                    Log Level is set to: Fine 
                    and log format: Detail text 
                    Maximum Log File Size (kb):  10240 
                    Maximum Number of Log Files : 100 

    12. How to make sure your form will render properly.   
      Test any business crucial form using developer studio documentation found    here 
     
    Rendering a form on midtier is an intensive process, it involves several API calls to AR server, including gathering Groups, Roles and Active Links. Once Midtier has gathered this information, it still needs to transform the form definitions to HTML and the active links to Javascript.  When the user logs in, cache management will get the proper cached version for this user and send it to the browser for rendering. 
      
     
    We encourage to test business crucial forms by using the "view in browser" feature from developer studio. This will run a full cycle from start to finish and will allow to detect issues in development or testing , possibly avoiding a situation in production systems  
      
      
    Contact BMC support if help is needed . Provide evidence that this article has been followed (screenshots, midtier/WEB-INF/classes/config.properties file, jvm snapshot and versions from your midtier, OS, tomcat and JVM ) 

     


    Article Number:

    000120022


    Article Type:

    FAQ/Procedural



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles