AR System® 5.1.2 Mid-Tier: Performance Tuning Apache 1.3.27 and ServletExec 4.1.1

Version 2

    This week's theme:  Cool Tech Tips

    AR System® 5.1.2 Mid-Tier: Performance Tuning Apache 1.3.27 and ServletExec 4.1.1

    Overall performance for a system is generally dictated by the speed of its slowest link. Therefore, identifying bottlenecks and updating slower areas will have a large impact on improving your system performance. Since the Web server bears the brunt of the large end-user loads with frequent automatic re-tries on failure – it is more often than not one of the most important components to tune.

    Tuning your environment should be an ongoing activity. Because each environment and operating system is different, there isn't a single correct solution. You need to take the network, operating system memory, disk Input/Output (I/O), swap, and countless other factors into consideration. Of those factors, I/O operations are among the slowest and most time-consuming tasks performed by a database management system.

    This document will discuss how to tune your Remedy Mid-Tier 5.1.2 with the Apache 1.3.27 Web server and ServletExec 4.1.1 on a Solaris 8 UNIX Operating system. We used the configuration described in this article to simulate a test with 500 to 1000 AR System users. Please be sure to refer to your Apache documentation as an additional resource, since new versions of Apache have different configuration files and differ on how it is set up.

    The Apache Web Server uses HTTP to talk to the browsers. HTTP is a simple request-response interaction, which can be viewed as a "Web Transaction." The following describes tasks associated with this transaction:

    1. Maps the server name to an IP Address
    2. Establishes a TCP connection with the server
    3. Transmits the request (URL + method + other information)
    4. Receives the response (HTML text or image or other information)
    5. Closes the TCP/IP connection (in HTTP 1.1. the connection remains open for embedded images)


    Before tuning your Web Server and ServletExec, make sure the other parts of your system are tuned correctly. To illustrate some of the types of tuning you may want to explore, the following are some hints for tuning your AR System Server and your Oracle Database. There are many other performance tuning variables to consider.  This is just an example.

    AR System Server Tuning

    • Add the /usr/lib/lwp library path to the start/stop script at the beginning of the LD_LIBRARY_PATH environment variable value. Add it to the beginning so it searches these libraries first, which helps improve performance.
    • Remedy workflow
      • Make sure that when your end users are searching on a specific field in your forms they are indexed and the QBE search is set to leading or equal.
      • Refer to Remedy's Education website to learn more about training options to help you achieve optimal Remedy AR System server performance

    Database Tuning

    • Determine the RAM on your Oracle Server. Add as much memory to the database as you can spare. For performance, it is better to use memory instead of the physical disk. The processing demands of the Oracle database must not exceed the real RAM memory of the server.
    • Understand the importance of a Swap Disk. The swap disk is a special disk area that provides for RAM sharing, primarily by sorting page frames of inactive disk programs. The least-frequently-used RAM pages are offloaded, so that new applications can simultaneously share the same memory. After inactive RAM frames are paged-out to disk, the operating system can utilize the freed memory for other active tasks.
    • Ask your Database Administrator to do the following:
      • Re-organize your database
      • Run Statistics to make sure all indexes are being utilized
      • Optimize Cache Hit ratios of 95% or better

    Note: Please refer to your Oracle documentation for detailed tuning information.

    Tuning the Web-Tier

    There are four components of the Web-Tier that require separate tuning.

    • Operating System
    • Web Server
    • Servlet Engine
    • Remedy Mid-Tier

    Operating System

    • In our Solaris test scenario, we added the maximum and current file descriptor limits by editing the /etc/system kernel file.  In our test scenario we set the limit to 30000.
      set rlim_fd_max=30000
      set rlim_fd_cur=30000
    • Verify Swap Space (Swap Disk)
      Tune for enough swap space. Swap space should be at least 2 times the amount of your physical RAM memory. If the memory of your operating system is 2GBs, then you swap space should be around 4GBs. This is done at the time your operating system is installed, so you can have the swap space on a separate disk partition. Please refer to your operating system documentation for the correct swap space size.

    Apache Configuration File: httpd.conf

    MaxClients and StartServers are two of the many parameters that can be changed to improve performance on your Apache webserver. Startservers is one of the parameters used to regulate how the parent processes creates children to serve requests. The MaxClients parameter sets the hard limit on the number of httpd processes that a machine can create. To increase the value of these parameters, edit the httpd.conf Apache 1.3.27 Configuration File. Please refer to the Apache website for more detailed information on this topic: On the Apache website there is an article on performance tuning the Apache webserver, which can provide some useful guidelines:

    1. cd <apache_install_dir>/config
    2. Edit the httpd.conf file
    3. In our testing scenario, we changed the StartServers to 10. This number may vary, depending on the number of users. You may have to fine tune this value after performing stress tests.
    4. In our scenario, we changed the MaxClients parameter to 1000. MaxClients should be big enough to handle as many simultaneous requests as you expect to receive, but small enough to assure that there is enough physical RAM for all processes. In general, Apache is very self-regulating, so most sites do not need to adjust these directives from their default values. Sites which need to serve more than 256 simultaneous requests may need to increase MaxClients, while sites with limited memory may need to decrease MaxClients to keep the server from thrashing (swapping memory to disk and back).

    Other parameters that can affect your performance include: MaxKeepAliveRequests, MaxRequestsPerChild, MaxSpareServers, and MinSpareServers. The following article has information about these parameters:


    Different versions of Apache 1.3.27 webserver have hard-coded settings in the httpd.h include file for the MaxClients directive. If the HARD_SERVER_LIMIT is not increased before the Apache webserver is installed, then the maximum limit for MaxClients is 256. If you need to increase the MaxClients in the httpd.conf file, you need to update the include file and change the HARD_SERVER_LIMIT from 256 to 2000 or greater and then re-compile and install the Apache webserver.

    In our testing scenario, we changed the MaxClients parameter from 150 to 1000. The HARD_SERVER_LIMIT was changed from 256 to 2000 before the re-compile and install of the Apache Web server. It is ideal if you make the changes to your Apache include files ahead of time before your initial installation of your Apache Web server.

    Following are excerpts from the httpd.h and httpd.conf file that includes information about the HARD_SERVER_LIMIT and MaxClients parameters. These are the sections that need to be changed if you want to revise the value of these parameters.

    Section from the httpd.h include file regarding HARD_SERVER_LIMIT:

    /* Limit on the total --- clients will be locked out if more servers than
    *this are needed. It is intended solely to keep the server from crashing when things get out of *hand. We keep a hard maximum number of servers, for two reasons --- first off, in case *something goes seriously wrong, we want to stop the fork bomb short of actually crashing the *machine we&#39;re running on by filling some kernel table. Secondly, it keeps the size of the scoreboard file small enough that we can read the whole thing without worrying too much about *the overhead.
    #ifdef WIN32
    #define HARD_SERVER_LIMIT 1024
    #define HARD_SERVER_LIMIT 2000

    Section from the Apache httpd.conf file regarding MaxClients:

    # Limit on total number of servers running, i.e., limit on the number
    # of clients who can simultaneously connect --- if this limit is ever
    # reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
    # It is intended mainly as a brake to keep a runaway server from taking
    # the system with it as it spirals down...
    MaxClients 1000

    When the Apache Web server doesn&#39;t have enough MaxClients the ServletExec engine crashes. This is an excerpt from the Apache Web server error log:

    [Fri Mar 21 16:38:07 2003] [crit] (125)Address already in use: make_sock: could not bind to port 80
    [Fri Mar 21 16:41:00 2003] [notice] Apache/1.3.14 (Unix) configured -- resuming normal operations
    [Fri Mar 21 17:06:57 2003] [error] server reached MaxClients setting, consider raising the MaxClients setting
    [Fri Mar 21 17:07:12 2003] [error] [http://client|http://client] Failed to read command byte from ServletExec.

    There are other variables that can be changed to improve performance. For more information on performance tuning, visit the Apache website.

    New Atlanta StartServletExec Startup Script

    Performance will improve if you increase the maximum size of the Java Virtual Machine. To increase the Java Virtual Machine maximum size, edit the startup script and add the JOPTS environment variable.

    The startup script is located in this directory:
    cd <ServletExecAS_directory>/se-<hostname>/

    Add the following JOPTS environment variable:
    JOPTS=&quot;-Xmx1000m&quot; ; export JOPTS

    The default RPC response size is 50K. Changing it to 1MB will result in better throughput. Instead of 50K chucks across the network, the packets will be in 1MB chucks for the RPC Response packets. This will make the Mid-Tier perform better. To change the RPC Response Size edit the startup script with the following change:

    Remedy Mid-Tier

    To tune the Remedy Mid-Tier, increase the maximum pooling total connections and connections per server parameters. For example:

    1. cd <midtier_install_dir>/WEB-INF/classes
    2. Edit the file and increase the following parameters:
    • arsystem.pooling_max_total_connections = 2000
    • arsystem.pooling_max_connections_per_server=1000

    In summary, performance tuning of the Web-Tier requires careful tuning for each of the following components: Operating System, Web Server, ServletExec, and Remedy Mid-Tier. Additionally, the AR System Server and database need to be tuned. To do performance tuning correctly, it needs to be prioritized as an ongoing project.

    Sr. Software QA Engineer
    Joined Remedy Customer Support UNIX Server/Database Team in 1995. Joined Remedy&#39;s Software QA Engineering Team in 1999. Conducted performance testing for Remedy Web-based products and developed and executed AR System Mid-Tier with Help Desk scripts for Mid-Tier performance benchmark testing.
    &quot;The Remedy Tenets that are special to me are: We are all sales and support people and We are our customer&#39;s advocates. When testing our product, I like to put myself in place of the customer to ensure a positive experience.&quot;