Tuning IBM HTTP Server to maximize the number of client connections to WebSphere Application Server

Resolving The Problem

First and foremost, you must determine the maximum number of simultaneous connections required for this Web server. Using mod_status or mod_mpmstats (available as part of the installation since 7.0) to display the active number of threads throughout the day will provide some starting data.

There are 3 critical aspects to MPM (Multi-processing Module) tuning in IBM HTTP Server.

1. Configuring the maximum number of simultaneous connections (MaxClients directive)
   
   
2. Configuring the maximum number of IBM HTTP Server child processes (ThreadsPerChild directive)
   
   
3. Less importantly, configuring the ramp-up and ramp-down of IBM HTTP Server child processes (MinSpareThreads, MaxSpareThreads, StartServers)

The first setting (MaxClients) has the largest immediate impact, but the latter 2 settings help tune IBM HTTP Server to accommodate per-process features in Apache modules, such as the WebSphere Application Server Web server plug-in.

Configuring the maximum number of simultaneous connections

If the number of concurrent connections required is smaller than the current value of MaxClients (default 600), then MaxClients should remain unchanged and further tuning is likely not required.

If the number of concurrent connections is larger than the current value of MaxClients, this will be a very serious inhibitor of performance. MaxClients represents the total number concurrent connections IBM HTTP Server can process.

1. Determine a new value for MaxClients, which is the desired number of total simultaneous connections you require the server to support. This new value should remain a multiple of ThreadsPerChild.
   
   
2. For values of MaxClients beyond approximately 2000, pay extra attention to testing under appropriately modeled load and watching for changes in response time, overall CPU usage, and the proportion of user to system CPU time used under load. The repercussions will vary based on other aspects of the confiuguration, operating system, workload, application, and topology.
   
   Some, but not all, of the repercussions of MaxClients in excess of 2000 would apply to multiple instances of IHS on the same system or virtual machine with an aggregate MaxClients in excess of 2000. For example, some things in common would be general scheduling overhead or competition for CPU or other kernel resources.
   
   For high availability and other operational advantages, webservers like IHS are typically deployed horizontally across multiple operating system instances. Usually this limits the necessity for drastic increases in MaxClients further scale-out should be preferred.
   
3. Update MaxClients in the httpd.conf file to reflect the desired capacity.
   
4. In order to increase MaxClients,it is typically necessary to make a corresponding increase in the value of the ServerLimit directive. ServerLimit does not directly change the available capacity, but acts as a limit on the total number of active IBM HTTP Server child processes.
   
   Calculate a new ServerLimit by dividing MaxClients by ThreadsPerChild(this ratio is the required number of processes). If ServerLimitexceeds this prescribed ratio, MaxClientscan then be increased during a graceful restart of IBM HTTP Server.
   
5. If MaxSpareThreads was previously equal to MaxClients, increase MaxSpareThreads to MaxClients.

Configuring the maximum number of HTTP Server child processes

The number of IBM HTTP Server child processes is controlled by the ratio of MaxClients divided by ThreadsPerChild. Since MaxClients is dictated exclusively by the amount of load the system will need to handle, ThreadsPerChild is the directive used to change this ratio.

This does not affect how much load IBM HTTP Server can handle, just the way that load is distributed among separate processes.

Some features of various Apache components might perform better when fewer child processes, each with more threads, are used. For example, the WebSphere Application Server Web server plug-in MaxConnections parameter and ESI cache are each per-process and are more effective with fewer child processes (higher ThreadsPerChild).

The number of IBM HTTP Server child processes should be independent of any WebSphere Application Server or front-end load balancer settings.

There are reasons to avoid a high value for ThreadsPerChild. Some operating system facilities might require per-process locks that more threads suffer contention on, such as access to the heap. A child process crash results in all threads terminating, so in environments with frequent child process crashes it is reasonable to use a lower ThreadsPerChild.

Third-party modules might have per-process caches or connection pools that might favor a larger ThreadsPerChild for efficiency purposes. However, their architectures could just as likely favor a smaller ThreadsPerChild if a per-process resource is protected by a mutex.

Generally, heavily loaded systems doing SSL favor a small ThreadsPerChild due to contention on the native heap.

If the ESI invalidation servlet is configured in the WebSphere Plugin, each additional process causes a dedicated web container thread to be permanently consumed.

If a change to the number of IBM HTTP Server child processes is required:

1. Determine the number of child processes desired, and divide the current value of MaxClients by this number.
   
   
2. Replace ThreadsPerChild with the value obtained from step 1.
   
   
3. Replace ThreadLimit with the value obtained from step 1.
   
   
4. If ThreadsPerChild has increased by a large amount, evaluate the current value of StartServers (the initial number of child processes created). Note that StartServers is not a very important tunable, because IBM HTTP Server re-evaluates the number of available threads every second.
   
5. Monitor the change to the system load (uptime) as well as the overall CPU usage, response time, and the ratio of user vs. system time for drastic changes to ThreadsPerChild.

Configuring the ramp-up and ramp-down of HTTP Server child processes

With the default httpd.conf file, IBM HTTP Server tries to maintain a pool of at least 25 spare processing threads and no more then 75 at any given time. These are specified with the MinSpareThreads and MaxSpareThreads directives respectively.

MaxSpareThreads
It is suggested that you disable the periodic termination of IBM HTTP Server child processes by setting MaxSpareThreads equal to MaxClients. Under this configuration, IBM HTTP Server child processes are created based on demand but not killed when the demand recedes. This is the recommended configuration because of complexities of graceful process termination and long-running requests; child processes are replaced as soon as they are scheduled for termination but might take considerable time to fully exit.

If periodic graceful child process termination is desired, MaxSpareThreads should be set to any value substantially smaller than MaxClients. If a module is maintaining a per-process cache or connection pool, like the WebSphere Application Server Web server plug-in, these will be discarded and created anew for the replacement child process.

Note: The default configuration has a relatively low value for MaxSpareThreads which errs on the side of conserving memory usage.

MinSpareThreads
IBM HTTP Server compares the number of available processing threads to MinSpareThreads every second to decide whether or not to spawn additional child processes. Each additional child process contains ThreadsPerChild number of threads.

MinSpareThreads is usually safely set to the same value (or double) ThreadsPerChild. Since this value is checked every second, it is not an especially sensitive setting.

Example 1:
This configuration uses multiple IBM HTTP Server child processes, with a fixed capacity of 2000 concurrent connections spread amongst 20 child processes.

If MaxConnections is used with the WebSphere Application Server Web server plug-in, MaxConnections will refer to the number of backend connections permitted in each IBM HTTP Server child process. While the number of processes is predictable in this case, the distribution of users over these processes is not. For this reason MaxConnections would be difficult to use in such an environment.

If the ESI cache is used, it is fragmented or duplicated between the child processes. Usually ESI caches are relatively small, and over the course of many thousands of requests, the reduction in hit rate is not a practical concern.

When a backend application server is misbehaving, each IBM HTTP Server child process must detect the markdown and the ultimately retry -- this information is not shared between child processes.

A crash of an individual child process affects between 1 and 100 threads only, depending on how busy the child process is.
<IfModule worker.c>
ServerLimit 20
# Upper limit on ThreadsPerChild
ThreadLimit 100

# Start with all 20 processes
StartServers 20
MaxClients 2000
MinSpareThreads 2000

# Don't kill child processes in response to load
MaxSpareThreads 2000

# Constant number of child processes, 2000/100 = 20.
ThreadsPerChild 100
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512
# in IHS/bin/envvars for high ThreadsPerChild
</IfModule>


Example 2:
This configuration uses multiple IBM HTTP Server child processes, with a variable capacity of 2000 concurrent connections spread amongst 1-20 child processes.

If the ESI cache is used, it is fragmented or duplicated between the child processes. Usually ESI caches are relatively small, and over the course of many thousands of requests, the reduction in hit rate is not a practical concern.

When a backend application server is misbehaving, each IBM HTTP Server child process must detect the markdown and the ultimately retry -- this information is not shared between child processes.

To mitigate the impact to the WebSphere Application Server Web server plug-in, you can configure IBM HTTP Server to conservatively create new child processes and aggressively kill off old ones. This ensures that the smallest working set of child processes will know about ESI, markdowns, and so on.

A crash of an individual child process affects between 1 and 100 threads only, depending on how busy the child process is.

Note: While MinSpareThreads and MaxSpareThreads do not have to be multiples of ThreadsPerChild, it is convenient to use multiples of ThreadsPerChild to ensure that there is no churn between creating a new child process and immediately killing it.

<IfModule worker.c>
ServerLimit 20
# Upper limit on ThreadsPerChild
ThreadLimit 100

# Start with only 2 processes
StartServers 2
MaxClients 2000

# Don't create a new process until we
# have one process worth free.
MinSpareThreads 100

# Kill a child when we've got an entire extra process
MaxSpareThreads 100

# Constant number of child processes, 2000/100 = 20.
ThreadsPerChild 100
MaxRequestsPerChild 0

# On linux, you may need to set ulimit -s 512
# in IHS/bin/envvars for high ThreadsPerChild

</IfModule>


Example 3:
This configuration uses a single IBM HTTP Server child process with 2000 threads.

Note: This example has been removed from this document because it only academically improves some process-wide data but it exposes the server to numerous shortcomings: lock contention, scalability problems, and crashes in one thread affecting 100% of clients connected to the server. Such an extreme MPM configuration can have significant effects on different Apache HTTP Server modules and should be tested, deployed, and observed with care.

Notes:

• Logging of static files might contribute to load on your IBM HTTP Server system, see the "conditional logging" topic of the IBM HTTP Server information center for further information:
  http://publib.boulder.ibm.com/httpserv/manual70/logs.html
  
  
• IHSDIAG has more detailed performance tuning information:
  http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html