Web module or application server stops processing requests

• If database connections are being assigned to a request but are not being released when the requests finish processing, ensure that the application code performs a close() on any opened Connection object within a finally{} block.
• If there is a steady increase in servlet engine threads in use, review application synchronized code blocks for possible deadlock conditions.
• If there is a steady increase in a JVM heap size, review application code for memory leak opportunities, such as static (class-level) collections, that can cause objects to never get garbage-collected.

You can also use the Tivoli performance viewer to detect memory leak problems.

• There is a memory leak in application code that you must address. To pinpoint the cause of a memory leak, enable the RunHProf property on the Java Virtual Machine page of the administrative console. server_name is the name of the problem application server. Once you enable the RunHProf property, you must:
  • Set the HProf Arguments field to a value similar to depth=20,file=heapdmp.txt. This value shows exception stacks to a maximum of 20 levels, and saves the heapdump output to the app_server_root/bin/heapdmp.txt file.
  • Save the settings.
  • Stop and restart the application server.
  • If possible, reenact the scenario or access the resource that caused the application server's process to spontaneously close, or its web modules to stop responding to new requests. Then stop the application server. If you cannot reenact the scenario or access the resource, wait until the problem reoccurs, and then stop the application server.
  • Examine the file into which the heap dump was saved. For example, examine the app_server_root/bin/heapdmp.txt file:
    • Search for the string, SITES BEGIN. This finds the location of a list of Java objects in memory, which shows the amount of memory allocated to the objects.
    • The list of Java objects occurs each time there was a memory allocation in the JVM. There is a record of what type of object the memory instantiated and an identifier of a trace stack, listed elsewhere in the dump, that shows the Java method that made the allocation.
    • The list of Java object is in descending order by number of bytes allocated. Depending on the nature of the leak, the problem class should show up near the beginning of the list, but this is not always the case. Look throughout the list for large amounts of memory or frequent instances of the same class being instantiated. In the latter case, use the ID in the trace stack column to identify allocations occurring repeatedly in the same class and method.
    • Examine the source code indicated in the related trace stacks for the possibility of memory leaks.
• The JVM is using the maximum heap size that it is allowed to use. In this situation, you should increase the maximum heap size setting for application server if you have enough storage available to do so.
• The server runtime is experiencing a problem. If you determine that there is a problem with the server runtime, make sure that you have applied all of the service updates for the product. If, after you apply all of the service updates, the problem still exists, contact IBM® Support.

The JVM creates a thread dump whenever an application server process spontaneously closes. You can also force an application to create a thread dump. After a dump is created, you can check the dump for clues as to why new requests are not being processed.

To force a thread dump:

1. Using the wsadmin command prompt, get a handle to the problem application server:

   wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server_name,*] 

   where server_name is the name of your server.

2. Generate the thread dump:

   wsadmin>$AdminControl invoke $jvm dumpThreads

   Note: The dumpThreads command creates other types of dump files depending on the -Xdumps settings. Dump output varies depending on the platform and might include system core files, heap, and snap dumps.
3. Look for an output file in the profile_root/logs directory with a name like javacore.jobnum.jobuser.jobname.timestamp.txt.
4. Look for an output file, in the installation root directory for the product, with a name like javacore.date.time.id.txt.

After the application creates the dump, you can check for the following clues:

• Error or exception information strings at the beginning of the file. These strings indicate the thread that caused the application server process to spontaneously close. These strings are not be present if you forced the dump.
• Look for an excessive current heap size. The thread dump shows information on the current Java heap size, and the minimum and maximum heap size settings.
• Look at the snapshot of each thread in the process. The thread dump contains a snapshot of each thread in the process, starting in the section labeled Full thread dump.
  • Look for threads with a description that contains state:R. Such threads are active and running when the dump is forced, or the process exited.
  • Look for multiple threads in the same Java application code source location. Multiple threads from the same location might indicate a deadlock condition (multiple threads waiting on a monitor) or an infinite loop, and help identify the application code with the problem.
• Look at the snapshot of each thread in the process. The thread dump contains a snapshot of each thread in the process, starting in the section labeled Thread Information.
  • Look for threads that are waiting on locks held by other threads.
  • Look for multiple threads in the same Java application code source location. Multiple threads from the same location might indicate a deadlock condition (multiple threads waiting on a monitor) or an infinite loop, and help identify the application code with the problem.

    It is normal for certain components in the product runtime to have certain types of threads in the same Java code source location. These components include the web container, the EJB container and the ORB thread pool.