There are two types of memory problems which will throw an OutOfMemoryError: If a Java™ object cannot be allocated or if there is no more memory available for native code allocations (c malloc) or threads. The first case is called Java heap exhaustion and the second, a native memory issue.
Finding the users of native memory can be a lengthy process, as there is no way of knowing which code allocated the native memory. There are no comparable heapdumps for native memory, so we have to use trial and error, going through each of the potential native memory users until we find the one that is causing the problem.
Following are the most common causes of native memory issues in WebSphere Application Server, so you should try them first. They are all trying to reduce the native memory used (the native memory footprint) so both the java heap memory and native memory can fit in the address space for the application server process.
If these steps do not solve your native memory issue, IBM support will guide you through the rest.
The process size for the application server will grow quite large. You may also see OutOfMemoryError’s thrown, malloc errors in SystemOut.log or errors creating threads.
JVMDBG001: malloc failed to allocate …
“Failed to fork OS thread”, or “cannot create anymore threads”.
Resolving the problem
1. Check the maximum heap size. This is the most common cause for native memory problems in 32bit WebSphere Application Server installs. If you have a 64bit install, please bypass this step, the heap size will not be causing the problem.
Since the native memory area is the memory for the process left over from the Java heap, the larger the heap, the smaller the native area. If the maximum heap size is too big, the address space reserved for it will encroach on the native memory area, causing there to not be enough native memory for the app server. If your maximum heap size is 1536m or higher, we will want to lower it to 1280m or even 1024m.
Each 32bit process has an address space of 2 ** 32, (for example., 4,294,967,296) bytes or 4GB. In each process, the operating system reserves just under 2GB of the address space. This is used for operating system libraries, etc. This leaves just over 2GB of address space for a process's code to use (in this case Java). This address space is shared by theJava heap and the native area. When you increase theJava heap size you are decreasing the native area, as it only has what is left over from the heap. It is the max heap setting here, not just how large the heap has expanded to. The heap may have only expanded to 256m, but if the max heap is set to 1792m, then 1792m of address space is reserved and there will only be just over 256m available for native memory use.
NOTE: zSeries machines have 31bit and 64bit versions. When it is not 64bit WebSphere Application Server, it will be 31bit. The Java Virtual Machine (JVM) version shows 32bit, but this is incorrect. With only 31 bits, the address space is half that of 32bit addressing (2**31 == 2GB). This makes the available space for the Java heap and native much less. In fact you have to make special operating system settings if you want a heapsize greater than 800m (mapped_base). If you are seeing a native OutOfMemoryError on Linux on zSeries, you will want to reduce the heap size if it is more than 600m.
2. WebContainer DirectByteBuffer use. The WebContainer may store a large number of buffers in native memory when sending large responses back to the requesters asynchronously. We can reduce the native memory footprint by setting a WebContainer property to send responses in synchronous mode. The downside of synchronous mode is that it will use a thread exclusively during the response..
See the following page for more detail and instructions:
Setting the WebSphere Application Server WebContainer to synchronous mode
3. AIO Native Transport. (WebSphere Application Server 6.1 and higher).
In general, running the WebSphere Application Server TCP Channel in AIO mode gives better performance and scalability than using the TCP Channel in NIO mode. But this faster performance, different timing, and slightly different threading design can increase the native memory footprint. If the application server is short on native memory, sometimes disabling the AIO native library can increase the available address space for native memory.
See the following page for more detail and instructions:
Disabling AIO (Asynchronous Input/Output) native transport in WebSphere Application Server
4. Thread pool issue. This native memory footprint issue can sometimes occur when data is attached to thread pool threads, via ThreadLocal java objects, and these threads are frequently created and destroyed. When a thread is released the memory it was holding onto is dereferenced for garbage collection. This includes heap and native memory. If there are any garbage collection problems or another object is referencing the ThreadLocal objects, then they cannot get garbage collected and therefore the native memory does not get freed either. To fix the problem, use a fixed size thread pool.
See the following page for more detail and instructions to set your thread pool size:
Potential native memory use in WebSphere Application Server Thread Pools
5. Reflection delegating classloaders. When using Java reflection, if the JVM uses a Java bytecode accessor on the class being reflected, it will use native memory. If there is frequent use of Java reflection, there can be a significant amount of native memory use.
See the following page for more detail and instructions on how to limit the native memory use of the Java reflection accessors:
Potential native memory use in reflection delegating classloaders