Troubleshooting
Problem
An OutOfMemory exception is returned when running a Java routine (stored procedure or user defined function) may be returned along with a javacore and java heap dump.
Symptom
Scenario #1: Java heap full
db2fmp process abnormally terminates due to an OutOfMemory exception reported by the Java Virtual Machine (JVM). A JVM heapdump file may be produced.
The db2diag.log may contain the message
2010-01-01-12.00.25.075958-240 I24394A370 LEVEL: Warning
PID : 834123 TID : 1 PROC : db2fmp (Java) 0
INSTANCE: db2inst1 NODE : 000
EDUID : 1 EDUNAME: db2fmp (Java) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadListener, probe:300
MESSAGE : Java heap is nearly full; no thread created
Scenario #2: UNIX/Linux operating system errors
2010-01-01-09.41.54.294287-240 E7048A1234 LEVEL: Error (OS)
PID : 1487054 TID : 3856 PROC : db2fmp (Java) 0
INSTANCE: db2inst1 NODE : 000
EDUID : 3856 EDUNAME: db2fmp (Java) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition,
probe:100
CALLED : OS, -, malloc
OSERR : ENOMEM (12) "There is not enough memory available now."
MESSAGE : Private memory and/or virtual address space exhausted, or data ulimit exceeded
OR
2010-01-01-09.41.54.294287-240 E7048A1234 LEVEL: Severe (OS)
PID : 5786 TID : 46912558215808PROC : db2fmpr
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloCreateAppThread, probe:100
CALLED : OS, -, pthread_create OSERR: ENOMEM (12)
Cause
Scenario #1 Common causes:
- Coding error in the application in which memory is continually allocated but not released or the routine relies on the JVM's garbage collector to clean up objects instead of manually releasing them. One example is the use of finalizer() objects.
- Finalizer() objects used in the legacy Type 2 JDBC driver (db2java.zip) preventing memory from being released quickly enough. db2java.zip has been deprecated. Use the new JDBC driver (db2jcc.jar) instead
- The Java Heap is insufficiently sized for the workload of the Java routines.
- Tune by increasing the DBM CFG parameter JAVA_HEAP_SZ and or setting the registry variable DB2_MAX_THREADS_PER_FMP (IZ08425) to limit the number of threadsafe Java routines sharing one Java Heap inside each db2fmp process.
- Change the definitions of routines which consume the most Java heap from THREADSAFE to NOT THREADSAFE.
Scenario #2 Common causes:
- Applies to all non-SQL routines such as Java, C and Cobol.
- Memory on the system has been exhausted.
- For "There is not enough memory available now." The ulimit for the data section of a process may be set too low. Resolve by setting ulimit data=unlimited for the id used to start the DB2 instance. Note that depending on your system's configuration this may be an id other than the DB2 instance id such as root. If increasing the soft ulimit (ulimit -aS) does not work, ensure that the hard ulimit is larger than the soft ulimit (ulimit -aH). With older releases of DB2, the db2fmp process's ulimit is inherited from the id used to start DB2. Thus it is suggested changing ulimit for BOTH DB2 instance user id and fenced user id.
- For "pthread_create". Check to ensure ulimit for the stack section does not exceed the size of physical memory + swap or that it is not set to an abnormally large number such as 4 GB. Resolve by lowering the value to 8 MB or the operating system default for the DB2 instance owner.
Resolving The Problem
To provide more background on db2fmp's, DB2 allocates a shared memory (DB2_FMP_COMM_HEAPSZ) for inter-process communications between the db2agent process and db2fmp process. All of the memory allocated by the db2fmp via a third party ty outside of DB2's control.
In the case of Java routines it would be the Solaris or IBM JVM, and the C runtime in the case of C routines. The probability of a memory leak originating from DB2 is extremely rare since the memory would either be consumed by DB2_FMP_COMM_HEAPSZ or the JDBC driver used to execute Java routines. If there was a leak with DB2_FMP_COMM_HEAPSZ there will be messages in the db2diag.log about increasing this parameter in the db2diag.log even though it is set to a very high setting such as 512 MB.
Monitoring Java heap usage must be done via third party tools since this memory is not managed by DB2. The Solaris or IBM JDK Diagnostics Guides may contain information on how to do this.
The Java heap is totally independent of DB2_FMP_COMM_HEAPSZ. Please see the link How to calculate the number of db2fmp processes allowed using the DB2_FMP_COMM_HEAPSZ variable for information on how to set this parameter.
For coding errors please review the Java routine with the Java routine developer. Download the IBM Java HeapAnalyzer and load the heapdump to determine which Java objects are consuming the largest amount of memory.
For Java routines using the deprecated legacy JDBC driver (db2java.zip) try migrating to the IBM Data Server Driver for JDBC and SQLJ (db2jcc.jar). Please note the new driver is fully compliant with JDBC specifications and thus the routine and application should be thoroughly tested to ensure expected results. Some of these changes are documented in the linksJDBC differences between versions of the IBM Data Server Driver for JDBC and SQLJ and Common application issues when migrating from the legacy JDBC driver.
Note: On DB2 v8.x by default the legacy is used to run Java routines, whereas on DB2 v9.x IBM Data Server Driver for JDBC and SQLJ. On v8.x systems the following registry key must be set to use the Universal JDBC driver: db2set DB2_USE_DB2JCCT2_JROUTINE=on -g
For workload issues try increasing the maximum size of the Java Virtual Machine heap size using:
db2 "update dbm cfg using java_heap_sz <new value>"
and/or tuning the number of threads via the DB2 registry variable DB2_MAX_THREADS_PER_FMP which determines the number of concurrent threadsafe Java routines which can be run inside inside each db2fmp process. This registry variable was introduced via IZ08425.
The new java_heap_sz takes effect when a db2fmp process is created when the DB2 instance is restarted or when db2fmp process is created.
Related Information
JDBC differences between versions of the IBM Data Serve
java_heap_sz - Maximum Java interpreter heap size confi
DB2_USE_DB2JCCT2_JROUTINE registry variable
Common application issues when migrating from the legac
Sun Solaris JDK Diagnostics Guide
How to calculate the number of db2fmp processes allowed
Was this topic helpful?
Document Information
Modified date:
23 June 2018
UID
swg21412612