Troubleshooting
Problem
System gets frozen and error messages "Maximum number of connections has been reached" are seen in WebSphere FFDC logs when running an IBM TRIRIGA server.
Symptom
WebSphere FFDC logs (<WAS_install_root>/profiles/<profile>/logs/ffdc) report the following error:
[MM/DD/YY HH:MM:SS:mmm TTT] FFDC Exception:com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException SourceId:Max connections reached ProbeId:869 Reporter:com.ibm.ejs.j2c.PoolManager@98575e96
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1782)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3872)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3092)
Maximum number of connections has been reached, and the connection request has been waiting longer than:ConnectionWaitTime. Two possible solutions : increase the max number of connections, or increase the:ConnectionWaitTime.:
Maximum Connections = :100
Current number of connections = :100
Connection Wait Timout = :180
Resolving The Problem
Your IBM WebSphere server/JVM (IBM TRIRIGA server) has reached its JDBC Data Source Maximum Connection limit, and system will stop responding to new processes requesting available connections to be used.
The error is printed and available in the WebSphere FFDC logs. For more information on the WebSphere FFDC logs, review this IBM WebSphere technote: Meaning and use of FFDC files
See that the JDBC Data Source Connection Pool's connections are set by server(WebSphere JVMs/Servers). If the IBM TRIRIGA Best Practices for System Performance has been followed, all configuration necessary is in place.
On page 18, under "3.1.3 Connection Pool" it reads:
"The JDBC data source connection pool properties should be configured to allow for enough connections required by your system. These values are dependent on the number of concurrent users expected, in addition to the number of transactions expected.
The following options and values can be configured for connection pools:
Minimum connections: Specifies the minimum number of connections to maintain in the pool.
Typical default: 1
Recommended: 10
Maximum connections: Specifies the maximum number of connection to maintain in the pool.
Typical default: 10
Recommended: 100
A setting of 100 maximum connections is common for a JVM that is not expected to exceed 500 concurrent users. For example if you have two JVMs both with 100 maximum connections then in theory you can support up to 1,000 concurrent users".
You need to adjust your Data Source Connection pool properties "Maximum connections" value for better tuning. As you have run out of connections, then increase this Maximum Connection value. The Performance Best Practice guide says to "start" with 100 number, but to "tune" it to customer's own environment. You need to set the maximum size to a number higher than 100 as part of tuning your own environment.
The connections are handled by the Application Server's Connection Manager (WebSphere). The INACTIVE database connections are handled by Application Server Connection Manager, and it makes the decision of keeping them up for being used for next process requiring a connection, or shutting it down. See this is handled by JVM. WebSphere database connection manager will try to use the INACTIVE sessions open for next processes, before opening new ones.
The issue here is about not enough database connections (>100) for the JVM to support its processes. This needs to be tuned based on customer's business usage of the system. The solution of this issue is setting Maximum connections to a higher value than the initial start one recommended on the IBM TRIRIGA Best Practices guides.
The way Database Connections are managed and their state are controlled by WebSphere Database Connection manager, and this is really dynamic process and can interface with Java SDK Garbage Collection (GC) process, meaning if a session is marked for deletion, after the time-out period, GC will be getting rid of that area and interfacing with the connection manager to close that database session. WebSphere database connection manager may keep some connections open based on system usage, for quickly responding for requests. This is handled out of IBM TRIRIGA software layer realm, happening on the Application Server layer.
It is really critical that you properly set Java SDK Garbage Collection (GC) based on the IBM TRIRIGA Best Practices recommendations as a start point for system tuning. Depending on specific business usage, GC may need further adjustments if confirmed by your WebSphere Administrator & support team. See the comments & information about GC set up & impact on the IBM TRIRIGA Best Practices guide:
(a) 3.1.1 Initial / maximum heap size values - page 17 (considerations about JVM heap size memory and GC runs)
(b) 3.1.2 JVM arguments - page 17 (-Xdisableexplicitgc)
(c) 11.1.4 Displaying garbage collection statistics on the server - page 49 (instructions on how to trace & analyze GC runs)
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21980790