Thread(s) hanging indefinitely on SocketInputStream.socketRead0 from DB2 JCC driver

Technote (FAQ)


Question

When using DB2 Version 9.5 or 9.7 with WebSphere Commerce, how can the DB2 JCC driver's call to SocketInputStream.socketRead0 be timed out to avoid hanging indefinitely?

In the SystemOut.log, messages, hung thread messages appear, such as:
[1/2/12 1:23:45:678 EDT] 0000001c ThreadMonitor W WSVR0605W: Thread "WebContainer : 15" (00000045) has been active for 722010 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.ibm.db2.jcc.t4.z.b(z.java:199)
at com.ibm.db2.jcc.t4.z.c(z.java:268)
at com.ibm.db2.jcc.t4.z.c(z.java:381)
at com.ibm.db2.jcc.t4.z.v(z.java:1149)
at com.ibm.db2.jcc.t4.cb.a(cb.java:40)
at com.ibm.db2.jcc.t4.q.a(q.java:32)
at com.ibm.db2.jcc.t4.rb.i(rb.java:135)
at com.ibm.db2.jcc.am.mn.ib(mn.java:2055)
at com.ibm.db2.jcc.am.nn.rc(nn.java:3220)
at com.ibm.db2.jcc.am.nn.b(nn.java:4002)
at com.ibm.db2.jcc.am.nn.ec(nn.java:728)
at com.ibm.db2.jcc.am.nn.executeQuery(nn.java:698)
There exists no deadlock or timeout recorded in the logs, even when there are lock timeout (LOCKTIMEOUT) and deadlock check time (DLCHKTIME) settings defined that are greater than 0.

Cause

The main cause of the hang scenario are most commonly associated with a dropped connection between the database and the application server. This typically is caused by a firewall or other network issues. However, transaction timeout values defined in the WebSphere Application Server Administration Console do not time out the socketRead0 operation. Therefore, threads that are hung in that condition would not time out as desired.

Answer

The cause of the dropped connection should be investigated and resolved. Investigating the cause can take time, requiring a need for immediate relief, particularly on a production environment.

The blockingReadConnectionTimeout custom property in the data source can time out the SocketInputStream.socketRead0 operation.

To add the blockingReadConnectionTimeout property:

  1. Open the WebSphere Application Server Administration Console.
  2. Expand Resources.
  3. Expand JDBC.
  4. Click Data sources.
  5. Click the "WebSphere Commerce DB2 Datasource demo" datasource in the main window.
  6. In the Additional Properties section, click Custom Properties.
  7. Click New.
  8. In the Name field, type blockingReadConnectionTimeout
  9. In the Value field, specify the time in seconds in which you want to time out after, for example: 900
  10. Click Apply then OK.
  11. Save your changes
  12. Restart the WebSphere Application Server.

The amount of time in seconds before a connection socket read times out. This property applies only to IBM Data Server Driver for JDBC and SQLJ type 4 connectivity, and affects all requests that are sent to the data source after a connection is successfully established. The default value is 0. A value of 0 means that there is no time out.

When the blockingReadConnectionTimeout property is enabled and a hung thread condition occurs, the relevant lines in the logs would appear:

[1/2/12 1:23:45:678 EDT] 0000001c ThreadMonitor W WSVR0605W: Thread "WebContainer : 15" (00000045) has been active for 722010 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
...
[1/2/12 1:25:45:678 EDT] 00000045 ConnectionEve W J2CA0206W: A connection error occurred. To help determine the problem, enable the Diagnose Connection Usage option on the Connection Factory or Data Source.
...
[1/2/12 1:25:46:000 EDT] 00000045 ConnectionEve A J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc/WebSphere Commerce DB2 DataSource demo. The exception is: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][4.12.56] A communication error occurred during operations on the connection's underlying socket, socket input stream, or socket output stream. Error location: Reply.fill() - socketInputStream.read (-1). Message: Read timed out. ERRORCODE=-4499, SQLSTATE=08001:java.net.SocketTimeoutException: Read timed out
...
[1/2/12 1:25:47:000 EDT] 00000045 RemoteExcepti E CNTR0019E: EJB threw an unexpected (non-declared) exception during invocation of method "findByProductFamilyIdAndColumnName". Exception data: com.ibm.ejs.persistence.EJSPersistenceException: find failed:; nested exception is:
com.ibm.ejs.persistence.EJSPersistenceException: getPStmt failed; nested exception is:
com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][10335][10366][4.12.56] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003
....
[1/2/12 1:25:48:000 EDT] 00000045 SystemOut O javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: javax.transaction.TransactionRolledbackException: ; nested exception is:
com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][10335][10366][4.12.56] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003 vmcid: 0x0 minor code: 0 completed: No
...
[1/2/12 1:25:49:000 EDT] 00000045 ThreadMonitor W WSVR0606W: Thread "WebContainer : 15" (00000045) was previously reported to be hung but has completed. It was active for approximately 904351 milliseconds. There is/are 0 thread(s) in total in the server that still may be hung.

The above logged lines are normal and are to be expected when the blockingReadConnectionTimeout occurs and the appropriate action is taken to roll back.

Additionally, with the StaleConnectionException being introduced by the blockingReadConnectionTimeout, it would be important to follow the article regarding: How to configure WebSphere data source parameters when a firewall is present.


Related information

How to configure WebSphere data source parameters when
Common IBM Data Server Driver for JDBC and SQLJ
A simplified Chinese translation is available

Cross reference information
Segment Product Component Platform Version Edition
Commerce WebSphere Commerce Professional Edition Database AIX, Linux, Solaris, Windows 7.0
Commerce WebSphere Commerce - Express Database Linux, Windows 7.0

Rate this page:

(0 users)Average rating

Document information


More support for:

WebSphere Commerce Enterprise
Database

Software version:

7.0

Operating system(s):

AIX, Linux, Solaris, Windows

Reference #:

1590014

Modified date:

2012-04-23

Translate my page

Machine Translation

Content navigation