IC89824: TPC-R unable to access internal datastore causing unexpected res ults after issuing commands and possibly fill filesystem
A fix is available
Closed as fixed if next.
The internal datastore that TPC-R uses can no longer be accessed due to transaction aborts. The internal datastore is no longer accessible therefore TPC-R cannot update pair state information. The issue resides in derby 10.1 and in effect causes the derby.log to grow very large. The derby.log becomes full of NullPointerExceptions This can lead to the file system filling if there is only a small amount of space allocated. Because the internal datastore can no longer be accessed, commands may appear to hang after being issued. The problem exhibits itself as seen in the csmTrace log as the following repeated entry: com.ibm.csm.server.db.DBCmdHandler DBCmdHandler java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openCon tainer(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openCon tainer(Unknown Source) at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown Source) .... com.ibm.csm.server.db.DBCmdHandler DBCmdHandler():setTransactionIsolation com.ibm.csm.server.db.exception.NoDBConnectionException: There is no connection with the db at com.ibm.csm.server.db.DBCmdHandler.setTransactionIsolation(DBCmd Handler.java:149) at com.ibm.csm.server.db.DBCmdHandler.<init>(DBCmdHandler.java:116) Even though the internal datastore cannot be accessed, this does not affect the running process of TPC-R due to the fact that connection and session states are cached. Commands will change states on the hardware, but state changes may not be reflected until TPC-R has been restarted and communication restored to the internal datastore.
The following steps can lessen the amount of text sent to the derby.log file which may prolong the time span before the filesystem will fill up, and will force the derby.log file to be truncated upon restart of the TPC-R WAS process. 1. Modify current <INSTALL_HOME>/eWAS/derby/derby.properties file to comment out these 2 lines (so default value takes over): derby.infolog.append=true derby.stream.error.logSeverityLevel=0 2. Restart the TPC-R service to re-establish communication to the internal datastore. - The append=true setting causes current derby.log file content got appended after Derby restart (so can grow very large over time). - The logSeverityLevel=0 setting creates very detailed traces in the log which makes the file very large in size. For the permanent fix contact support to obtain the necessary patch, or upgrade to TPC 5.1+
A highly intermittent timing issue in derby 10.1 https://issues.apache.org/jira/browse/DERBY-2141 causes the database server to shut down. The TPC-R server continues to attempt inserts and queries against the database. Each database access causes a NullPointerException to get logged in the derby.log. This in turn causes the derby.log to grow very large in a short period of time.
Refer to "Local Fix" section above. Permanent solution is to apply updated derby.jar as described, and workaround is to configure the log to truncate upon restart of the TPC-R WAS process.
Reported component name
TPC FOR REPLICA
Reported component ID
Last modified date
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Applicable component levels
More support for:
Tivoli Storage Productivity Center for Replication
Software version: 420
Reference #: IC89824
Modified date: 2013-09-30