IC89824: TPC-R unable to access internal datastore causing unexpected res ults after issuing commands and possibly fill filesystem

Fix Pack 5.1.1.1 (April 2013) for Tivoli Storage Productivity Center

APAR status

• Closed as fixed if next.

Error description

• The internal datastore that TPC-R uses can no longer be
  accessed due to transaction aborts. The internal datastore
  is no longer accessible therefore TPC-R cannot update pair
  state information. The issue resides
  in derby 10.1 and in effect causes the derby.log to grow very
  large. The derby.log becomes full of NullPointerExceptions
  This can lead to the file system filling if there is only a
  small amount of space allocated.
  
  Because the internal datastore can no longer be accessed,
  commands may appear to hang after being issued.
  
  The problem exhibits itself as seen in the csmTrace log as the
  following repeated entry:
  
  
  
  
  
  
  
  com.ibm.csm.server.db.DBCmdHandler DBCmdHandler
  java.lang.NullPointerException
  at
  org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openCon
  tainer(Unknown Source)
  at
  org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openCon
  tainer(Unknown Source)
  at
  org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown
  Source)
  ....
  
  
  com.ibm.csm.server.db.DBCmdHandler
  DBCmdHandler():setTransactionIsolation
  com.ibm.csm.server.db.exception.NoDBConnectionException: There
  is no connection with the db
  at
  com.ibm.csm.server.db.DBCmdHandler.setTransactionIsolation(DBCmd
  Handler.java:149)
  at
  com.ibm.csm.server.db.DBCmdHandler.<init>(DBCmdHandler.java:116)
  
  Even though the internal datastore cannot be accessed, this does
  not affect the running process of TPC-R due to the fact that
  connection and session states are cached. Commands will change
  states on the hardware, but state changes may not be reflected
  until TPC-R has been restarted and communication restored to the
  internal datastore.
  

Local fix

• In order to prevent the file system from filling up due
  to the <INSTALL_HOME>/eWAS/derby/derby.log file the following
  modifications need to be made
  
  1. Modify current <INSTALL_HOME>/eWAS/derby/derby.properties
  file to comment out these 2 lines (so default value takes over):
   derby.infolog.append=true
   derby.stream.error.logSeverityLevel=0
  
  2. Restart the TPC-R service to re-establish communication to
  the internal datastore.
  
  
  
  Contact support for the necessary patch, or upgrade to TPC
  5.1+
  

Problem summary

• A highly intermittent timing issue in derby 10.1
  https://issues.apache.org/jira/browse/DERBY-2141
  causes the database server to shut down. The TPC-R
  server continues to attempt inserts and queries
  against the database. Each database access causes
  a NullPointerException to get logged in the
  derby.log. This in turn
  causes the derby.log to grow very large
  in a short period of time.
  

Problem conclusion

Temporary fix

• In order to prevent the file system from filling up due
  
  to the <INSTALL_HOME>/eWAS/derby/derby.log file the following
  modifications need to be made:
  
  1) Modify current
  <INSTALL_HOME>/eWAS/derby/derby.properties
  file to comment out these 2 lines (so default value takes over):
   derby.infolog.append=true
   derby.stream.error.logSeverityLevel=0
  
  2) Delete the <INSTALL_HOME>/eWAS/derby/derby.log
  3) Restart the TPC-R server
  
  Check the Derby database consistancy to ensure there is no
  database corruption causing the NPE :
  > http://wiki.apache.org/db-derby/DatabaseConsistencyCheck
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

• R420 PSN

     UP