IC89824: TPC-R unable to access internal datastore causing unexpected res ults after issuing commands and possibly fill filesystem

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as fixed if next.

Error description

  • The internal datastore that TPC-R uses can no longer be
    accessed due to transaction aborts. The internal datastore
    is no longer accessible therefore TPC-R cannot update pair
    state information. The issue resides
    in derby 10.1 and in effect causes the derby.log to grow very
    large. The derby.log becomes full of NullPointerExceptions
    This can lead to the file system filling if there is only a
    small amount of space allocated.
    
    Because the internal datastore can no longer be accessed,
    commands may appear to hang after being issued.
    
    The problem exhibits itself as seen in the csmTrace log as the
    following repeated entry:
    
    
    
    
    
    
    
    com.ibm.csm.server.db.DBCmdHandler DBCmdHandler
    java.lang.NullPointerException
    at
    org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openCon
    tainer(Unknown Source)
    at
    org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openCon
    tainer(Unknown Source)
    at
    org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown
    Source)
    ....
    
    
    com.ibm.csm.server.db.DBCmdHandler
    DBCmdHandler():setTransactionIsolation
    com.ibm.csm.server.db.exception.NoDBConnectionException: There
    is no connection with the db
    at
    com.ibm.csm.server.db.DBCmdHandler.setTransactionIsolation(DBCmd
    Handler.java:149)
    at
    com.ibm.csm.server.db.DBCmdHandler.<init>(DBCmdHandler.java:116)
    
    Even though the internal datastore cannot be accessed, this does
    not affect the running process of TPC-R due to the fact that
    connection and session states are cached. Commands will change
    states on the hardware, but state changes may not be reflected
    until TPC-R has been restarted and communication restored to the
    internal datastore.
    

Local fix

  • The following steps can lessen the amount of text sent to the
     derby.log file which may prolong the time span before the
    filesystem will fill up, and will force the derby.log file to be
    truncated upon restart of the TPC-R WAS process.
    
    
    
    
    
    
    1. Modify current <INSTALL_HOME>/eWAS/derby/derby.properties
    file to comment out these 2 lines (so default value takes over):
     derby.infolog.append=true
     derby.stream.error.logSeverityLevel=0
    
    2. Restart the TPC-R service to re-establish communication to
    the internal datastore.
    
    - The append=true setting causes current derby.log file content
    got appended after Derby restart (so can grow very large over
    time).
    - The logSeverityLevel=0 setting creates very detailed traces in
    the log which makes the file very large in size.
    
    
    
    For the permanent fix contact support to obtain the necessary
    patch, or upgrade to TPC 5.1+
    

Problem summary

  • A highly intermittent timing issue in derby 10.1
    https://issues.apache.org/jira/browse/DERBY-2141
    causes the database server to shut down. The TPC-R
    server continues to attempt inserts and queries
    against the database. Each database access causes
    a NullPointerException to get logged in the
    derby.log. This in turn
    causes the derby.log to grow very large
    in a short period of time.
    

Problem conclusion

Temporary fix

  • Refer to "Local Fix" section above.  Permanent solution is to
    apply updated derby.jar as described, and workaround is to
    configure the log to truncate upon restart of the TPC-R WAS
    process.
    

Comments

APAR Information

  • APAR number

    IC89824

  • Reported component name

    TPC FOR REPLICA

  • Reported component ID

    5608TRM00

  • Reported release

    420

  • Status

    CLOSED FIN

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2013-01-25

  • Closed date

    2013-02-13

  • Last modified date

    2013-09-30

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

  • R420 PSN

       UP



Rate this page:

(0 users)Average rating

Document information


More support for:

Tivoli Storage Productivity Center for Replication

Software version:

420

Reference #:

IC89824

Modified date:

2013-09-30

Translate my page

Machine Translation

Content navigation