IBM Support

PI27033: RDZ NOT ACCEPTING LOGONS AND/OR HIGH CPU AND/OR ABEND878-10 FOLLOWED BY ABEND40D CAN BE OBSERVED.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Rational Developer for System z not accepting logons and/or high
    CPU and/or ABEND878-10 followed by ABEND40D can be observed.
    
    When some abend occurs in lock routine, FFS lock condition
    handler is kicked.  But Another abend occurs in FFS lock
    condition handler. After all, the FFS lock condition handler was
    kicked forever (i.e. entering loop ).
    
    When standard logging is active
    ABSTRACT:
    RDz not accepting logons and/or high CPU and/or ABEND878-10
    followed by ABEND40D can be observed.
    
    DESCRIPTION:
    RDz not accepting logons and/or high CPU and/or ABEND878-10
    followed by ABEND40D can be observed.
    
    When some abend occurs in lock routine, FFS lock condition
    handler is kicked.  But Another abend occurs in FFS lock
    condition handler. After all, the FFS lock condition handler was
    kicked forever (i.e. entering loop ).
    
    When standard logging is active
    _RSE_JAVAOPTS="$_RSE_JAVAOPTS テ  Denable.standard.log=true"
    The system can easily run out of space on the filesystem that
    contains the server log directory /var/rdz/logs/server.
    
    This can be observed by
    EDC5133I No space left on device. (errno2=0xEF01604E)
    EDC5124I Too many open files. (errno2=0x050B0146)
    
    
       3022    2014.09.09 07:30:57:889 EDT PID:50398810
    THREAD:23BDF00000000017 TCB:009C73E8 USER:STCRDZ ERROR: class
    com.ibm.etools.zos.server.LogHandler$ServerStdoutHandler:java.io
    .IOException: EDC5133I No space left on device.
    (errno2=0xEF01604E)
       3023    java.io.IOException: EDC5133I No space left on
    device. (errno2=0xEF01604E)
       3024        at java.io.RandomAccessFile.writeBytes(Native
    Method)
       3025        at
    java.io.RandomAccessFile.write(RandomAccessFile.java:488)
       3026        at
    com.ibm.etools.zos.server.LogHandler.writeServerStandardLog(LogH
    andler.java:738)
       3027        at
    com.ibm.etools.zos.server.LogHandler$ServerStdoutHandler.run(Log
    Handler.java:691)
       3028        at java.lang.Thread.run(Thread.java:781)
       3029    2014.09.09 07:31:25:113 EDT PID:50398810
    THREAD:23BDF00000000017 TCB:009C73E8 USER:STCRDZ ERROR: class
    com.ibm.etools.zos.server.LogHandler$ServerStdoutHandler:java.io
    .FileNotFoundException: /var/rdz/logs/server/stdout.3.log
    (EDC5124I Too
       3030    java.io.FileNotFoundException:
    /var/rdz/logs/server/stdout.3.log (EDC5124I Too many open files.
    (errno2=0x050B0146))
       3031        at java.io.RandomAccessFile.open(Native Method)
       3032        at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
       3033        at
    com.ibm.etools.zos.server.LogHandler.writeServerStandardLog(LogH
    andler.java:732)
       3034        at
    com.ibm.etools.zos.server.LogHandler$ServerSt2014.09.09
    07:32:48:808 EDT PID:50398810
    
    
    IOEZ00312I Dynamic growth of aggregate ITSX0$P.ZFS.RDZ.VAR.RD01
    in progress, (by user STCRDZ).
    STC35990  +FEK114E accept() failed. reason=(EDC5124I Too many
    open files. (errno2=0x050B0146))
    
    This problem can also cause lead to multiple ABEND878-10 abends
    that end with an ABEND40D (recursive abend).
    
    Also high CPU usage for RSE servers and the RSE daemon can be
    observed.
    Java stack frames during Hi CPU:
    method: String com/ibm/etools/zos/server/CoreJNI.receive(int,
    int, int)  (Native Method)
    method: void
    com/ibm/etools/zos/server/LogHandler$ServerStdoutHandler.run()
    source: LogHandler.java:692
    method: void java/lang/Thread.run()  source: Thread.java:795
    The system can easily run out of space on the filesystem that
    contains the server log directory /var/rdz/logs/server.
    
    This can be observed by
    EDC5133I No space left on device. (errno2=0xEF01604E)
    EDC5124I Too many open files. (errno2=0x050B0146)
    
    
       3022    2014.09.09 07:30:57:889 EDT PID:50398810
    THREAD:23BDF00000000017 TCB:009C73E8 USER:STCRDZ ERROR: class
    com.ibm.etools.zos.server.LogHandler$ServerStdoutHandler:java.io
    .IOException: EDC5133I No space left on device.
    (errno2=0xEF01604E)
       3023    java.io.IOException: EDC5133I No space left on
    device. (errno2=0xEF01604E)
       3024        at java.io.RandomAccessFile.writeBytes(Native
    Method)
       3025        at
    java.io.RandomAccessFile.write(RandomAccessFile.java:488)
       3026        at
    com.ibm.etools.zos.server.LogHandler.writeServerStandardLog(LogH
    andler.java:738)
       3027        at
    com.ibm.etools.zos.server.LogHandler$ServerStdoutHandler.run(Log
    Handler.java:691)
       3028        at java.lang.Thread.run(Thread.java:781)
       3029    2014.09.09 07:31:25:113 EDT PID:50398810
    THREAD:23BDF00000000017 TCB:009C73E8 USER:STCRDZ ERROR: class
    com.ibm.etools.zos.server.LogHandler$ServerStdoutHandler:java.io
    .FileNotFoundException: /var/rdz/logs/server/stdout.3.log
    (EDC5124I Too
       3030    java.io.FileNotFoundException:
    /var/rdz/logs/server/stdout.3.log (EDC5124I Too many open files.
    (errno2=0x050B0146))
       3031        at java.io.RandomAccessFile.open(Native Method)
       3032        at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
       3033        at
    com.ibm.etools.zos.server.LogHandler.writeServerStandardLog(LogH
    andler.java:732)
       3034        at
    com.ibm.etools.zos.server.LogHandler$ServerSt2014.09.09
    07:32:48:808 EDT PID:50398810
    
    
    IOEZ00312I Dynamic growth of aggregate ITSX0$P.ZFS.RDZ.VAR.RD01
    in progress, (by user STCRDZ).
    STC35990  +FEK114E accept() failed. reason=(EDC5124I Too many
    open files. (errno2=0x050B0146))
    
    This problem can also cause lead to multiple ABEND878-10 abends
    that end with an ABEND40D (recursive abend).
    
    Also high CPU usage for RSE servers and the RSE daemon can be
    observed.
    Java stack frames during Hi CPU:
    method: String com/ibm/etools/zos/server/CoreJNI.receive(int,
    int, int)  (Native Method)
    method: void
    com/ibm/etools/zos/server/LogHandler$ServerStdoutHandler.run()
    source: LogHandler.java:692
    method: void java/lang/Thread.run()  source: Thread.java:795
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED: 01.All RDz users                             *
    ****************************************************************
    * PROBLEM DESCRIPTION: 01.The condition handler registered in  *
    *                         the resource serialization logic     *
    *                         sets the wrong result value.         *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    01.Because a condition handler sets the wrong result value of
       20, it is called repeatedly when an exception occurs while
       serializing a resource.
    

Problem conclusion

  • 01.The condition handler is updated to set the correct result
       value of 10 to be called just once when an exception occurs
       while serializing a resource.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PI27033

  • Reported component name

    RD/Z HOST

  • Reported component ID

    5724T0723

  • Reported release

    910

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2014-10-06

  • Closed date

    2014-12-05

  • Last modified date

    2014-12-09

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Modules/Macros

  • FEKFENVR FEKFMAIN FEKFMAI6 FEKFMINE FEKFOMVS
    FEKFTSO  FEKFZOS
    

Fix information

  • Fixed component name

    RD/Z HOST

  • Fixed component ID

    5724T0723

Applicable component levels

  • R910 PSY UI23650

       UP14/12/09 I 1000

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: Rational Developer for System z

Software version: 9.1

Reference #: PI27033

Modified date: 09 December 2014