IBM Support

PI32169: ASYNCBEANS LOGS ERROR ASYN0066E AND CAN LEAD TO A SLOW MEMORY LEAK DUE TO TIMING ISSUES RELATED TO WORK COMPLETION

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • When using the WebSphere asynchbeans WorkManager with the Object
    pool service enabled, the following error may occur in the
    logs.
    
    [2/13/15 2:09:22:934 GMT] 0000004a WorkTimeoutLi E
    ASYN0066E: The method <null> threw an Throwable: {1}.
    
    Also a memory leak related to the
    java/util/concurrent/ConcurrentLinkedQueue$Node holding
    com/ibm/ws/asynchbeans/WorkTimeoutListener objects is possible.
    The memory analyzer tools may report leak suspects like below.
    
    A Linked List Data Structure Detected.
    212,757,328 bytes (30.13 %) of Java heap is used by 13,297,159
    instances of java/util/concurrent/ConcurrentLinkedQueue$Node
    Contains the following object:
    
    - 13,297,158 instances of
    com/ibm/ws/asynchbeans/WorkTimeoutListener holding 425,509,056
    bytes
    - Contained under array of java/util/HashMap$Entry holding
    591,810,208 bytes at 0x97390bc8
    
    Tree :
    com.ibm.ws.objectpool.ObjectPoolImpl @ 0x93b2ff88
    |           24 |   566,417,296 |     75.00%
    '- com.ibm.ws.objectpool.FastObjectPoolImpl @ 0x949ca290
    |           32 |   566,417,272 |     75.00%
    |- java.util.concurrent.ConcurrentLinkedQueue @ 0x949ca2c0
    |           16 |   566,417,200 |     75.00%
    |  |- java.util.concurrent.ConcurrentLinkedQueue$Node @
    0x9e52a730
    |           16 |   566,417,136 |     75.00%
    |  |  '- java.util.concurrent.ConcurrentLinkedQueue$Node @
    0x9e52a740         |           16 |   566,417,120 |     75.00%
    |  |     '- java.util.concurrent.ConcurrentLinkedQueue$Node @
    0x9e52a750      |           16 |   566,417,104 |     75.00%
    |  |        '-
    java.util.concurrent.ConcurrentLinkedQueue$Node @
    0x9e52a760   |           16 |   566,417,088 |     75.00%
    |  |           '-
    java.util.concurrent.ConcurrentLinkedQueue$Node @
    0x9e52a770|           16 |   566,417,072 |     75.00%
    

Local fix

  • N/A
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server                                      *
    ****************************************************************
    * PROBLEM DESCRIPTION: Asyncbeans logs ASYN0066E and can       *
    *                      lead to a slow memory leak              *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    The asynchbeans component uses the Object pool service to pool
    the WorkTimeoutListener alarm objects needed to keep track of
    the Work Timeout. The Object pool internally uses the unbounded
    queue based linked nodes structure
    java/util/concurrent/ConcurrentLinkedQueue for maintaing the
    objects pooled within it. When the work starts, a
    WorkTimeoutListener alarm object is obtained from the pool and
    likewise when the work completes, the same is returned back to
    the pool as part of the cleanup. However there is a timing
    window possible that causes the cleanup operation to be
    performed by both the work thread as well as the worktimeout
    alarm thread. This cause the pooled object to be returned back
    twice. Since the datastructure
    java/util/concurrent/ConcurrentLinkedQueue is unbounded, when
    the object is returned back twice, there is a chance for the
    Object pool to leak since the objects that are returned gets
    added to the tail of the queue.
    This APAR is applicable :
    a) When you notice the following error messages logged
    repeatedly in the SystemOut.log
    WorkTimeoutLi E ASYN0066E: The method <null> threw an
    Throwable: {1}.
    b) When heapdump analysis shows memory leak suspects
    indicating large number of
    java/util/concurrent/ConcurrentLinkedQueue$Node instances
    containing com/ibm/ws/asynchbeans/WorkTimeoutListener
    instances. Note : The leak can be very slow and it can take
    several days for the memory leak and OOM to manifest.
    Example :
    212,757,328 bytes (30.13 %) of Java heap is used by 13,297,159
    instances of java/util/concurrent/ConcurrentLinkedQueue$Node
    Contains the following object:
    - 13,297,158 instances of
    com/ibm/ws/asynchbeans/WorkTimeoutListener holding 425,509,056
    bytes
    The cause of this issue is that the APAR PI05902 corrected a
    problem where alarms were getting fired incorrectly. That is,
    WorkTimeoutListener alarms were getting fired even if the work
    had been completed well before the timeout. However there is a
    possibility where cleanup can occur twice due to timings with
    which work gets completed and worktimeout expires. This can
    cause the WorkTimeoutListener to get returned back to the
    object pool twice leading to the accumulation of
    WorkTimeoutListener objects and ASYN0066E.
    

Problem conclusion

  • The asynchbeans code has been corrected to handle
    the cleanup of the WorkTimeoutListener alarm objects properly.
    
    The fix for this APAR is currently targeted for
    inclusion in fix packs 8.5.5.6, 8.0.0.11, and 7.0.0.39. Please
    refer to the Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PI32169

  • Reported component name

    WEBSPHERE APP S

  • Reported component ID

    5724J0800

  • Reported release

    800

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2015-01-06

  • Closed date

    2015-04-03

  • Last modified date

    2015-04-03

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE APP S

  • Fixed component ID

    5724J0800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP

  • R850 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 8.0

Reference #: PI32169

Modified date: 03 April 2015