IBM Support

IT14818: FLOW HANGS DUE TO DEADLOCK IN FILEOUTPUT NODE (APAR IT10092 INCOMPLETE)

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as duplicate of another APAR.

Error description

  • APAR IT10092 (FLOW HANGS DUE TO DEADLOCK IN FILEOUTPUT NODE)
    introduced a fix for a Java deadlock between a message flow
    thread currently executing code in a FileOutput node and a
    background reaper thread that closes leftover files.
    However this fix does not fix all possible deadlocks, and
    additionally introduced another possible deadlock.
    Inspection of the Java threads in the hung DataFlowEngine
    process using a javacore or Java debugger shows a deadlock as
    follows:
    
    3XMTHREADINFO      "FlowGroupFilePool file close daemon"
    J9VMThread:0x00000000119C5000, j9thread_t:0x00007F3FF02BECF0,
    java/lang/Thread:0x0000000001FABF38, state:B, prio=5
    3XMJAVALTHREAD      (java/lang/Thread getId:0x17, isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x663B, native
    priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread
    flags:0x00000201)
    3XMTHREADINFO2            (native stack address range
    from:0x00007F3FDF373000, to:0x00007F3FDF474000, size:0x101000)
    3XMCPUTIME               CPU usage total: 0.320237489 secs
    3XMTHREADBLOCK     Blocked on:
    com/ibm/broker/flowgroupfiles/FlowGroupFilePool@0x0000000001FAC7
    78
    Owned by: "Thread-35" (J9VMThread:0x0000000011B4C800,
    java/lang/Thread:0x0000000002140AC8)
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0
    (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/broker/flowgroupfiles/FlowGroupFile.basicClose(FlowGroup
    File.jav
    a:362(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8,
    entry
    count: 3)
    4XESTACKTRACE                at
    com/ibm/broker/flowgroupfiles/FlowGroupFile.closedIfNoLongerReta
    ined(Flo
    wGroupFile.java:313(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8,
    entry
    count: 2)
    4XESTACKTRACE                at
    com/ibm/broker/flowgroupfiles/FlowGroupFilePool.tryToReap(FlowGr
    oupFileP
    ool.java:625(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8,
    entry
    count: 1)
    4XESTACKTRACE                at
    com/ibm/broker/flowgroupfiles/FlowGroupFilePool.run(FlowGroupFil
    ePool.ja
    va:575)
    4XESTACKTRACE                at
    java/lang/Thread.run(Thread.java:798)
    
    3XMTHREADINFO      "Thread-35" J9VMThread:0x0000000011B4C800,
    j9thread_t:0x00007F4020018120,
    java/lang/Thread:0x0000000002140AC8,
    state:B, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x2F,
    isDaemon:false)
    3XMTHREADINFO1            (native thread ID:0x6934, native
    priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread
    flags:0x00000201)
    3XMTHREADINFO2            (native stack address range
    from:0x00007F3FC9980000, to:0x00007F3FC9B01000, size:0x181000)
    3XMCPUTIME               CPU usage total: 24.151153362 secs
    3XMTHREADBLOCK     Blocked on:
    com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8
    Owned
    by: "FlowGroupFilePool file close daemon"
    (J9VMThread:0x00000000119C5000,
    java/lang/Thread:0x0000000001FABF38)
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0
    (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/broker/flowgroupfiles/FlowGroupFilePool.open(FlowGroupFi
    lePool.j
    ava:195(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    com/ibm/broker/flowgroupfiles/FlowGroupFilePool@0x0000000001FAC7
    78,
    entry count: 1)
    4XESTACKTRACE                at
    com/ibm/broker/flowgroupfiles/FlowGroupFilePool.getOutputStream(
    FlowGrou
    pFilePool.java:458(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/broker/nodes/filenodes/AbstractFileOutputNode.sendData(A
    bstractF
    ileOutputNode.java:358(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/broker/nodes/filenodes/ComIbmFileOutputNode.sendData(Com
    IbmFileO
    utputNode.java:397(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/broker/plugin/MbOutputTemplateNode.sendData(MbOutputTemp
    lateNode
    .java:54(Compiled Code))
    
    This deadlock is more likely to occur when running with
    additional instances, or if closure of the files by the flow
    thread is delayed -
    for example, if there is a large number of files being
    transferred to a remote system over FTP.
    

Local fix

  • NA
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    Closed as duplicate of IT13290.
    
    
    Platforms affected:
    z/OS, MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    Closed as duplicate of IT13290.
    

Problem conclusion

Temporary fix

Comments

  • This APAR is a duplicate of IT13290
    

APAR Information

  • APAR number

    IT14818

  • Reported component name

    INTEGRATION BUS

  • Reported component ID

    5724J0530

  • Reported release

    900

  • Status

    CLOSED DUB

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-04-16

  • Closed date

    2016-05-26

  • Last modified date

    2016-05-26

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSNQK6","label":"IBM Integration Bus"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
23 March 2020