APAR status
Closed as duplicate of another APAR.
Error description
APAR IT10092 (FLOW HANGS DUE TO DEADLOCK IN FILEOUTPUT NODE) introduced a fix for a Java deadlock between a message flow thread currently executing code in a FileOutput node and a background reaper thread that closes leftover files. However this fix does not fix all possible deadlocks, and additionally introduced another possible deadlock. Inspection of the Java threads in the hung DataFlowEngine process using a javacore or Java debugger shows a deadlock as follows: 3XMTHREADINFO "FlowGroupFilePool file close daemon" J9VMThread:0x00000000119C5000, j9thread_t:0x00007F3FF02BECF0, java/lang/Thread:0x0000000001FABF38, state:B, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x17, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x663B, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201) 3XMTHREADINFO2 (native stack address range from:0x00007F3FDF373000, to:0x00007F3FDF474000, size:0x101000) 3XMCPUTIME CPU usage total: 0.320237489 secs 3XMTHREADBLOCK Blocked on: com/ibm/broker/flowgroupfiles/FlowGroupFilePool@0x0000000001FAC7 78 Owned by: "Thread-35" (J9VMThread:0x0000000011B4C800, java/lang/Thread:0x0000000002140AC8) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/ibm/broker/flowgroupfiles/FlowGroupFile.basicClose(FlowGroup File.jav a:362(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8, entry count: 3) 4XESTACKTRACE at com/ibm/broker/flowgroupfiles/FlowGroupFile.closedIfNoLongerReta ined(Flo wGroupFile.java:313(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8, entry count: 2) 4XESTACKTRACE at com/ibm/broker/flowgroupfiles/FlowGroupFilePool.tryToReap(FlowGr oupFileP ool.java:625(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8, entry count: 1) 4XESTACKTRACE at com/ibm/broker/flowgroupfiles/FlowGroupFilePool.run(FlowGroupFil ePool.ja va:575) 4XESTACKTRACE at java/lang/Thread.run(Thread.java:798) 3XMTHREADINFO "Thread-35" J9VMThread:0x0000000011B4C800, j9thread_t:0x00007F4020018120, java/lang/Thread:0x0000000002140AC8, state:B, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x2F, isDaemon:false) 3XMTHREADINFO1 (native thread ID:0x6934, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201) 3XMTHREADINFO2 (native stack address range from:0x00007F3FC9980000, to:0x00007F3FC9B01000, size:0x181000) 3XMCPUTIME CPU usage total: 24.151153362 secs 3XMTHREADBLOCK Blocked on: com/ibm/broker/flowgroupfiles/FlowGroupFile@0x0000000001B729E8 Owned by: "FlowGroupFilePool file close daemon" (J9VMThread:0x00000000119C5000, java/lang/Thread:0x0000000001FABF38) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/ibm/broker/flowgroupfiles/FlowGroupFilePool.open(FlowGroupFi lePool.j ava:195(Compiled Code)) 5XESTACKTRACE (entered lock: com/ibm/broker/flowgroupfiles/FlowGroupFilePool@0x0000000001FAC7 78, entry count: 1) 4XESTACKTRACE at com/ibm/broker/flowgroupfiles/FlowGroupFilePool.getOutputStream( FlowGrou pFilePool.java:458(Compiled Code)) 4XESTACKTRACE at com/ibm/broker/nodes/filenodes/AbstractFileOutputNode.sendData(A bstractF ileOutputNode.java:358(Compiled Code)) 4XESTACKTRACE at com/ibm/broker/nodes/filenodes/ComIbmFileOutputNode.sendData(Com IbmFileO utputNode.java:397(Compiled Code)) 4XESTACKTRACE at com/ibm/broker/plugin/MbOutputTemplateNode.sendData(MbOutputTemp lateNode .java:54(Compiled Code)) This deadlock is more likely to occur when running with additional instances, or if closure of the files by the flow thread is delayed - for example, if there is a large number of files being transferred to a remote system over FTP.
Local fix
NA
Problem summary
**************************************************************** USERS AFFECTED: Closed as duplicate of IT13290. Platforms affected: z/OS, MultiPlatform **************************************************************** PROBLEM DESCRIPTION: Closed as duplicate of IT13290.
Problem conclusion
Closed as duplicate of IT13290. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.0 9.0.0.7 v10.0 10.0.0.5 The latest available maintenance can be obtained from: http://www-01.ibm.com/support/docview.wss?rs=849&uid=swg27006041 If the maintenance level is not yet available,information on its planned availability can be found on: http://www-1.ibm.com/support/docview.wss?rs=849&uid=swg27006308 ---------------------------------------------------------------
Temporary fix
Comments
This APAR is a duplicate of IT13290
APAR Information
APAR number
IT14818
Reported component name
INTEGRATION BUS
Reported component ID
5724J0530
Reported release
900
Status
CLOSED DUB
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-04-16
Closed date
2016-05-26
Last modified date
2016-05-26
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSNQK6","label":"IBM Integration Bus"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
23 March 2020