IT31225: IBM MQ Managed File Transfers from a SFTP server occasionally hang

APAR status

• Closed as program error.

Error description

• An IBM MQ V9.1.0.3 Managed File Transfer protocol bridge agent
  connects to a file server using the SFTP protocol, and acts as
  the source agent for managed transfers.
  
  Intermittently, a managed transfer being processed by the agent
  starts and then appears to hang.
  
  A Javacore from the agent taken when the managed transfer is
  stuck shows that an internal FileIOWorker thread has become
  blocked. An example of the call stack for this thread, as seen
  in the Javacore, is shown below:
  
  3XMTHREADINFO      "FileIOWorker-0:6"
  J9VMThread:0x0000000001B73F00, omrthread_t:0x00007F47400422E8,
   java/lang/Thread:0x00000000820A4ED8, state:CW, prio=5
  (java/lang/Thread getId:0x4F5, isDaemon:true)
  (native thread ID:0x18D81, native  priority:0x5, native
  policy:UNKNOWN, vmstate:CW, vm thread flags:0x00080181)
  (native stack address range from:0x00007F475513E000,
  to:0x00007F475517E000, size:0x40000)
  CPU usage total: 1.805280309 secs, current
  category="Application"
  Waiting on:
    com/jcraft/jsch/Channel$MyPipedInputStream@0x0000000082096278
  Owned by: <unowned>
  Heap bytes allocated since last GC cycle=0 (0x0)
  Java callstack:
  at java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:218(Compiled Code))
  at
  java/io/PipedInputStream.read(PipedInputStream.java:337(Compiled
   Code))
  (entered lock:
  com/jcraft/jsch/Channel$MyPipedInputStream@0x0000000082096278,
  entry count: 2)
  at
  java/io/PipedInputStream.read(PipedInputStream.java:388(Compiled
   Code))
  (entered lock:
  com/jcraft/jsch/Channel$MyPipedInputStream@0x0000000082096278,
  entry count: 1)
  at java/io/InputStream.skip(InputStream.java:246)
  at com/jcraft/jsch/ChannelSftp.skip(ChannelSftp.java:2921)
  at com/jcraft/jsch/ChannelSftp.access$600(ChannelSftp.java:36)
  at
  com/jcraft/jsch/ChannelSftp$RequestQueue.cancel(ChannelSftp.java
  :1270)
  at com/jcraft/jsch/ChannelSftp$2.read(ChannelSftp.java:1429)
  at
  java/io/DataInputStream.readFully(DataInputStream.java:206(Compi
  led Code))
  at
  com/ibm/wmqfte/io/bridge/BridgeFileChannelImpl.read(BridgeFileCh
  annelImpl.java:798)
  at
  com/ibm/wmqfte/io/bridge/BridgeFileChannelImpl.read(BridgeFileCh
  annelImpl.java:734)
  at
  com/ibm/wmqfte/transfer/frame/impl/TransferFrameSenderImpl.reque
  stChunk(TransferFrameSenderImpl.java:697)
  at
  com/ibm/wmqfte/transfer/frame/impl/TransferFrameSenderImpl.acces
  s$000(TransferFrameSenderImpl.java:95)
  at
  com/ibm/wmqfte/transfer/frame/impl/TransferFrameSenderImpl$Chunk
  Requester.processFileIORequest(TransferFrameSenderImpl.java:137)
  at
  com/ibm/wmqfte/transfer/frame/impl/TransferFrameIOWorker.doWorkI
  mpl(TransferFrameIOWorker.java:318)
  (entered lock: java/lang/Object@0x0000000082096000, entry count:
  1)
  at
  com/ibm/wmqfte/io/impl/FTEFileIOWorker.doWork(FTEFileIOWorker.ja
  va:118)
  at
  com/ibm/wmqfte/io/impl/FTEFileIORequestQueue.run(FTEFileIOReques
  tQueue.java:244)
  at java/lang/Thread.run(Thread.java:812)
  at com/ibm/wmqfte/thread/FTEThread.run(FTEThread.java:70)
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of IBM MQ Managed File Transfer who
  have protocol bridge agents that:
  
  - Connect to file servers using the SFTP protocol.
  - And act as the source agent for managed transfers.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  IBM MQ Managed File Transfer includes a fork of the third-party
  JSch library.  This library is used by protocol bridge agents
  (PBAs) when communicating to file servers using the SFTP
  protocol. If a PBA is acting as the source agent for a managed
  transfer, it will make use of functions provided by JSch to
  connect to the file server and receive the file data for the
  source file.
  
  When reading file data from a file server, JSch sends a number
  of requests to the file server, asking for data. Each of these
  requests contains a sequence number, After the requests have
  been sent, JSch starts receiving the responses from the file
  server. The responses also contain a sequence number,
  corresponding to the request that the response is for.  JSch
  uses the sequence number in the response to ensure that the
  responses are received in the correct order.  After all of the
  responses have been received, JSch passes the file data back to
  the PBA for processing.
  
  If JSch detected that a response had arrived out of sequence, it
  would attempt to skip over this response, and then receive the
  next one.  However, due to an issue within the JSch code, the
  out of sequence response was not processed correctly.
  
  This led to undefined behaviour when trying to process
  subsequent responses which had been returned from the SFTP
  server.  This typically resulted in waiting for more data from
  the SFTP server than had been sent.  This in turn resulted in an
  internal FileIOWorker thread associated with a managed transfer
  within the PBA to hang.
  

Problem conclusion

• To resolve this issue, the fork of the JSch library included
  with IBM MQ Managed File Transfer has been updated to ensure
  that out of order responses are processed correctly.  After the
  responses have been dealt with, the JSch library will request
  the file data from the file server again.  Once the file data
  has been successfully received, JSch will pass it back to the
  protocol bridge agent for processing.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.1 LTS   9.1.0.7
  v9.2 LTS   9.2.0.1
  v9.x CD    9.2.1
  
  The latest available MQ maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  IBM MQ MFT V9.1

• Fixed component ID

  5724H7272

Applicable component levels