IT19455: Files transferred using the IBM MQ Managed File Transfer FTP protocol bridge agent have corrupted names

APAR status

• Closed as program error.

Error description

• A file is transferred from a Linux system to an FTP server
  running on Microsoft Windows using the IBM MQ Managed File
  Transfer via a Protocol Bridge Agent.
  
  The filename contains a mix of ASCII and Japanese characters.
  After the transfer has been completed, the filename on the
  Windows system is corrupt, containing multiple non-ASCII
  characters in place of each Japanese character.
  
  In addition, it is observed that when attempting to transfer
  files from the FTP server back to the Linux system, where the
  files contain characters outside of the ASCII range, IBM MQ
  Managed File Transfer fails the transfer reporting that it
  cannot locate the files.
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the Protocol Bridge Agent who are
  transferring files to/from an FTP/FTPS server which have a name
  (which includes the location of where the file is, or is going
  to on the FTP server) which contains characters which are
  outside of the 7-bit ASCII range.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  When the Protocol Bridge Agent is used to transfer a file to or
  from an FTP server, data sent over the control channel which is
  established between the Protocol Bridge Agent and the FTP server
  is encoded by default using the UTF-8 encoding sequence, as
  strongly recommended by RFC 2640 in section 2:
  
      https://tools.ietf.org/html/rfc2640
  
  which extended the original FTP specification to add support for
  a level of internationalisation.
  
  The Protocol Bridge Agent is capable of using other encoding
  formats, and is configured through the use of the
  "controlEncoding" property, as described in the Knowledge Center
  on the following URI:
  
  https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.
  ibm.wmqfte.doc/protocol_bridge_properties_format.htm
  
  However it should be noted that the FTP specification does not
  provide a mechanism for negotiation of the character encoding
  used on the control channel.  This means that changing the value
  of "controlEncoding" may not have desirable effects unless the
  user also configures the FTP(S) server to match the defined
  character encoding as specified on the "controlEncoding"
  property.  This  requires the FTP server to provide a mechanism
  to configure the encoding.
  
  
  Unfortunately not all FTP servers have adopted the above RFC
  2640 recommendation of using UTF-8 for the default encoding of
  the control channel.  For example the Microsoft Windows server
  2012 IIS FTP server running in the EN locale appears to
  interpret the bytes sent over the control channel using the
  Windows-1252 encoding.
  
  The result of this is that is you are transferring a file to the
  Windows FTP server, where the filename is:
  
    ABAB?ABAB
  
  which has the UTF-8 encoded byte values:
  
    0x41, 0x42, 0x41, 0x42, 0xe5 0x8c 0xbb, 0x41, 0x42, 0x41, 0x42
  
  When this is transferred to the Windows FTP server, three
  characters are seen in place of the 3-byte UTF-8 encoded
  character:
  
    ABABå?»ABAB
  
  It was also observed that other FTP clients ('lftp' and
  'FileZilla') can successfully transfer the same filename to the
  Windows FTP server, while preserving the character
  representation.
  

Problem conclusion

• An expired RFC draft publication provides a hint for what
  command the Microsoft Windows IIS FTP server is expecting to
  receive to negotiate the character encoding used on the control
  channel, which is the RFC:
  
    https://tools.ietf.org/html/draft-ietf-ftpext-utf-8-option-00
  
  This states:
  
  "The user issues the OPTS UTF-8 command to indicate its
  willingness to
     send and receive UTF-8 encoded pathnames over the control
  connection.
     Prior to sending this command, the user should not transmit
  UTF-8
     encoded pathnames."
  
  After testing, it was found that the Windows FTP server has a
  "UTF8" feature, which makes its receptive to the FTP enabling
  command:
  
    OPTS UTF8 ON
  
  This results in the Windows FTP server interpreting the bytes
  received as being encoded in UTF-8.   As a consequence, support
  to make use of this options command has been added to the
  Protocol Bridge Agent.
  
  In order to configure the Protocol Bridge Agent to request this
  option, the following attribute and value needs to be added to
  the "tns:ftpServer" element of the Protocol Bridge Agent's
  "ProtocolBridgeProperties.xml" file:
  
      requestUTF8ControlEncoding="true"
  
  For example, for a default location MQ installation your
  "ProtocolBridgeProperties.xml" file will be located on the
  filesystem at the location:
  
  /var/mqm/mqft/config/<MFT_QMGR>/agents/<PBA_AGENT_NAME>/Protocol
  BridgeProperties.xml
  
  replacing <MFT_QMGR> with your MFT queue manager name, and
  <PBA_AGENT_NAME> with your protocol bridge agent's name.  Within
  this file you will find an element with the name
  "tns:ftpServer".  Add the above attribute and value this
  element, for example as follows:
  
      <tns:ftpServer
                  name="FTP_server_host"
                  host="FTP_server_address"
                  platform="WINDOWS"
                  timeZone="Europe/London"
                  locale="en_GB"
                  fileEncoding="UTF-8"
                  controlEncoding="UTF-8"
                  requestUTF8ControlEncoding="true"
                  listFormat="unix"
                  limitedWrite="false"  />
  
  
  By default this option is not used, in keeping with the
  recommendations of the FTP specification.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v8.0       8.0.0.8
  v9.0 CD    9.0.4
  v9.0 LTS   9.0.0.2
  
  The latest available FTE maintenance can be obtained from
  'Fix List for WebSphere MQ File Transfer Edition 7.0'
  http://www-01.ibm.com/support/docview.wss?uid=swg27015313
  
  The latest available MQ maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ MFT V8.0

• Fixed component ID

  5724H7252

Applicable component levels

• R800 PSY

     UP