IT20204: MONITORING SVC 7.8.1 CAUSES SPECTRUM CONTROL TO HANG WITH NO CONNECTIONS

IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)

APAR status

• Closed as program error.

Error description

• After upgrading SVC firmware to 7.8.1, the device server becomes
  inaccessible and appears to hang.  Cannot access the Spectrum
  Control user interface until the connection between Spectrum
  Control and the SVC is shut down.
  
  The follow messages are observed repeatedly for different port
  numbers in dmSvcTrace.log and traceTPCDeviceServer.log:
  
    SSH Port forwarding failed for local  port <#>
  com.ibm.tpc.monitor.SVCEventListener
  
    establishSshTunnelPortForwarding
  SVC-Notification-xxxx-xxxx-xxxx
  
    Connection failed. <SVC_IP_Address>
      com.ibm.tpc.common.api.napi.NAPIConnectionException: rc = 0,
  msg = No connection established.
  
  You would also see a large number of network connections owned
  by the device server pid. (ps -ef | grep deviceServer)
  > netstat -lanp | grep LISTEN | grep <device server pid>
  
  As a result, errors like the following are also logged:
    Failed to create input file
  /opt/IBM/TPC/device/log/epm/xxxx.txt
      java.io.FileNotFoundException:
  /opt/IBM/TPC/device/log/epm/xxxx.txt (Too many open files)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Control 5.2.x users monitoring                  *
  * SVC/Storwize/Flash devices at firmware level 7.8.1           *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * IBM Spectrum Control 5.2.x device server becomes             *
  * inaccessible and appears to hang after adding or upgrading a *
  * SVC/Storwize/Flash device to firmware to 7.8.1.  The device  *
  * server becomes inaccessible and appears to hang.  Users      *
  * cannot access the Spectrum Control user interface until the  *
  * connection between Spectrum Control and the SVC is shut      *
  * down.                                                        *
  *                                                              *
  * See IBM Support Flash (Alert) for additional details:        *
  * http://www-01.ibm.com/support/docview.wss?uid=swg22002420    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply IBM Spectrum Control fix maintenance when available.   *
  *                                                              *
  * Upgrade SVC/Storwize/Flash devices to firmware to 7.8.1.1 or *
  * higher.  Or contact IBM Support to request an ifix for 7.8.1 *
  * -- reference SVC APAR HU01531                                *
  ****************************************************************
  

Problem conclusion

• The solution to this problem includes fixes for both IBM
  Spectrum Control and the storage system.  Apply the fix for
  Spectrum Control to stabilize the system and prevent it from
  hanging.  Reboot the Spectrum Control server to free up
  connections prior to applying a fix.  Also apply a minimum
  firmware level of 7.8.1.1, or the ifix for APAR HU01531 to
  resolve connection issues with the storage system for event
  notification.
  
  
  The Spectrum Control fix for APAR IT20204 is targeted for the
  following maintenance package:
  
  | refresh pack | 5.2-TIV-TPC-RP0014 - target May 2017
  
  Fixed in IBM Spectrum Control 5.2.14
  
  http://www.ibm.com/support/docview.wss?&uid=swg21320822
  
  The target dates for future refresh packs do not represent a
  formal commitment by IBM. The dates are subject to change
  without notice.
  
  Recommended to upgrade SVC/Storwize/Flash devices to firmware to
  7.8.1.1 or higher.  Or contact IBM Support to request an ifix
  for 7.8.1 -- reference SVC APAR HU01531.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TPC

• Fixed component ID

  5608TPC00

Applicable component levels