PM22988: WEBSPHERE CONTROLLER HANG

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• A WebSphere server becomes unresponsive.
  
  Review of the servant regions show 26 threads stuck in
  bboosout(BBOOSOUT_Functions,ORB_Request_SharedMemberData*,lo
              +00000000              BBGBOA
  ORB_Request::comm_outbound_request(unsigned int)
              +00000ADE              BBGBOA
  
  and threads stuck in
  bboosout(BBOOSOUT_Functions,ORB_Request_SharedMemberData*,lo
              +00000000              BBGBOA
  ORB_Request::comm_outbound_locate(unsigned int)
              +000015B8              BBGBOA
  
  Review of controller region shows 1 ACRW thread in
  
  Entry      Offset    Function
  -----      ------    --------
  30a00f38   00000000  bbocfasy(long long,int*,int*,int*)
  309b6d98   00000708  CF_TCP_Connection::send_data(bool,bool)
  309b6128   00000b02
  CF_TCP_Connection::async_send_msg(msghdr*,int,bool,bool,int)
  309fa980   000001be
  CF_TCP_Request::write(void*,msghdr*,int,int,bool,bool)
  309e5728   000007fa
  Java_com_ibm_ws390_tcp_channel_ZAioTCPChannelCPPUtilities_write
  7bb2d1b0   844d2e50   RUNCALLINMETHOD
  7bb63488   0000003e   gpProtectedRunCallInMethod
  7bb60fe0   0000001c   signalProtectAndRunGlue
  7b92eb08   00000356   j9sig_protect
  7bb659c0   000000a4   gpCheckCallin
  7bb5ea20   00000072   callStaticObjectMethodA
  30211de8   000012de
  ZIOPChannelBridge::send_outbound_request(ORB_Request*,void*,int,
  volatile int*,int)
  
  The other 24 ACRW threads are stuck behind the one above with
  this call stack:
  
  Entry      Offset     Function
  -----      ------     --------
  2f7aed78   000020c4   CEEOPCT
  2fa90c68   000000c0   pthread_cond_timedwait
  3215f728   00000412   monitor_wait
  32162398   00000014   j9thread_monitor_wait_timed
  7bba2660   00000010   callMonitorWaitTimed
  7bb3c4e0   0000030a   Z_OBJECTMONITORENTERBLOCKING
  7bb91b10   00000014   objectMonitorEnterBlocking
  7bb2d1b0   ffc4ca14   RUNCALLINMETHOD
  7bb63488   0000003e   gpProtectedRunCallInMethod
  7bb60fe0   0000001c   signalProtectAndRunGlue
  7b92eb08   00000356   j9sig_protect
  7bb659c0   000000a4   gpCheckCallin
  7bb5ea20   00000072   callStaticObjectMethodA
  30211de8   000012de
  ZIOPChannelBridge::send_outbound_request(ORB_Request*,void*,int,
  volatile int*,int)
  
  Javacore of this same controller does indicate that the first
  TCB holds a flat lock while sending data on the connection.
  The send (bbocfasy) has blocked.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server V7.0                                 *
  ****************************************************************
  * PROBLEM DESCRIPTION: WebSphere Application Server for z/OS   *
  *                      Controller receives errors due to a     *
  *                      native out of memory condition.         *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  When WebSphere Application Server for z/OS has several threads
  that need to send data outbound over IIOP on the same
  connection, the first thread actually begins to send the data
  and the other threads queue their data up for sending after
  the first thread is finished.  If there is an external delay
  sending data (such as TCP/IP not responding for a long time),
  this queue can build up and use excess native memory.  A dump
  of the Controller will show a linked list of WsByteBuffer
  objects owned by a ZIOPAsyncWriteQueue.  Each WsByteBuffer is
  a direct buffer backed by native storage.
  

Problem conclusion

• A new WebSphere for z/OS environment variable,
  iiop_max_send_queue_megsize, was introduced to cap the maximum
  amount of storage used by a single IIOP connection's send
  queue.
  
  APAR PM22988 requires changes to documentation.
  
  NOTE: Periodically, we refresh the documentation on our
  Web site, so the changes might have been made before you
  read this text. To access the latest on-line
  documentation, go to the product library page at:
  
  http://www.ibm.com/software/webservers/appserv/library
  
  The following change to the WebSphere Application Server
  Version 7.0 Information Center will be made available in
  February, 2011.
  
  The following description of the
  iiop_max_send_queue_megsize environment variable will be
  added to the "Application server custom properties that
  are unique for the z/OS platform" topic:
  
  iiop_max_send_queue_megsize
  
  Specifies, in megabytes, the maximum amount of data that
  can be queued up to send asynchronously over a single IIOP
  connection. If the amount of data queued exceeds the
  specified value, future IIOP requests over this connection
  fail with a C9C26A4D minor code. The minimum value for this
  property is 0, which indicates that there is no limit to
  the amount of data that can be queued for sending. The
  maximum value is 2048.
  
  Data Type       Integer
  Default         0
  Used by Daemon  No
  
  APAR PM22988 is currently targeted for inclusion in
  Service Level (Fix Pack) 7.0.0.15 of WebSphere
  Application Server.
  
  Please refer to URL:
  //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
  for Fix Pack availability.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PM22587

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE FOR Z

• Fixed component ID

  5655I3500

Applicable component levels

• R700 PSY UK65061

     UP11/03/04 P F103

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.