IBM Support

PI36965: Deadlocks with XIO network communication are possible when you run the WebSphere eXtreme Scale teardown command.

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • When running the teardown command, a possible deadlock may
occur with XIO network communication.  When stopping servers, a
server with the deadlock may not stop and be in a hung state.

Javacore dumps taken at the time of the deadlock show
information similar to the following:


Deadlock detected !!!
---------------------
Thread "Thread-129" (0x0000000041DA5400)
is waiting for:
sys_mon_t:0x0000010022D57BF8 infl_mon_t:
0x0000010022D57C40:
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry@0x000
000076E0F0028

which is owned by:
Thread "xioNetworkThreadPool : 1" (0x0000000042F17D00)
which is waiting for:
sys_mon_t:0x0000010022D57C88 infl_mon_t:
0x0000010022D57CD0:
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService@0x000000
076E0F01
08

which is owned by:
Thread "Thread-129" (0x0000000041DA5400)



The threads in question have the following call stacks:



"Thread-129"
J9VMThread:0x0000000041DA5400,j9thread_t:0x00000100231857C0,
java/lang/Thread:0x000000075B8A02A0,state:B, prio=5
(java/lang/Thread getId:0x1B1, isDaemon:true)
(native thread ID:0x3DC01B5, native priority:0x5, native
policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
CPU usage total: 0.072845000 secs, user: 0.047762000 secs,
system: 0.025083000 secs
Blocked on:
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry@0x000000076E0F0028 Owned by: "xioNetworkThreadPool : 1"
(J9VMThread:0x0000000042F17D00,
java/lang/Thread:0x00000007752060E8)
Heap bytes allocated since last GC cycle=0 (0x0)
Java callstack:
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry.removeController(XIOConnectionControllerFactory.java:322)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionController.chec
kForNewConnection(XIOConnectionController.java:814)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionController.erro
r(XIOConnectionController.java:713)
(entered
lock:com/ibm/ws/xs/xio/transport/channel/XIOConnectionController
@0x000000075B93BDE8, entry count: 1)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionController.conn
ectFailed(XIOConnectionController.java:572)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnCallback.destroy(XIOC
onnCallback.java:29)
at
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService$ProbeAtt
empt.destroy(XIOOutboundTransportService.java:506)
at
com/ibm/ws/channel/framework/impl/OutboundVirtualConnectionImpl.
destroy(OutboundVirtualConnectionImpl.java:136)
at
com/ibm/wsspi/channel/base/OutboundConnectorLink.destroy(Outboun
dConnectorLink.java:63)
at
com/ibm/ws/xs/tcp/channel/impl/TCPConnLink.destroy(TCPConnLink.j
ava:533)
at
com/ibm/wsspi/channel/base/OutboundConnectorLink.close(OutboundC
onnectorLink.java:50)
at
com/ibm/ws/xs/tcp/channel/impl/TCPConnLink.close(TCPConnLink.jav
a:498)
at
com/ibm/ws/xs/tcp/channel/impl/TCPChannel.destroyConnLinks(TCPCh
annel.java:565)
at
com/ibm/ws/xs/tcp/channel/impl/TCPChannel.stop(TCPChannel.java:4
32)
at
com/ibm/ws/channel/framework/impl/ChannelFrameworkImpl.stopChann
el(ChannelFrameworkImpl.java:1520)
at
com/ibm/ws/channel/framework/impl/ChannelFrameworkImpl.stopChain
Internal(ChannelFrameworkImpl.java:3089)
(entered
lock:com/ibm/ws/channel/framework/impl/WSChannelFrameworkImpl@0x
0000000673820C90, entry count: 2)
at
com/ibm/ws/channel/framework/impl/OutboundVirtualConnectionFacto
ryImpl.destroy(OutboundVirtualConnectionFactoryImpl.java:198)
(entered
lock:com/ibm/ws/channel/framework/impl/WSChannelFrameworkImpl@0x
0000000673820C90, entry count: 1)
at
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService.stop(XIO
OutboundTransportService.java:260)
at
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService.stopIfRu
nning(XIOOutboundTransportService.java:120)
(entered lock:
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService@0x000000
076E0F0108, entry count: 1)
at
com/ibm/ws/objectgrid/server/impl/ServerImpl.stopXIO(ServerImpl.
java:4180)
at
com/ibm/ws/objectgrid/server/impl/ServerImpl.stopService(ServerI
mpl.java:1924)
.....



"xioNetworkThreadPool : 1"
J9VMThread:0x0000000042F17D00,j9thread_t:0x000001002A8D8740,
java/lang/Thread:0x00000007752060E8,state:B, prio=5
(java/lang/Thread getId:0xBF, isDaemon:true)
(native thread ID:0x4FA0069, native priority:0x5, native
policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
CPU usage total: 0.209290000 secs, user: 0.173591000
secs, system: 0.035699000 secs
Blocked on:
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService@0x000000
076E0F0108 Owned by: "Thread-129"
(J9VMThread:0x0000000041DA5400,java/lang/Thread:0x000000075B8A02
A0)
Heap bytes allocated since last GC cycle=0 (0x0)
Java callstack:
at
com/ibm/ws/xs/xio/transport/XIOOutboundTransportService.getInsta
nce(XIOOutboundTransportService.java:288)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionController.<ini
t>(XIOConnectionController.java:233)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry.addConnectionController(XIOConnectionControllerFactory.java:2
34)
(entered lock:
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry@0x000000076E0F0028, entry count: 1)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry.getConnectionController(XIOConnectionControllerFactory.java:1
35)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionControllerFacto
ry.getController(XIOConnectionControllerFactory.java:199)
at
com/ibm/ws/xs/xio/transport/channel/XIOConnectionController.chec
kForNewConnection(XIOConnectionController.java:822)
.....

Local fix

  • 



Problem summary


    

  • ****************************************************************
* USERS AFFECTED:  Customers who use eXtremeIO (XIO)           *
****************************************************************
* PROBLEM DESCRIPTION: A JVM deadlock can occur when an        *
*                      eXtreme Scale container attempts to     *
*                      stop while traffic is still being       *
*                      processed over XIO.                     *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A JVM deadlock can occur during a WebSphere eXtreme Scale
container server shutdown.

    



Problem conclusion


    

  • The code has been updated to correct this problem.

    



Temporary fix


    

  • 



Comments


    

  • 



APAR Information




    

  • APAR number
    PI36965
    • 

  • Reported component name
    WS EXTREME SCAL
    • 

  • Reported component ID
    5724X6702
    • 

  • Reported release
    860
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Special Attention
    NoSpecatt
    • 

  • Submitted date
    2015-03-13
    • 

  • Closed date
    2015-03-30
    • 

  • Last modified date
    2015-03-30
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    WS EXTREME SCAL
    • 

  • Fixed component ID
    5724X6702
    • 



Applicable component levels


    

  • R860  PSY
       UP
    • 








      
              
                            

              

              [{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSTVLU","label":"WebSphere eXtreme Scale"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"860","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
              

                          

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            30 March 2015
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback