IBM Support

TroubleShooting: Object Request Broker (ORB) problems

Technote (troubleshooting)


Problem(Abstract)

TroubleShooting for ORB problems with IBM WebSphere Application Server. This should help address common issues with this component before calling IBM support and save you time.

Resolving the problem

Troubleshooting ORB problems in IBM WebSphere Application Server. This page should help you address common issues with ORB before engaging IBM support which can save you time in resolving the issue.


Tab navigation

Troubleshooting topics:

Tab navigation

Overview

This topic discusses common ORB issues that can lead to deadlocks or hangs.

Topics Covered:

socketWrite Hangs


socketWrite Hangs

SocketWrite issues can occur on either the client or server ORB, during the writing of request data (client-side) or reply data (server-side). Since the socketWrite() method is a blocking call, it will only complete when the remote side completely reads the data from the incoming buffer. If for any reason the receiving side fails to read the data from the socket, the sending socketWrite thread can hang. In many cases, the problem is not confined to a single thread, but usually affects many/all of the ORB threads trying to write data to sockets.


Usually only one thread is stuck in socketWrite, and due to the nature of threads sharing a single ORB connection, all other ORB threads (WebContainer threads included) will be blocked in a slightly different stack waiting for access to the same socket. The following shows each of these stack conditions:

Client-side socketWrite stack example (SSL stack will include com.ibm.jsse2 calls):

3XMTHREADINFO "ORB.thread.pool : 1023" (TID:0x30554760, sys_thread_t:0x57323028, state:R, native ID:0x14C33) prio=5
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:103)
at java.net.SocketOutputStream.write(SocketOutputStream.java:147)
at com.ibm.rmi.util.buffer.SequentialByteBuffer.flushTo(SequentialByteBuffer.java:410)
at com.ibm.rmi.util.buffer.SequentialByteBuffer.flushTo(SequentialByteBuffer.java:439)
at com.ibm.rmi.iiop.IIOPOutputStream.writeTo(IIOPOutputStream.java:541)
at com.ibm.rmi.iiop.Connection.write(Connection.java:2225)
at com.ibm.rmi.iiop.Connection.send(Connection.java:2267)
at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:338)
at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:424)


Client-side stack waiting to get into socketWrite:

3XMTHREADINFO "ORB.thread.pool : 1011" (TID:0x30543190, sys_thread_t:0x55E2E1A8, state:CW, native ID:0x14426) prio=5
4XESTACKTRACE at com.ibm.rmi.util.buffer.SequentialByteBuffer.flushTo(SequentialByteBuffer.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.iiop.IIOPOutputStream.writeTo(IIOPOutputStream.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.iiop.Connection.write(Compiled Code)
4XESTACKTRACE at com.ibm.rmi.iiop.Connection.send(Connection.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java(Compiled Code))


How to determine if a JVM is experiencing socketWrite issues:

1. Check SystemOut.log for hung thread messages (WSVR0605W) which will show socketWrite stacks.

2. Take a javacore/threaddump to examine ORB threads.


Possible Causes:

1. Network issues causing slow traffic or dropped packets.

2. OOM on the receiving side which kills the receiving reader or worker thread (in the case of reading fragments).

3. Server's ORB worker threads are all hung. This in turn causes its ReaderThreads to become hung waiting for an available worker thread, prohibiting the reading of the incoming request msg/fragments from the client. In this case, javacores are needed on the server to determine its threads' states and why they are hung.


Solution:

As seen above, socketWrite hangs can occur for any number of reasons. Obviously addressing the root problem (network, OOM, etc) can help prevent socketWrite issues, but the best measure is a preventative one using a number of ORB properties. The main property to set is com.ibm.CORBA.SocketWriteTimeout. This property will cause the writing thread to timeout, at which point a com.ibm.rmi.iiop.IIOPOutputStream$SocketWriteTimeOutException will be thrown, and the request will fail. Normal retry logic applies where the ORB will retry once by default.

Property: com.ibm.CORBA.SocketWriteTimeout
Settings:
  • Set in seconds
  • Default: 0
  • Set on client and/or server. In most cases, the problem is experienced on the client.
  • Recommended values. 2 options:
1. Set to a smaller value such as 5 or 10 seconds
If a server is truly hung and not slow, this allows client threads to clean up quickly. This approach however could terminate requests which aren't actually hung in socketWrite but are just taking a longer time than normal (due to socket contention, network slowdown, CPU spike, extra-long server process time, etc).

2. Set to a larger value such as 20 or 30 seconds
This is more forgiving of slow server response times, CPU spikes, etc. by not giving up too quickly on the socketWrites.

NOTE: The ORB com.ibm.CORBA.RequestTimeout timer will not start until the request message is successfully written to the server (ie socketWrite completes). In light of that, the SocketWriteTimeout (SWTO) property can be set to any value irrespective of the ORB RequestTimeout value.


Additional properties that can be set:

Property: com.ibm.CORBA.FragmentSize
Settings:
  • Default: 1024
  • Recommended values. 2 options:
1. Set to 0 (no fragmentation). This means the sending thread can write the entire message once it obtains the socketWrite lock.

2. Set to a larger fragmentSize, such as 10240, if the average message size is large (over 50k). This minimizes the number of times a particular thread has to obtain the socketWrite lock.
  • Set on all clients and/or servers.


Property: com.ibm.CORBA.ConnectionMultiplicity
Settings:
  • Default: 1
  • Recommended values: 5 – 10
This property designates how many connections a client will have to a given server. Increasing this number gives the client more connections (and hence more sockets) to each server which sending threads can share, thereby reducing contention over a single socket.

NOTE: Additional resources (more reader threads and sockets) will be required on both the client and server(s) so be cautious of setting too large a value.
  • Usually set on the client side (but can be set on a server).


Property: com.ibm.websphere.orb.threadPoolTimeout
Settings:
  • Set in ms
  • Default: 0
  • Recommended value: 10000
This property governs how long a server ReaderThread will wait for an available WorkerThread to handle the new incoming request. If there are NO free WorkerThreads, the ReaderThread will wait indefinitely by default until a free WorkerThread becomes available, which can lead to deadlocks or hangs. Setting this property >0 allows any particular ReaderThread to timeout after a certain period of time, which then allows the sending client side thread stuck in socketWrite to be released.

NOTE: Tuning this property is similar to SocketWriteTimeout in that there are 2 basic approaches: small or large timeout setting.
  • Set on the server

Example of hung ReaderThread:

3XMTHREADINFO "RT=51:P=946784:O=0:WSTCPTransportConnection[addr=1.2.3.4,port=47847,local=35293]" (TID:0x7000000000B21E0, sys_thread_t:0x11AF3D548, state:CW, native ID:0x27B3D) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.util.BoundedBuffer.put(BoundedBuffer.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.util.ThreadPool.execute(ThreadPool.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.util.ThreadPool.execute(ThreadPool.java(Compiled Code))
4XESTACKTRACE at com.ibm.ejs.oa.pool.ThreadPool.startWorkerThread(ThreadPool.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.iiop.Connection.processInput(Connection.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.iiop.Connection.doReaderWorkOnce(Connection.java(Compiled Code))
4XESTACKTRACE at com.ibm.rmi.transport.ReaderThread.run(ReaderPoolImpl.java:137)


APARS needed:
Shipped in JDK60 SR10, and JDK626 and above
Shipped in JDK60 SR14, 626SR6, 70SR5
The NPE is logged in SystemErr.log, so this can be checked to confirm issue with the SocketTimer thread.
Shipped in WAS 7.0.0.29, 8.0.0.7, 8.5.5.0

For more information on threads stuck in socketWrite see:
Hung thread behavior with SocketWriteTimerThread




Document information

More support for: WebSphere Application Server
Object Request Broker (ORB)

Software version: 7.0, 8.0, 8.5, 8.5.5, 9.0

Operating system(s): AIX, HP-UX, IBM i, Linux, Solaris, Windows

Software edition: Base, Express, Network Deployment

Reference #: 2015464

Modified date: 11 April 2018


Translate this page: