Troubleshooting
Problem
Application Server startup fails with the following exception: [10/20/14 10:31:36:877 CDT] 0000000a WsServerImpl E WSVR0009E: Error occurred during startup com.ibm.ws.exception.RuntimeError: com.ibm.ws.exception.RuntimeError: com.ibm.ejs.EJSException: Could not register with Location Service Daemon, which could only reside in the NodeAgent. Make sure the NodeAgent for this node is up an running.; nested exception is: org.omg.CORBA.ORBPackage.InvalidName: LocationService:org.omg.CORBA.COMM_FAILURE: purge_calls:1897 Reason: CONN_ABORT (1), State: ABORT (5) vmcid: IBM minor code: 306 completed: Maybe at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:199) at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:140) Caused by: org.omg.CORBA.COMM_FAILURE: purge_calls:1897 Reason: CONN_ABORT (1), State: ABORT (5) vmcid: IBM minor code: 306 completed: Maybe at com.ibm.rmi.iiop.Connection.purge_calls(Connection.java:1896) at com.ibm.rmi.iiop.Connection.doReaderWorkOnce(Connection.java:2991) at com.ibm.rmi.transport.ReaderThread.run(ReaderPoolImpl.java:138) The nodeagent logs do not show any exception. Object Request Broker tracing shows that application server sends a request to the ORB_LISTENER_ADDRESS port of the nodeagent, but this message is never received by the nodeagent. The server then logs an almost immediate IOException. [10/20/14 10:31:29:849 CDT] 00000013 ORBRas 3 com.ibm.rmi.iiop.Connection doReaderWorkOnce:2989 RT=0:P=687396:O=0:WSTCPTransportConnection[addr=123.456.78.90,port=9101,local=43411] The following exception was logged java.io.IOException: bytesRead < 0 at com.ibm.rmi.iiop.Connection.readMoreData(Connection.java:1605) at com.ibm.rmi.iiop.Connection.createInputStream(Connection.java:1406) at com.ibm.rmi.iiop.Connection.doReaderWorkOnce(Connection.java:2980) at com.ibm.rmi.transport.ReaderThread.run(ReaderPoolImpl.java:138)
Symptom
Communication sent from client-side ORB (For Example: AppServer, full client, Java™ client) is not being received on the server-side ORB (For Server, NodeAgent).
Cause
The problem is caused by a port conflict with another process that also uses the ORB_LISTENER_ADDRESS used by the nodeagent. This accounts for why the server is able to send the request, but it never makes it to the nodeagent. The other process that listens on the port receives the request but does not know what to do with it and hence closes it. So, if a WebSphere Application Server process is able to make a connection, write data to that connection, and then gets an almost immediate java.io.IOException: bytesRead < 0, message because the connection was closed, the it is most likely a port conflict or hijacking (Another process using the same port to which WebSphere Application Server process is trying to write to) that causes it.
If we extrapolate, though this specific problem is an application server startup problem, this problem and resolution can apply to any EJB call that fails. From an Object Request Broker trace, if we can determine that the client is able to send a locreq message to the server and succeeds, but the request never makes it to the server and the client receives an almost immediate java.io.IOException: bytesRead < 0, the most likely cause could be that the port on the server side node to which client is trying to connect to is in use by a different process and the port conflict leads to this problem
Diagnosing The Problem
Run "netstat -an" command to determine if there are multiple entries for the port in question in the LISTEN/LISTENING state. UNIX tool LSOF or Windows Process Explorer can help determine which other process is using the port.
ORB trace (with CommTrace turned on ) will clearly show the outgoing message from the "client" side, but the server side ORB trace will show no indication of the message ever having arrived.
Resolving The Problem
Look for any other processes that are running and using the same port that the WebSphere Application Server process is trying to connect to and failing. Assign a different unique port to the WebSphere Application Server process or stop the other process that is using this port to resolve the problem. In this case, the ORB_LISTENER_ADDRESS of the nodeagent was changed to a unique port and the nodeagent was recycled which allowed the application server to then successfully startup.
Was this topic helpful?
Document Information
Modified date:
23 June 2018
UID
swg21407925