IBM Support

WebSphere Application Server SOAP Connection Issues: "java.lang.SocketException: Connection reset" or "java.net.SocketTimeoutException: Read timed out"

Technote (troubleshooting)


Problem(Abstract)

When network latency issues exist, WebSphere Application Server might experience SOAP connectivity issues if a response is not received within 5 seconds of the original SOAP request.

Symptom

Symptoms include:

- addNode/removeNode command, wsadmin scripting, and/or SOAP connectivity errors

- "java.lang.SocketException: Connection reset" and/or "java.net.SocketTimeoutException: Read timed out" exceptions


Cause

Network latency/delays


Environment

WebSphere Application Server Express, Base, Network Deployment - All versions

Diagnosing the problem

The following pieces of data can be gathered to help determine the cause of the SOAP connectivity issues:


1) Use the -trace option on the addNode/removeNode Qshell commands
i.e. app_server_root/bin/addNode <host> <port> -conntype SOAP -profileName <profile> -trace

2) Specify the following diagnostic trace string for the Deployment Manager you are attempting to connect to:
*=info:com.ibm.ws.management.*=all:com.ibm.websphere.management.application.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all

For detailed instructions on how to set up diagnostic tracing in WebSphere Application Server, please refer to the following URL:
http://www-01.ibm.com/support/docview.wss?uid=swg21254706

3) TCP/IP traces from the both the DMGR and Application Server perspectives.
Use the TRCCNN command to gather the TCP/IP traces.

Command from the Application Server (who is connecting to the DMGR) perspective:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP () (SOAPPORT))
TRCCNN SET(*OFF) TRCTBL(IBM)

Command from the DMGR perspective:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP (SOAPPORT) ())
TRCCNN SET(*OFF) TRCTBL(IBM)

where SOAPPORT is the SOAP port of the DMGR you are attempting to connect to.

=======================================================================

Here are a few examples of typical error message that can occur when experiencing a SOAP connectivity issue with WebSphere Application Server:


[SOAPException: faultCode=SOAP-ENV:Client; msg=Connection reset; targetException=java.net.SocketException: Connection reset]

com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke

[8/19/13 11:20:00:614 EDT] 00000088 SOAPUtils 3 unable to extract the SOAP Call
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.ibm.ws.io.Stream.read(Stream.java:17)
at com.ibm.ws.io.ReadStream.read(ReadStream.java:184)
at com.ibm.ws.http.ContentLengthInputStream.read(ContentLengthInputStream.java:49)
at com.ibm.ws.io.ReadStream.read(ReadStream.java:184)
at org.apache.soap.transport.TransportMessage.<init>(TransportMessage.java:141)
at org.apache.soap.server.ServerUtils.readEnvelopeFromInputStream(ServerUtils.java:107)
at com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall(SOAPUtils.java:97)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:166)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)


[8/19/13 11:20:00:615 EDT] 00000088 SOAPConnector 3 Caught exception
[SOAPException: faultCode=SOAP-ENV:Server.Exception; msg=ADMC0011E: The SOAP remote procedure call (RPC) cannot be unmarshalled.; targetException=
java.net.SocketTimeoutException: Read timed out]
at com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall(SOAPUtils.java:182)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:166)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)


Resolving the problem

Review the trace data (especially the TCP/IP traces) to determine if the time out is occurring due to a network delay. If you find the SOAP request/response is not being received by the DMGR in a timely manner, the client will need to work with their network administration team to resolve the latency in the network leading to the SOAP connection timeouts.


The following options are available as a temporary work around until the SOAP connectivity issue can be resolved:

a) Use the connection type of RMI and specify the bootstrap port on the addNode command. Configure the Deployment Manager and Node Agent to use the RMI connector.

    i.e. addNode <DMGR_hostName> <DMGR_bootstrapPort> -conntype RMI -profilleName <profile>

    How to configure the Deployment Manager to use the RMIConnector:
      1) Open a session to your WebSphere Applicaiton Server Integration Solutions Console in your web browser.
      2) Expand the "System administration" section.
      3) Click on the "Deployment manager" link.
      4) Under the "Additional Properties" section, click on the "Administration services" link.
      5) Change the "Remote connector" from SOAPConnector to RMIConnector.
      6) Click the OK button and then the "Save" link at the top to save the changes to the master configuration.
      7) Restart the Deployment Manager application server for the changes to take affect.
    How to configure the Node Agent to use the RMIConnector:
      1) Open a session to your WebSphere Applicaiton Server Integration Solutions Console in your web browser.
      2) Expand the "System administration" section.
      3) Click on the "Node agents" link.
      4) Click on your node agent name.
      5) Under the "Additional Properties" section, click on the "Administration services" link.
      6) Change the "Remote connector" from SOAPConnector to RMIConnector.
      7) Click the OK button and then the "Save" link at the top to save the changes to the master configuration.
      8) Restart the Node Agent application server for the changes to take affect.

b) If the issue exists on the addNode/removeNode commands, you can set the following generic JVM argument for the Deployment Manager to increase the timeout wait period for SOAP connections.

    1) Open a session to your WebSphere Applicaiton Server Integration Solutions Console in your web browser.
    2) Expand the "System administration" section.
    3) Click on the "Deployment manager" link.
    4) Under "Server Infrastructure", expand "Java and Process Management", and click on the "Process definition" link.
    5) Under "Additional Properties", click on the "Java Virtual Machine" link.
    6) Specify the following generic jvm argument in the "Generic JVM arguments" field. Each argument should be separated by a space.

    -Dcom.ibm.ws.http.ConnectionIOTimeout=60000

    NOTE: This property will increase the connection io timeout to 60 seconds from the default of 5 seconds. This value may need to be adjusted based on the amount of delay on your network. This is only a temporary work around and should not be used as a permanent solution. If you are having to increase this timeout, there is likely a bigger issue with your TCP/IP network; which is causing a delay in the SOAP requests/responses leading to the timeout of 5 seconds to be reached.

    7) Restart the application server.

c) If the SOAP connection exists with an already federated node, follow the instructions in the URL below to implement the ConnectionIOTimeOut custom HTTP Transport property to increase the connection timeout.

http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=/com.ibm.websphere.base.doc/ae/xrun_transport.html

============================================================================

For more information on Web Services client runtime errors, please refer the following URL:
http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.nd.iseries.doc/ae/rwbs_trbclientruntime.html

Related information

Common Timeouts effecting Web Services HTTP and SOAP
How to set the ConnectionIOTimeOut custom HTTP Transpor
Typical Web Services client runtime errors


Cross reference information
Segment Product Component Platform Version Edition
Operating System IBM i 7.2
Operating System IBM i 7.1
Operating System IBM i 6.1
Operating System IBM i 7.3

Document information

More support for: IBM i
WebSphere Application Server

Software version: 5.4.0, 5.4.5, 6.1.0, 6.1.1, 7.1, 7.2, 7.3

Operating system(s): IBM i

Reference #: N1019742

Modified date: 01 August 2016


Translate this page: