Websphere Application Server with SSL enabled - request fails with the following message in the http_plugin.log: Failed in r_gsk_secure-Soc_init: Peer not recognized or badly formatted message received.(gsk rc = 410)

Diagnosing The Problem

Enable the JVM custom property javax.net.debug. Here are the instructions:

Follow these steps to set up the tracing and recreate the issue.

• In the Administrative Console set the javax.net.debug system property using one of the following options, depending on where the SSL issue is occurring:
• For tracing an Application server, select the following: Servers > Server Types > WebSphere Application Servers > server_name > Expand Java and Process Management (under Server Infrastructure) - >Process definition > Java Virtual Machine > Custom properties > New...
• For tracing a Deployment Manager, select the following: System Administration > Deployment manager > Expand Java and Process Management (under Server Infrastructure) >Process definition > Java Virtual Machine > Custom properties > New...
• For tracing a Nodeagent, select the following: System Administration > Node agents > (pick a nodeagent) > Expand Java and Process Management (under Server Infrastructure) >Process definition > Java Virtual Machine > Custom properties > New...

Note: If you were not told which JVM to trace, or for some reason you are not sure which of the JVMs need this kind of tracing... set it on all of them.

Type the following:
Name: javax.net.debug 
Value: true

Note: Support may request this value be set to ssl:handshake to limit the volume of trace output.

Click Apply, and Save your changes to the master configuration.

Expand Troubleshooting > Logs and trace > server_name.

Select Diagnostic Trace. Set the Maximum Number of Historical Files to 20.

Click Apply, then select Change log detail levels.

Set the trace specification string to:
*=info : SSL=all

Click OK, then OK.

Select JVM Logs. Ensure under System.out under Installed Application Output that the
Show application print statements box is checked.

Click OK, and Save to the master configuration.

Stop the server(s) and backup/clear the logs directory for the server(s) you are tracing and the FFDC directory as well.

The trace.log will show similar to the following:

[11/10/16 11:12:48:636 NZDT] 00000067 O UOW= source=SystemOut org=IBM
prod=WebSphere component=Application Server thread=[WebContainer : 0]
WebContainer : 0, READ: TLSv1.1 Alert, length = 2
[11/10/16 11:12:48:636 NZDT] 00000017 O UOW= source=SystemOut org=IBM
prod=WebSphere component=Application Server thread=[Finalizer thread]
Finalizer thread, called closeInternal(true)
[11/10/16 11:12:48:637 NZDT] 00000067 O UOW= source=SystemOut org=IBM
prod=WebSphere component=Application Server thread=[WebContainer : 0]
WebContainer : 0, RECV TLSv1 ALERT: fatal, protocol_version

The http_plugin.log shows the following:

ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: Peer not
recognized or badly formatted message received.(gsk rc = 410)

In this example, the Websphere Application Server was trying to negotiate TLSV1 and the IBM i operating system was set to only support TLSV1.1 and TLSV1.2. The operating system is configured by the system value QSSLPCL. A value of *OPSYS means support whatever that version of OS supports. They had it set to the following:

*TLSV1.2
*TLSV1.1

This was on a V7R1 system so in this case *SSLV3 and *TLSV1 were excluded.

See the following links in regards to the SSLV3 security vulernabilities:

http://www-01.ibm.com/support/docview.wss?uid=swg21687173

http://www.ibm.com/support/docview.wss?uid=nas8N1020431

http://www.ibm.com/support/docview.wss?uid=nas8N1020451