IBM Support

MSGSSL002 error received when trying to connect with ACS using SSL

Troubleshooting


Problem

Error message MSGSSL002 -"IBMi server application is not trusted for secure socket connection."   or  "System not trusted for SSL connection." is recieved when trying to connect with Access Client Solutions with SSL enabled. We often see this error when trying to connect via Telnet with SSL enabled and also when trying to connect to ACS 5250 Console on older releases behind on PTF's. 

Symptom

  Connection fails causing user to be unable to connect. MSGSSL002 error is posted by Access Client Solutions.                                       

Cause

The cause of this error is due to the current Certificate Authority or Server Certificate assigned to the IBM i server application (Console or Telnet and Host servers) being used, does not meet current restrictions set by Java. Java does not allow SSL connections if the certificate is using older specifications that do not meet updated requirements.

Environment

Access Client Solutions 1.1.8 with Java 8. Error may occur on all current releases of IBM i operating system.

Diagnosing The Problem

To confirm the exact cause, use the following documentation to gather a TRCCNN of a recreate. Access Client Solutions service logs will also help. IBM i Support can use these to locate the exact cause.
1.  Instructions for gathering TRCCNN: http://www-01.ibm.com/support/docview.wss?uid=nas8N1016231
2. Along with the TRCCNN, generate and package the ACS service logs with the following steps:
- After the error prompts, select the option to Generate Service logs. Click OK through the Prompts.  
- Navigate to the Access Client Solutions main interface, select Package Service Logs from the             Tools menu. This may take a minute.
- Wait for the MSGGEN002- Function completed successfully and take the option to "Open Target Directory"
- Submit the resulting .zip folder along with the TRCCNN QSYSPRT.txt file.

The ACS service logs will show errors similar to the following:

-Caused by: class java.security.cert.CertPathValidatorException: Algorithm constraints check failed on keysize limits. RSA 512bit key used with certificate:

- Caused by: Class ava.security.cert.CertPathValidatorException: algorithm constraints check failed on signature algorithm: SHA1withRSA.

Resolving The Problem

1. ACS 5250 Console

To correctly resolve this issue when using ACS 5250 Console, the following PTF's need to be applied if not already on your system. Console does use SSL when making its connection and the certificate being used by the system is not meeting updated Java requirements. Applying these PTF's will correct the certificate being used by console. If you are unable to apply the PTF in a timely manor, you can choose to use the circumvention steps below to modify the java.securities file.

o  V6R1M1 -- PTF is MF60292 (there are no TRs in that release)
o  V7R1M0 -- PTF is MF60291 (requires MF99010)
o  V7R2M0 -- PTF is MF60290 (requires MF99102)

2. ACS 5250 Telnet connection

To correctly resolve this issue for SSL Telnet connections, the Local Certificate Authority or the Server Certificate will need to be renewed using a higher cipher spec within Digital Certificate Manager. This process is intrusive and may require reboot of the Telnet and Host servers. If the certificate cannot be renewed in a timely manor you can modify the pc's java.security file to temporarily allow for low level ciphers.

The circumvention is to modify the JRE's java.security file and comment out two lines.  Doing this affects anything that uses the JRE on the PC and exposes you to sites that use the older ciphers where data could be compromised.

The java.security file will be located in the path of your JRE installation.  The path will be under the base directory structure Oracle creates; however, when the JRE is installed, the user could specify a different location.  

64-bit PCs:
Typically, the path is c:\Program Files\Java\<JRE version>\Lib\security

If you are using a JDK with the JRE, the path is slightly different:
c:\Program Files\Java\<JDK version>\JRE\Lib\security

The above would be for 64-bit versions of Java; 32-bit versions of Java would be in the Program Files (x86) directory.

Note: 32-bit PCs have only the Program Files directory.

The file to edit for a circumvention is the java.security file.  At the time of this writing, Java 8 has the following two lines needing to be remove in order to temporarily connect with older specs:

o  jdk.certpath.disabledAlgorithms: MD5, SSLv3, DSA, RSA keySize <2048
o  jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DHkeySize < 768 , \EC keySize <224

Document information

More support for: IBM i Access Client Solutions

Software version: All Versions

Operating system(s): IBM i

Reference #: 0720287

Modified date: 08 August 2018