Resolving Secure Sockets Layer errors

Secure Sockets Layer (SSL) errors can be attributed to an incorrect environment setup, a bad server certificate, connection problems, out-of-sync conditions, or other causes.

Use the following guidance to resolve common SSL client-to-server and server-to-server problems:
Not connecting to the server after using a vendor-acquired certificate authority (CA) certificate
If you are using a vendor-acquired certificate and it was not added to the server, specify the root certificate as trusted in the server key database. To add the root certificate to the database, issue this command:
gsk8capicmd -cert -add -db cert.kdb -pw password
-label name -file .der_file -format ascii
The CA root certificate was not added to the client
Add the root certificate as trusted into the client key database:
gsk8capicmd -cert -add -db dsmcert.kdb -pw password
-label my CA -file ca.arm -format ascii 
Unable to run gsk8capicmd.exe (IBM Global Security Kit [GSKit])
In most cases, this Windows error is generated by an incorrect environment setup. See the IBM Tivoli Storage Manager Administrator's Guide and set up the PATH variable as directed before you run the gsk8capicmd utility.
ANS1595E Bad server certificate
This error is reported when the server certificate is not known to the client or server. The "bad server certificate" error can occur under these conditions:
  • The certificate was never imported
  • The cert256.arm certificate file was corrupted before the certificate was imported
  • The command to import the certificate was entered incorrectly
  • The DSM_DIR variable points to the wrong directory, which contains an incorrect client key database (dsmcert.kdb)
  • The server is set up for Transport Layer Security (TLS) 1.2 but the client is not at a sufficient level (6.3 is required).
  • The server is set up for TLS 1.2 but the client imported the cert.arm file instead of the cert256.arm file.
  • The server is set up for TLS 1.2 but the client imported the cert256.arm file instead of the cert.arm file.
Repeat all the steps necessary for importing the server certificate and check the DSM_DIR variable. For more information about the failure, see the dsmerror.log file. The client error log might also contain information about specific IBM GSKit failure.
ANS1592E Failed to initialize SSL protocol
This error occurs on the client and indicates that the SSL connection was not established. For more information about the failure, see the client error log. The server does not accept SSL sessions on the port to which the client or server is trying to connect. Determine whether the client or server points to the correct server port (TCPPort), which can be a port number that is different from the default 1500.
ANR8583E and GSKit return code 406
This error might indicate that a non-SSL-enabled client is trying to contact an SSL port. When a client contacts a Tivoli® Storage Manager server at a port that is defined by SSLTCPPORT or SSLTCPADMINPORT, the server establishes a session and initiates an SSL "handshake." If the client is not SSL-enabled, it cannot complete the SSL handshake process. The session then seems to stop, but times out through the server IDLEWAIT option or end when the server administrator issues the CANCEL SESSION command to manually cancel it. The example illustrates a session in this state, from the server:
   
TSM:SERVER1>query session
ANR2017I Administrator SERVER_CONSOLE issued command: QUERY SESSION

  Sess Comm.  Sess     Wait   Bytes   Bytes Sess  Platform Client Name
Number Method State    Time    Sent   Recvd Type
------ ------ ------ ------ ------- ------- ----- -------- -------------
     1 SSL    IdleW   17 S        0       0 Node
Important: Because the computing environment might cause a valid handshake process to take some time, do not assume that the result always indicates a non-SSL client.
ANR8583E and GSKit return code 420, and ANR8581E with GSKit return code 406 occur for the same Tivoli Storage Manager client session
When server messages ANR8583E and ANR8581E occur for the same client session, it is likely that the client generated an ANS1595E message. Message ANS1595E typically occurs while Tivoli Storage Manager attempts to establish a session with the server. If true, follow the guidance in the Tivoli Storage Manager message manual for ANS1595E to eliminate these errors.
ANR3338E TLS is at an earlier level than 1.2
This error is reported when the server and the storage agent attempt to connect with an SSL protocol earlier than TLS 1.2. For server and storage agent communication, if the SSLDISABLELEGACYTLS option is specified, TLS sessions must connect at a minimum level of TLS 1.2 or the session is rejected.
Cross-defining servers without SSL=YES causes a server hang
If you plan to use SSL communication, the SSL infrastructure must be in place on the source and target replication servers. Required SSL certificates must be in the key database file that belongs to each server. The SSL function is active if the server options file contains the SSLTCPPORT or SSLTCPADMINPORT option or if a server is defined with SSL=YES at startup.

An entry occurs when a vendor-acquired certificate in use was not added to the server, or the CA certificate was not added to the client. When an SSL session is started, the session startup message includes the serial number from the server certificate. Therefore, the certificate that is being used can be uniquely identified.