Troubleshooting the “SSL connection failed” message since DSM 2.1.4

Resolving The Problem

The key database (.kdb file) generated by GSkit uses SHA1WithRSA signature algorithm by default. Therefore, you have to use a stronger signature algorithm to generate the certificate and key database for the cipher-suites that are used in DSM.  
 
Update the certificate

The following procedure shows how to generate and update a DB2 server certificate with a more secure algorithm.

 
1.   Log in to the DB2 server as the instance owner.
2.   Create the key database by using a strong signature algorithm. For example, SHA256.
            sqllib/gskit/bin/gsk8capicmd_64 -cert -create -db "myfooserver.kdb"  -sigalg SHA256 -label "testselfsigned" -dn "CN=db2svr01.test.dsm.com" 
3.   Stash the password into the myfooserver.sth file
           sqllib/gskit/bin/gsk8capicmd_64 -keydb -stashpw -db "myfooserver.kdb"
4.   Extract the certificate file for the DSM server
            sqllib/gskit/bin/gsk8capicmd_64 -cert -extract -db "myfooserver.kdb"  -label "testselfsigned" -target "myfooserver.arm" -format ascii
5.   Update the instance configuration with the following SSL info. And make sure the SSL_VERSIONS is configured to TLSV12, which means only TLSV12 cipher-suites will be used.
        update dbm cfg using SSL_SVR_KEYDB <location of the kdb file in db2 server>
        update dbm cfg using SSL_SVR_STASH <location of the stash file in db2 server>
        update dbm cfg using SSL_SVR_LABEL <the label of the certificate file>
        update dbm cfg using SSL_SVCNAME <port of the ssl connection>
        update dbm cfg using SSL_VERSIONS  TLSV12
      The updated instance configuration looks like below after the changes.
             SSL server keydb file                  (SSL_SVR_KEYDB) = /home/db2inst1/myfooserver.kdb
             SSL server stash file                  (SSL_SVR_STASH) = /home/db2inst1/myfooserver.sth
             SSL server certificate label           (SSL_SVR_LABEL) = testselfsigned
             SSL service name                        (SSL_SVCENAME) = 50002
             SSL versions                            (SSL_VERSIONS) = TLSV12
            
6.   Copy the certificate file, myfooserver.arm, to the DSM server machine.
7.   Run the keytool utility to import the certificate to the truststore. The keytool utility can be found in the DSM JRE.
            /ibm-datasrvrmgr/java/jre/bin/keytool -import -file ./myfooserver.arm -keystore mydsmclientkeystore.jks
8.   Update the connection profile to specific the new jks file location by using the "Edit" connection button of the Setting->Managing connections page on the DSM UI.
 
Re-enable the SHA1 algorithm
 
If you accept the risk of using the weak SHA1 algorithm cipher-suites and do not want to change the certificate file. You can also enable the SHA1 support on the DSM. 
1.   Open the dswebserver_override.properties file under the Config folder of the DSM installation directory.
2.   Remove the SHA1 value for the dsweb_tls_disabledAlgorithms key.