MustGather: Documentation for diagnosing SSL handshake failures on WebSphere Application Server for z/OS

The initial documentation consists of:
1. RACF output
2. Complete Job output for Control/Servant/Adjunct region
3. Collector jar for DeploymentManager and Application Server Nodes

In the sections below the control/servant region refer to the DeploymentManager, Node Agent, or Application Server address space where the SSL error is occurring.

1. RACF output

a) Confirmation that Control/Servant/Adjunct region userid or WebSphere administrative group (ie. WSCFG1) is permitted to FACILTY class IRR.DIGTCERT.LIST and IRR.DIGTCERT.LISTRING

RACF commands:
RLIST FACILITY IRR.DIGTCERT.LIST ALL
RLIST FACILITY IRR.DIGTCERT.LISTRING ALL

If RDATALIB class is being used instead of IRR.DIGTCERT.* class

RLIST RDATALIB CR_USERID.WASKeyring .LST ALL
RLIST RDATALIB SR_USERID.WASKeyring .LST ALL
RLIST RDATALIB ADJUNCT_USERID.WASKeyring .LST ALL

Change the name of CR_USERID to reflect the Control region userid
Change the name of SR_USERID to reflect the Servant region userid
Change the name of ADJUNCT_USERID to reflect the Adjunct region userid if there is an Adjunct region running.

Change the name WASKeyring to reflect the name of the keyring used in the WebSphere Application Server.

b) A listing of all certificates on the Control region and Servant region userid keyring

RACF commands:

RACDCERT LISTRING(WASkeyring) ID(CR_USERID)
RACDCERT LISTRING(WASkeyring) ID(SR_USERID)
RACDCERT LISTRING(WASkeyring) ID(ADJUNCT_USERID)

Change the name of CR_USERID to reflect the Control region userid
Change the name of SR_USERID to reflect the Servant region userid
Change the name of ADJUNCT_USERID to reflect the Adjunct region userid if there is an Ajdunct region running.

Change the name WASKeyring to reflect the name of the keyring used in the WebSphere Application Server.
If the keyring name is unknown at the time of problem occurrence the WASKeyring in the SAF command can be substituted with an asterisk (*) to get a list of all keyrings for that address space userid.

For example:
RACDCERT LISTRING(*) ID(USERID)


c) Details of the signer certificates (certificates with USAGE CERTAUTH) on the keyring

RACF command:
RACDCERT CERTAUTH LIST(LABEL('CERTAUTH_LABEL'))

The CERTAUTH_LABEL is obtained from the RACF LISTRING output in #1b.

This information is needed to
- confirm the certificate has a status of TRUST
- confirm certificates are not expired
- the private key type (NONE, NON-ICSF or ICSF)
- confirm the certificate chain for all root signer and intermediate signing certificates using issuer/subject DN information.


d) Details of Personal certificates (certificates with USAGE PERSONAL) on the keyring

RACF commands:
RACDCERT LIST ID(CR_USERID)
RACDCERT LIST ID(SR_USERID)
RACDCERT LIST ID(ADJUNCT_USERID)

This information is needed to confirm
- the certificate has a status of TRUST
- certificates are not expired
- the private key type (NONE, NON-ICSF or ICSF)
- the certificate chain for all root signer and intermediate signing certificates using issuer/subject DN information.



2. Complete Job output of Control/Servant/Adjunct region.

Job output should include (JESMSGLG, JESJCL, JESYSMSG, SYSOUT, and SYSPRINT) for the control region, servant region, as well as adjunct region (if there is an adjunct region).

The output can be saved to a dataset easily by typing "XDC" next to the address space in SDSF.

JESMGSGLG, JESYSMSG are of importance as this indicates the userid the address space is running under.
SYSPRINT is needed as it will usually contain the error or exception related to the SSL failure.

Note: If the logs are large, please terse


3. A collector jar containing the security.xml file and any FFDC logs.

Please issue these two USS commands in a directory outside of your WebSphere config directory (ie. /tmp).

export IBM_JAVA_OPTIONS=-Dfile.encoding=ISO8859-1

WAS_HOME/DeploymentManager/profiles/default/bin/collector.sh
and
WAS_HOME/AppServer/profiles/default/bin/collector.sh

This will create a jar labeled
hostname.cellname.nodename.default-WASenv.jar
in the directory where the command was issued.

Obtain the entire collector jar as it may be necessary to look at other xml files
(ie. endpoints, ports, or application descriptor files)

Send the following items to support when opening the PMR:

- RACF output from section 1a-d
- Complete Job output for Control/Servant/Adjunct region (section 2)
- hostname.cellname.DMGRnodename.default-WASenv.jar in binary not tersed
- hostname.cellname.AppSrvnodename.default-WASenv.jar in binary not tersed