Problem Determination

The Problem Determination topics explain how to address problems encountered in a Security Key Lifecycle Manager for z/OS environment.

You can enable debugging for an individual component, multiple components, or all components of the Security Key Lifecycle Manager for z/OS.

There are three places to look for errors:

The Security Key Lifecycle Manager for z/OS audit log: Most error messages appear in the audit log. The location and file name are set in the Security Key Lifecycle Manager for z/OS ISKLMConfig.properties.zos file in the Audit.handler.file.directory and Audit.handler.file.name properties.
Standard Error (stderr): When running the Security Key Lifecycle Manager for z/OS as a started task using the Security Key Lifecycle Manager for z/OS console wrappers, examine the Execution LOG for errors. When running the Security Key Lifecycle Manager for z/OS in the foreground using USS/OMVS, these errors appear where you have directed STDERR.
The Security Key Lifecycle Manager for z/OS debug log: The location is set in the Security Key Lifecycle Manager for z/OS ISKLMConfig.properties.zos file debug.output.file property. The data written to the file is controlled by the debug property. For space reasons, it is best that you initially set the property to debug = none. If an error is encountered while Security Key Lifecycle Manager for z/OS is running you can turn debug on. You can turn debug on by submitting the modconfig –set –property debug –value all command. If you run into a problem and did not get any debug information from the Security Key Lifecycle Manager for z/OS audit log or Standard Error, set debug=all.
Note: The debug log should only be turned on at the direction of IBM service while debugging a specific problem and must only be turned on for a limited time. The debug log captures large amounts of data which might fill up the file system and cause an outage.

This is a list of errors and their possible causes that you might see when running the Security Key Lifecycle Manager for z/OS:

Error 1

com.ibm.keymanager.j [Caused by java.security.PrivilegedActionException: java.io.IOException: The private key of ISLKMSERVE is not a software or icsf key. Error creating key entry because private key is not available.]

Possible causes: If you are using a RACF® keystore type (JCECCARACFKS or JCERACFKS):

This error can occur if the user ID running the Security Key Lifecycle Manager for z/OS is not the owner of the KeyRing/Private key. RACF only allows a private key to be retrieved by its owner.
This error can occur when starting the Security Key Lifecycle Manager for z/OS. This error occurs if your keyring has a public key that does not contain a corresponding private key, such as a business partners key and that key was not connected as CERTAUTH (see directions in Business Partner and Remote z/OS Systems).

Error 2

Runtime event:[
  timestamp=Wed Sep 06 13:30:54 EDT 2006
  event source=com.ibm.keymanager.g.fb
  outcome=[result=unsuccessful]
  event type=SECURITY_RUNTIME
  message= ***Error: Information not available for protected private keys.. 
     ErrorCode=0xEE0F
  resource=[name= Drive Serial Number: 000001350808 WWN: 500507630F04BC1B 
     Key Alias/Label[0]: Tape_Sol_Tst_Shr_Pvt_1024_Lbl_02;type=file]
  action=stop

Possible cause: This error can occur if unrestricted policy files were not installed. Refer to Copying the unrestricted policy files. This error usually appears in the Security Key Lifecycle Manager for z/OS audit log.

Note: This error can also occur with an EE31 error code and the same text. It can be resolved by installing the unrestricted policy files.

Error 3

# java.lang.NoClassDefFoundError: javax/crypto/b
        at javax.crypto.Cipher.a(Unknown Source)
        at javax.crypto.Cipher.getInstance(Unknown Source)
        at com.ibm.keymanager.g.b.a(b.java:189)
        at com.ibm.keymanager.g.fb.a(fb.java:937)
        at com.ibm.keymanager.g.fb.run(fb.java:1277)

Possible cause: The wrong version, or a corrupt copy, of unrestricted policy files was installed. This error is sent to STDERR (your job execution log) and not the Security Key Lifecycle Manager for z/OS audit log.

Error 4

***Error: no such provider: IBMJCE4758. ErrorCode=0xEE0F
Runtime event:[
  timestamp=Mon Sep 18 22:43:26 EDT 2006
  event source=com.ibm.keymanager.logic.MessageProcessor
  outcome=[result=unsuccessful]
  event type=SECURITY_RUNTIME
  message= ***Error: no such provider: IBMJCE4758. ErrorCode=0xEE0F
  resource=[name= Drive Serial Number: 000001350699 WWN: 500507630F0C851C;type=file]
  action=stop
  ]

Possible cause: The Java hardware provider has not been added to the java.security provider list. This action must be done each time there is a new Java installation/upgrade if you are planning to use ICSF hardware keys. See Add the Java Hardware Provider (Only if Using ICSF).

Error 5

java.security.PrivilegedActionException: java.io.IOException: R_datalib (IRRSDL00) error: error while getting certificate or trust info (8, 8, 80)

Possible cause: Quotation marks surround the keyring name specified in the ISKLMConfig.properties.zos file (for example, config.keystore.file = safkeyring:"//ISKLMSRV/ISKLMRing"). Remove the quotation marks.

Error 6

java.security.PrivilegedActionException: java.io.IOException: Failed validating certificate paths
        at java.security.AccessController.doPrivileged1(Native Method)
        at java.security.AccessController.doPrivileged(AccessController.java:351)
        at com.ibm.keymanager.b.a.a(a.java:23)
        at com.ibm.keymanager.b.a.a(a.java:148)
        at com.ibm.keymanager.b.a.b(a.java:138)
        at com.ibm.keymanager.i.a.a.h(a.java:711)
        at com.ibm.keymanager.i.a.a.c(a.java:595)
        at com.ibm.keymanager.KMSAdminCmd.main(KMSAdminCmd.java:2)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)

Possible cause: A CA Certificate is not connected to the KeyRing (note: At least one CERTAUTH cert is required, even if all certificates are self-signed). This message can only be displayed in the debug log and occur when attempting to start the Security Key Lifecycle Manager for z/OS Server.

Error 7

java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: com/ibm/keymanager/logic/EncryptionCBDQuery     
:at com.ibm.keymanager.logic.RequestEEDKs.createMsg(RequestEEDKs.java:48)       
:at com.ibm.keymanager.logic.RequestEEDKs.<init>(RequestEEDKs.java:39)          
:at com.ibm.keymanager.logic.MessageProcessor.ProcessMessage(MessageProcessor.java:351)
:at com.ibm.keymanager.logic.MessageProcessor.run(MessageProcessor.java(Compiled Code))

Possible cause: Java is not available. Possibly the file system where Java is installed has been dismounted. If using in-band key management, you can also see this IOS error:

IOS628E ENCRYPTION ON DEVICE E0A4 HAS FAILED DUE TO COMMUNICATION TIME OUT   
IOS000I E0A4,D6,IOE,01,0E00,,**,A0209A,ITSXZ071 948                    
804C08C022402751 0301FF0000000000 0000000000000092 2004E82062612111    
ENCRYPTION FAILURE                                                     
CU = 03 DRIVE = 000000 ISKLM = 000000

Error 8

JVMJZBL2999T JvmExitHook entered with exitCode=-3,  
javaMainReturnedOrThrewExcep  
JVMJZBL1043N The Java virtual machine completed with  
System.exit(-3)

ISKLM server is now terminating abnormally with a return code of 4093.

Possible cause: The Java version (or just the Encryption Key Manager JAR version) was replaced such that the Encryption Key Manager was upgraded from a build earlier than 20070503 to a build equal to or later than 20070503. If that is the case, you must define the Audit.metadata.file.name property in the ISKLMConfig.properties.zos file. This is the name of the XML file where metadata is saved. This property is required to start versions of Encryption Key Manager with build date 20070503 (when metadata support was added) and later. See Using Metadata. Check your current Encryption Key Manager version Before you decide to upgrade to the latest Encryption Key Manager.

Error 9

Hardware error from call CSNDSYI 
java.lang.IllegalArgumentException: System Error: Key 
unwrapping is not supported in AMODE(64).. ErrorCode=0xEE31
resource=[name= Drive Serial Number: ds8k_device1 WWN: 
57574E414D453030 Key Alias/Label[0]: cert1;type=file]

Possible cause: This error can occur when using Java 6.0 for 64-bit SDK with the ICSF level not updated to HCR7770 and the requiredHardwareProtectionForSymmetricKeys is set to true. This error can occur when a JCECCAKS keystore type is used.

This error can be resolved by updating your ICSF version to HCR7770.

ICSF Hardware Error

Under some circumstances, an error may occur when Security Key Lifecycle Manager for z/OS tries to access ICSF hardware crypto functionality. Security Key Lifecycle Manager for z/OS captures the hardware error message from ICSF, writes the message to audit log, and closes the socket connection with tape drive or storage device. The error code and return code in the error message are in decimal values. A sample message is:Hardware error from call CSNBRNG returnCode 12reasonCode 0.

For more information, see, http://publib.boulder.ibm.com/infocenter/zos/v1r10/index.jsp?topic=/com.ibm.zos.r10.csfb400/rcrcdes.htm. It documents the error code and return code in both hexadecimal and decimal forms.