Product Documentation
Abstract
Flash 1631465 describes new command security restrictions that were introduced in ITM 6.3. The Flash states that ITM 6.3 Fixpack1 will provide remediation measures to help you migrate to the enhanced command security. This Flash is the documentation to provide the additional details.
Content
Background
Starting with ITM 6.3 all automation commands, such as Take Action, situation-driven reflex actions, tacmd executeCommand, and Policy Workflow commands, operate in a more secure environment. Specifically, ITM 6.3 will now:
(1) Capture detailed Audit information about every ITM command request.
(2) Reject any command that comes from an untrusted source.
The mechanism for rejecting untrusted commands relies on an encrypted security token that accompanies each command request as it gets passed between ITM components. The encryption requirement means that all ITM servers and agents must have access to local cryptographic functions. On distributed platforms, GSKit provides the cryptographic support. On z/OS, the Integrated Cryptographic Services Facility (ICSF) must be available.
Note: Any agents running with a pre-ITM 6.3 framework are not affected and no encryption or auditing is enforced. This technote only applies to agents whose ITM infrastructure is v6.3 or higher.
ITM 6.3.0 Behavior
Beginning with v6.3, all ITM components on all platforms (including z/OS) with a v6.3 framework require an encrypted security token when handling automation commands. The enforcement occurs at the agent.
Security tokens are generated at the source of the command (See Table 1. for information about sources). Consequently, all ITM 6.3 infrastructure components must understand how to generate these tokens. To support pre-ITM 6.3 components (TEMS, TOM*, TEPS), a “TEMS Security Compatibility mode” is available at the TEMS and RTEMS to generate a default token on their behalf.
Table 1. ITM Automation Command Sources
| Command Source | Source Tivoli Monitoring Component for Encryption |
| Tacmd executeCommand | HUB Tivoli Enterprise Monitoring Server |
| Tacmd executeAction | HUB Tivoli Enterprise Monitoring Server |
| TEP Client – Take Action | Tivoli Enterprise Portal Server |
| Tivoli Omegamon Manager | HUB Tivoli Enterprise Monitoring Server* |
| Situation with reflex take action | The HUB Tivoli Enterprise Monitoring Server or Remote Tivoli Enterprise Monitoring Server that the agent is connected to. |
| Workflow policy with situation | The HUB Tivoli Enterprise Monitoring Server or Remote Tivoli Enterprise Monitoring Server that the agent is connected to. |
ITM 6.3 Fixpack1 Changes
The Integrated Cryptographic Services Facility (ICSF) must be available on z/OS to encrypt and decrypt command security tokens. The main reason for the command security changes in ITM 6.3 FP1 is to give z/OS customers more time to configure ICSF.
The ITM 6.3 FP1 changes depend on a TEMS environment variable, KMS_SECURITY_COMPATIBILITY_MODE. It has two possible values, Y or N. Y enables compatibility mode, which offers more lenient command security than N. The value of this environment variable is specified either in a TEMS configuration command line prompt, or using the “TEMS Security Compatibility Mode” checkbox, as shown here:
The box is checked by default, which corresponds to the KMS_SECURITY_COMPATIBILITY_MODE=Y value. Leaving the box checked does the following three things:
(1) It allows commands to still execute even if cryptographic facilities are not available at the TEMS or the agent. This gives you time to ensure that your z/OS TEMS has ICSF available.
(2) It supports backwards compatibility when commands originate from a pre-6.3 source. For example, let's say you have a 6.23 TEPS connected to a 6.3 FP1 HUB. If a TEP client issues a Take Action command, the 6.23 TEPS will not create an encrypted security token because that capability was not added until ITM 6.3. In that scenario, the 6.3 FP1 HUB will automatically generate a default security token to still allow the Take Action command to execute at the agent endpoint.
(3) It allows commands to execute at an agent even if the agent's system clock is many hours ahead of the system clock at the connected TEMS.
Disabling Security Compatibility Mode
When you are ready to implement stricter command security, you should reconfigure your TEMS to disable security compatibility mode, which will change the value of KMS_SECURITY_COMPATIBILITY_MODE from Y to N. Reconfiguration can be accomplished either by unchecking the “TEMS Security Compatibility Mode” box in the GUI, or by responding to the prompt during command line configuration and setting the option to disable TEMS Security Compatibility Mode.
Before making this change, you should be aware that:
(1) All commands must have encrypted security tokens, and the agents must have cryptographic software to decrypt the tokens.
(2) System clocks must be synchronized between agents and servers within a range of no more than 75 minutes.
Note that this 75 minute restriction does accommodate timezone differential. The restriction only applies to system clocks whose Coordinated Universal Time (UTC) values are off by more than 75 minutes; it does not apply to systems whose local times are different. If the time delta exceeds 75 minutes, a command request will be marked as expired when it reaches the destination, and it will fail to execute.
On z/OS systems, you must ensure that:
(1) The ICSF started task is running.
(2) The ICSF SCSFMOD0 load library is concatenated to the RKANMODL DD statement in the started task JCL for the z/OS TEMS and z/OS Agent. Note that ICAT and PARMGEN automatically insert the SCSFMOD0 library if ICSF enablement is requested during z/OS TEMS configuration.
(3) The KAES256 member, containing the encrypted ITM private key, is present in the RKANPARU dataset.
If you leave the default KMS_SECURITY_COMPATIBILITY_MODE=Y setting, the three z/OS configuration steps listed above are not needed. That's because the 6.3 FP1 command security feature will detect at startup if the z/OS TEMS or z/OS Agent does not have local cryptographic support and, if so, will automatically disable the checking for encrypted command tokens.
After restarting your TEMS with KMS_SECURITY_COMPATIBILITY_MODE=N you will see a TEMS Audit message that looks like this:
Also, the following message will be written to the TEMS RAS1 log file:
*** Identify Manager security compatibility operation is disabled.
Troubleshooting Command Security Problems
If you believe that an ITM command failed to execute because of a security-related problem, your chief sources of information are:
(1) The Audit Log and RAS1 log of the Agent where the command failed.
(2) The RAS1 log of the TEMS to which the Agent is connected because the TEMS is normally the source of the command security token.
The two most common Agent Audit messages indicating that a security problem prevented command execution are:
KGEA0002 “Process <process_name> Identity Manager creating session token authentication signature failed with status <status_code>“
KGEA0003 “Process <process_name> Identity Manager session token authentication signature verification failed with status <status_code>“
“KGE” is the component that implements command security in ITM. KGE is often referred to as the “Identity Manager” because its purpose is to capture the identity of each command invoker and to pass that information through the ITM infrastructure.
Here are some possible troubleshooting scenarios to be aware of:
Scenario 1: Mismatched keys between Agent and Server
If the key label (for example, “IBM_Tivoli_Monitoring_Certificate”) that was used to generate a private key at the TEMS does not match the label that was used at the Agent, the mismatched keys will cause security token encryption and decryption to fail. If you were to distribute a Take Action command to that Agent, the command would fail with an error popup such as this:
You will also see a KGEA0003 message in the Agent's Audit log. Note that these errors will occur even with the default KMS_SECURITY_COMPATIBILITY_MODE=Y. The ITM command security feature requires that all servers and agents that process automation commands must share a common private key.
Scenario 2: ICSF not available at z/OS TEMS or z/OS Agent
If ICSF is not available at the z/OS TEMS or z/OS Agent, for example, because there is no CSF.SCSFMOD0 load library concatenated to the RKANMODL DD statement, you will see the following RKLVLOG messages at startup time indicating that the ICSF functions could not be loaded:
"initializeICC") Cannot get CSNBXAE function pointer
"initializeICC") Cannot get CSNBXEA function pointer
"initializeICC") Cannot get CSNBSYE function pointer
"initializeICC") Cannot get CSNBSYD function pointer
"initializeICC") Cannot get CSFPSKE function pointer
"initializeICC") Cannot get CSFPSKD function pointer
"initializeICC") Cannot get CSFPPRF function pointer
"initializeICC") Cannot get CSNBOWH function pointer
"initializeICC") Cannot get CSFPGSK function pointer
"initializeICC") Cannot get CSFPTRD function pointer
"initializeICC") Cannot get CSNBSMG function pointer
"initializeICC") Cannot get CSNBSMV function pointer
"initializeICC") Cannot get CSFPTRC function pointer
"CRY_RANDX") Function initialization failed with error code 44
"generateTransactionID") Random number generation failed
"registerEntity") ***KGE Unable to add Entity Type USER Name sysadmin Data to Identity Manager registry - cannot generate unique Transaction ID.
"KGE_IM_UnitTest") *KGEVLOG Verification RC = -2
"runInstallVerify") *KGE-INFO: Installation Verification Failed - Status -2.
But note that with the default KMS_SECURITY_COMPATIBILITY_MODE=Y in effect, these startup error messages will not prevent automation commands from succeeding.
To eliminate the error messages in the RKLVLOG, you must ensure that
(1) The ICSF started task is running, and
(2) The CSF.SCSFMOD0 library has been concatenated to RKANMODL
By contrast, when there is a valid local cryptographic environment, you will see these messages near the top of the RKLVLOG or RAS1 log:
"KGE_IM_UnitTest") *KGEVLOG Verification RC = 0
"runInstallVerify") *KGE-INFO: Installation Verification Successful.
Scenario 3: Missing KAES256 member at z/OS TEMS or z/OS Agent
If the KAES256 member of RKANPARU is missing, which means there is no private key member to support encryption and decryption of command security tokens, you will see this Audit message:
KGEA0019 “Process <process_name> Application Encryption Verification Procedure failed - Status -6 key file unavailable! Authorization Verification and Extended Auditing disabled.”
Again, with the default KMS_SECURITY_COMPATIBILITY_MODE=Y in effect, a missing KAES256 member will not cause automation commands to fail. To create a valid KAES256 member, you can do the following:
(1) If you use the ITM configuration tool, you should enable ICSF in your z/OS TEMS, which will cause a KAES256 PDS member to be automatically created for you. This PDS member equates to the kaes256.ser file in a distributed ITM installation.
(2) In an Agent-only RTE, there is currently no provision in the configuration tool for enabling ICSF. There are two ways to handle this:
(a) If you have a distributed ITM environment, you can FTP the kaes256.ser file to your Agent's &rhilev.&rte.RKANPARU(KAES256).
(b) You can copy the KAES256 member from the RKANPARU of a z/OS TEMS to the RKANPARU of your z/OS Agent.
Related Information
[{"Product":{"code":"SSTFXA","label":"Tivoli Monitoring"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"--","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.3","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
20 April 2020
UID
swg27038508