IBM Tivoli Monitoring 6.3 Take action command fails with message "automation request rejected"

Content

The enhanced command security feature requires the Integrated Cryptographic Services Facility (ICSF) to be available on z/OS. ICSF is a standard z/OS feature. For ITM 6.3's enhanced command security, no hardware cryptographic co-processors are needed--all encryption and decryption functions can be satisfied with ICSF software facilities.

This pre-req is being remediated in ITM 6.3 Fixpack2 because not all z/OS ITM 6.3 agents will have ICSF enabled. In the meantime, workarounds have been provided below. Without the workarounds, if you attempt to execute a "Take Action" command on a z/OS ITM 6.3 Agent, you will see the following message: "Automation request rejected. User 'userid' has no security Authorization credential"

The ITM 6.3 command security feature uses encryption to verify the authenticity of the credentials. Each command request is encrypted and time-stamped inside a security token that gets passed between ITM 6.3 components. As a rule, Tivoli enterprise management servers (TEMS) encrypt security tokens, while ITM agents decrypt tokens.

A command can fail security validation on an ITM 6.3 agent for two reasons:

1. The agent's system does not have the necessary cryptographic software to decrypt the security token.

2. The system clock where the agent is running differs from the system clock where the command originated by more than 15 minutes.

All the following ITM automation commands are affected by this feature, they include:
· Take-actions
· tacmd executeCommand and tacmd executeAction
· situation-driven reflex actions
· TEMS policy-driven actions

Note that pre-6.3 agents are completely unaffected by security tokens, clock synchronization, or cryptographic availability because they do not check for a security token to be present.

Current requirements for enhanced command security:

1. To successfully encrypt and decrypt command security tokens, an ITM 6.3 process must have access to local cryptographic software. On distributed platforms, GSKit provides this support. For a z/OS TEMS or z/OS Agent, ICSF must be configured and available. In addition, encryption key member KAES256 must be in the RKANPARU dataset for all z/OS TEMS and agents. The ICSF and encryption key configuration requirements were not documented as an ITM 6.3 prerequisite, and therefore it’s possible some z/OS OMEGAMON customers will be missing one or both of the requirements. Furthermore, in the configuration of an agent-only run-time environment (RTE), there is no provision in the configuration tool to generate the KAES256 RKANPARU member. Consequently, a manual copy step must be performed to store the KAES256 member in the agent's RKANPARU.

2. The ITM 6.3 command security feature requires that the sending and the receiving system clocks must be synchronized with a delta of less than 15 minutes. This 15 minute restriction does accommodate timezone differential. The restriction only applies to system clocks whose Coordinated Universal Time (UTC) values are off by more than 15 minutes, it does not apply to systems whose local times are different. If the time delta exceeds 15 minutes, a command request will be marked as expired when it reaches the destination, and it will fail to execute.


Pre-ITM 6.3 FP1 z/OS Workaround for the 1^st requirement:

Before ITM 6.3 FP1 is available, here are the three z/OS cryptographic conditions must be met to satisfy the 1st requirement:

1. The Integrated Cryptographic Services Facility (ICSF) started task must be executing.
2. The ICSF load library must be in the RKANMODL DD of the TEMS and agent's started task JCL.
3. The &rhilev.&rte.RKANPARU dataset of the TEMS and agent must have a valid KAES256 member.

If any of the three conditions are not met, the z/OS ITM 6.3 TEMS or agent will not be able to encrypt and decrypt security tokens, and command requests will fail.


If you use the ITM configuration tool to enable ICSF in your z/OS TEMS, a KAES256 PDS member is automatically created for you. This PDS member equates to the kaes256.ser file in a distributed ITM installation. There are two ways to enable the enhance command security support in an agent only RTE configuration:

1. If you have a distributed ITM environment, you can FTP the kaes256.ser file to your agent's &rhilev.&rte.RKANPARU(KAES256)
2. You can copy the KAES256 member from the RKANPARU of your ITM 6.3 z/OS TEMS to the RKANPARU of your z/OS agent.

Pre-ITM 6.3 FP1 z/OS and Distributed Workaround for the 2^nd requirement:

You do not need to perform this workaround for the 2nd requirement if your system clocks are synchronized within 15 minutes between your TEMS and the receiving destination.

If the time delta exceeds 15 minutes, you have a couple of options. The first option is to fix the system clock at the agent and reboot your agent machine. The second option is to add the following parms at the TEMS the agent is connected to and recycle the TEMS:

KGE_EXPIRE_INTERVAL_ENTITY=nnnn
KGE_EXPIRE_INTERVAL_PERMIT=nnnn


nnnn is a value in minutes of the system clock differential between your TEMS and the receiving agent destination. If, for example, the agent's system clock was 6 hours different from the TEMS system clock, you would add these two environment variables to your TEMS and set the nnnn values to a number greater than 360 (60 * 6), the nnnnvalues in both parms should be the same.