Tivoli Enterprise Monitoring Server (TEMS) unexpected shutdown at startup

Due to circumstances that may cause TEMS initialization delays, the case may arise where an e3270ui DRA that is running under the z/OS TEMS address space interferes with TEMS normal initialization, that is causing the TEMS to terminate during startup. This case may arise when the e3270ui DRA startup completes before the TEMS initialization is complete. The problem symptoms are evidenced by the following messages being written to the TEMS RKLVLOG log dataset:

"(001A-D3E8C823:kdcsuse.c,120,"KDCS_UseFamily") status=1c010005, "cant bind socket", ncs/KDC1_STC_CANT_BIND_SOCKET Connection failure: "ip.pipe:#x.x.x.x:1918 status=1c020006, "location server unavailable", ncs/KDC1_STC_SERVER_UNAVAILABLE Endpoint unresponsive: "ip:#x.x.x.x:1918", 1C010001:1DE0000F, 33, CT/DS RPC Error: DSR031 - CTDS_CreatePath RPC abend status=1c020006, "location server unavailable", ncs/KDC 5,"socket__inq_my_netaddr") status=10020002, "socket buffer too small", ncs/KDC1 CT/DS status returned: 155 Check local failed. status = 5 Create Path Error. status 155 path NCS:{SOCKET=sna:&netid.CTDDSLB } CT/DS:{SERVER=SRVR01 USER=OB} "Find_HUB") KOBDRA76E Unable to connect the TEMS" .

In this case, it is necessary to change the TEMS configuration to force a delay of the e3270ui DRA startup. This is accomplished by modifying the DRA startup command (IRAMAN) in the &rhilev.&rte.RKANCMDU(KDSSTART) runtime (RTE) dataset.

Note: changes documented below describing manual updates to the &rhilev.&rte.RKANCMDU(KDSSTART) dataset are addressed by APAR: OA40162, PTF: UA66248.

To change the configuration, edit the &rhilev.&rte.RKANCMDU(KDSSTART) dataset and apply the modification described below; that is, increase the delay time value specified in the KOBAGENT IRAMAN START command statement; that is:

change:

AT ADD ID=KOB DELAY=00:00:10 CMD='IRAMAN KOBAGENT START'

to:

AT ADD ID=KOB DELAY=00:01:00 CMD='IRAMAN KOBAGENT START'

The above modification would change the KOBAGENT startup delay from ten (00:00:10) seconds to one minute (00:01:00). Note: It is possible that a one minute delay is insufficient, increase this value as required, for example, to two minutes (00:02:00) to satisfy site requirements.

There is a well known circumstance that may cause TEMS initialization delays. That is, in the case where an RTE configured with one or more of the products listed below has been upgraded to version 5.1.0 or 5.1.1.

- OMEGAMON XE for CICS on z/OS V5.1.0
- OMEGAMON XE for DB2 Performance Expert V5.1.1
- OMEGAMON XE on z/OS V5.1.0

In the RTE upgrade case, the &rhilev.&rte.RKANDATV RTE dataset may contain obsolete members. The members in question are listed below:

- KC5ATR
- KC5CAT
- KD5ATR
- KD5CAT
- KHLATR
- KHLCAT

The potential for TEMS initialization delays problem symptoms will be evidenced by messages such as those shown below written to the TEMS RKLVLOG log dataset.

(0001-C769FC4B:kglatrvl.c,193,"add_to_index") Warning: attribute name conflict

(0006-C769FC4B:kglatrbk.c,893,"atrblk_set_format") app <OMCICS>table <CICSEVD> column <CAPTURE> file <KC5ATR> timestamp
<1110601172956000> line <5926>
.
.
(70C6-C769FC4B:kglatrvl.c,194,"add_to_index") app <KDP> table <KDP_CFG> column <ORIGINNODE>
(70C7-C769FC4B:kglatrvl.c,194,"add_to_index") file <KDPATR> timestamp <1120426032318000> line <3>
.
.
(282F-C769FC4B:kglatrvl.c,195,"add_to_index") app <KM5> table <HLHCHKS> column <ORIGINNODE>
(2830-C769FC4B:kglatrvl.c,195,"add_to_index") file <KM5ATR> timestamp <1120502060715000> line <29579>
.
(2F99-C769FC4B:kglatrvl.c,195,"add_to_index") enum count

Note: The ICAT or PARMGEN RTE Load job (pp#2xxxx ICAT / KCIJPLOD PARMGEN) typically deletes these members in the DELRUN clean-up step, however, if the job is not able to get exclusive control of the RKANDATV dataset, it will terminate with a warning condition code and fail to delete the members. In that case, the members should be manually removed.

Note that manual removal should only be done after having verified that: (a) the product in question has indeed been upgraded in the RTE and (b) the above indicated messages are being written to the TEMS RKLVLOG dataset.