IBM Support

Troubleshooting Persistent Data Store (PDS) Problems

Preventive Service Planning


Abstract

Experiencing PDS issues in my agent and / or Tivoli Management Server (TMS), how do I debug these issues.

Content

This document will assist you in troubleshooting a PDS issue in z/OS. This addresses V622, V623 and V630 of IBM Tivoli Monitoring (ITM).

Updated maintenance for PDS is in the following PTFs:

ITM Version
PTF
Description
622
UA65748
ITM 622 FP9 is recommended.

UA76530
Persistent Data Store files goes offline after the maintenance jobs completes.
623
UA73317/UA73318
ITM 623 FP5 recommended.

UA76315
ABEND U1208 or ABEND S0C4 while running the PDS extract job, KPDXTRAJ.

UA76531
Persistent Data Store files goes offline after the maintenance jobs completes.

UA76635
Memory allocations intended to be 4k are actually allocated as 8k.
630
UA76202
ITM630 FP4 is recommended.

UA76316
ABEND U1208 or ABEND S0C4 while running the PDS extract job, KPDXTRAJ.

UA76532
Persistent Data Store files goes offline after the maintenance jobs completes.

UA76636
Memory allocations intended to be 4k are actually allocated as 8k.

UA92858
ABEND 0C4 may be reported by KPDMANE.
If possible, ensure PDS performs a graceful shutdown, in cases where this is not possible, the PDS maintenance will attempt to close the PDS files after writing the data in the buffer to the PDS files.

The latest PDS maintenance will attempt to create PDS dumps which will assist support in isolating your problem. Before reporting PDS issues to IBM Support Center, ensure the PTFs specified above are installed. If problem persists with PDS after the above PTFs are installed, provide the documentation as specified below and open a SR with the IBM Support Center.

PROBLEM
RESOLUTION
ABEND occurred in line nnn of @@nnnnn(ITBL) ,offset nnnnnn
Program terminated by operating system. ABEND code = S0C4
Send in dump and all logs from this address space.
LSCX010 **** ERROR ****
ABEND occurred in ASRTFL called from line nnn of _DYNAMN(MANE) ,
Program terminated by call to abort. ABEND code = U12nn
Calling trace:
Check for another abend prior to this abend. The first abend should be addressed first.

If no other abend, send in the dump and all logs from this address space.

KPDIFIL: Invalid status information detected in file hiLevel.Qualifier.table name
(group group-name).........The file had not been closed properly.
Try to issue the recover command and see if this resolves the issue: F address-space,KPDCMD RECOVER FILE=DSN:datasetname (where datasetname is the full-qualified name of the file, without quotes)
KPDMN004 PDS IS INACTIVEFor some reason PDS did not initiate correctly or it has abended, in the current design there is no command to re-activate PDS. A recycle of the address space is required.
KPDIFIL: Error opening file pdshilev.lowlev, RC = 14202Verify pdshilev.lowlev exists, if it exists verify it was built correctly.
If built correctly, verify it is not being allocated by another address space.
Command: EXEC
DSN:&rhilev.&rte.RKANPARU(KppPG)
LSCX504 **** WARNING **** ERRNO = ENFOUND

Generated in FOPEN called from line 214 of @@320897(EXEC)
, offset 0008D2
PDS member not found: KppPG
Verify the KppPG member exists in the &rhilev.&rte.RKANPARU data set.
2017/03/29 11:23:36 KPDDSTR: Any data associated with table table_name
on this file will be unavailable.
2017/03/29 11:23:36 KPDDSTR: Dictionary / DataStore mis-match in
DSN:&pdshilev.RKppPDS1
This message is saying the KppPDICT in the &rhilev.&rte.RKANPAR does not match the dictionary loaded in the actual PDS file. To resolve this issue you can try to perform: F address-space,KPDCMD SWITCH GROUP=cccccccc EXTRACT
command but a rebuild of the PDS file may be required to resolve this issue.
2016/07/07 05:43:52 KPDDSTR: CONNECT processing started for DataStore
file DSN:&pdshilev.RKppHISn
0LSCX100 **** WARNING **** ERRNO =
ENOMEM

Generated in CALLOC called from line 159 of @@333671
(HEAP) , offset 00010E
Memory unavailable for new allocation: 8128 bytes
required.
Calling trace:

Function Line Offset Context
MALLOC(L$CHEAP) ----
000134
CALLOC(L$CALOC) ----
00005A
@@333671(HEAP) 159
00010E
@@566861(L$POPN2) 8
000050
@@626075(DSTR) 1013
0007D2
@@473281(CNFG) 616
000414
@@473258(CNFG) 289
000488
DYNAMN(MANE) 1021
000EFA
MAIN(ARES) 254
000086
Reduce MINIMUM in &rhilev.&rte.RKANPARU(KppSYSIN).
KLVFM002 STORAGE RELEASE ERROR S000 U0100 THREAD 1F8FAF10 TASK
008FCB60
This abend may occur and it may fill up the JES spool with dumps, if the initial problem before this abend is storage shortage, verify the Warehouse Proxy Agent (WPA) is up and running. If you specify you want the history data to be warehoused and the WPA is not available, it will cause havoc on your address space, most likely it will run out of storage.
  1. If PDS continues to abend, you can execute a batch job to try to recover the data. The batch jobs can be found in &rhilev.&rte.RKANSAMU:
    1. KPDEXPTJ - Batch job to export a persistent datastore
    2. KPDRESTJ - Batch job to restore a persistent datastore from a file that was created by KPDEXPTJ.
    3. KPDXTRAJ - Batch job to extract table data from a persistent datastore dataset.
  2. If all else fails, you can scratch and reallocate the PDS files. In PARMGEN the WKANSAMU dataset contains standalone jobs to reallocate the PDS files, run the composite KCIJPALO RTE build job after deleting the PDS files. In ICAT, you can execute the #P and #Q jobs after deleting the PDS files.

QUESTION
ANSWER
Is there a way I can force the PDS maintenance job to execute? Yes, you can execute the following command against the address space the PDS files are connected to:

F address-space,KPDCMD SWITCH GROUP=cccccccc EXTRACT
How can I find the GROUP name? Issue the following commands:
F <addr-space>,KPDCMD QUERY CONNECT
F <addr-space>,FLUSH
How come I am not getting history data, I am seeing the data less than 24 hours old but nothing greater than 24 hours? 1. Verify the historical collection is turned on via the history icon in the TEP and if available in the history pulldown in the enhanced 3270ui.
2. Verify the UADVISOR for this table is started.
3. Verify the PDS files are correctly being processed and there is no errors in RKPDOUT.
How can I prevent the PDS Maintenance job from running? If a PDS file is corrupted, the maintenance job is needed to fix it. So it is not a good idea to prevent the maintenance job to execute all together.

[{"Product":{"code":"SSTFXA","label":"Tivoli Monitoring"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"--","Platform":[{"code":"PF035","label":"z\/OS"},{"code":"PF025","label":"Platform Independent"}],"Version":"6.2.0;6.2.1;6.2.2;6.2.3;6.3.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Product":{"code":"SSLSDR","label":"Tivoli OMEGAMON XE for CICS on z\/OS"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Product":{"code":"SS2JNN","label":"Tivoli OMEGAMON XE on z\/OS"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.1.1;5.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Product":{"code":"SSQH9J","label":"Tivoli OMEGAMON XE for CICS Transaction Gateway on z\/OS"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Product":{"code":"SSUSPS","label":"Tivoli OMEGAMON XE for DB2 Performance Expert on z\/OS"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.1.1;5.2.0;5.3.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}},{"Product":{"code":"SSUSPA","label":"Tivoli OMEGAMON XE for DB2 Performance Monitor on z\/OS"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.1.1;5.2.0;5.3.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}},{"Product":{"code":"SSXS8U","label":"Tivoli OMEGAMON XE for IMS on z\/OS"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Product":{"code":"SS2JL7","label":"Tivoli OMEGAMON XE for Mainframe Networks"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"5.1.0;5.1.1;5.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Product":{"code":"SS2JFP","label":"Tivoli OMEGAMON XE for Storage on z\/OS"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"4.2.0;5.1.0;5.2.0;5.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Product":{"code":"SSRLD6","label":"Tivoli OMEGAMON XE for Messaging for z\/OS"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"7.0;7.0.1;7.1.0;7.3.0","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}}]

Document Information

Modified date:
20 April 2020

UID

swg21623100