ITM Pure Situation events and Event Merging Logic
Why does the count of events arriving at TEC/Omnibus sometimes not match the count of pure situation triggering conditions?
The TEMS Agent proxy [subsystem that handles agent communications] has logic to merge multiple identical pure situation events into one. The duplicate key for the pure events is the combination of the following four attributes:
timestamp [to second]
originnode [managed system name]
DisplayItem [if present]
This means you only see this when situation events arrive tagged with the same timestamp [to the second] and the same DisplayItem [if present]. The design goal was to reduce TEMS processing when the same information was appearing over and over again. That is reasonable for a Portal Client view of the monitoring environment but may cause problems when the events are transmitted to other event processors.
One immediate way around the issue is to make sure each event has a unique display item. This has an extreme side effect resulting in steady TEMS storage growth and eventual failure. The TEMS logic keeps a Situation Status Cache in storage for efficient calculation of recent status. If DisplayItem values are all different, the required storage grows until it reaches the platform process limit which is near two gigabytes. This can happen rapidly or slowly depending on how many such situations events are processed. In such cases, the TEMS involved will have to be recycled periodically to avoid an unplanned outage.
Pure Event One Row Processing
The fix for APAR IZ76059 includes new logic that can change merging logic from these maintenance levels onward:
ITM 621 FP3
ITM 622 FP3
There may be a performance impact, so its use should be carefully tested before production usage. There will be more events processed and that will increase TEMS CPU usage. In rare cases, you may need to use a more powerful server to run the TEMS.
At ITM 621 FP3 and later an environment variable must be set:
KPX_PUREEVT_ONE_ROW=Y [Default is N]
In Linux/Unix add this to the ms.ini file and reconfigure TEMS. On Windows, environment variables are added from the MTEMS GUI - right Click on TEMS line, select Advanced/Edit Variables... and add a variable.
At ITM 622 FP3 no environment variable is needed.
In both cases a file is needed named KPXATRGP containing one line for each table where processing will to be altered. Each line will be the application and name of the table involved. Add a blank line at the end of the list. This file is stored in:
On Linux/Unix the KPXATRGP file must have the same attributes as the TEMS database files. For example a test system
# ls -al QA1CSTSC.DB
-rwxrwxrwx 1 root root 269878 Dec 8 12:29 QA1CSTSC.DB
After creating KPXATRGP in that specific environment you would need to do some or all of these commands
chmod 777 KPXATRGP
chown root KPXATRGP
chgrp root KPXATRGP
The file attributes, owner and group will change depending on how the TEMS is installed, what userid it will run as and whether secureMain has been run.
Calculating the Application and Table Name
The best way to determine the name and table is to examine the ODI file for the attribute group. This file has the form "docXXX" where XXX represents the product. For example, Windows OS Agent is docknt. You can examine the Portal Client Managed System Status workspace view to determine the product code.
The ODI file exists on the server running the TEPS in the following directory:
View the file and find the *APPL line which specifies the application. Here is an screen capture from a notepad edit of docknt
The situation formula will use some attributes. Locate an attribute in the ODI table, search backwards for the *OBJECT line and then just before that for the *TABLE for the attribute group involved. For the attribute group was NT_Event_Log attribute group here is a notepad screen capture:
In this case the table entry would be
Don't forget the required last blank line.
After the TEMS(es) are recycled, the Agent Proxy will not merge events for the specified tables.
Changing application and table names
In some cases, the application and table names may change over time. One example is a Universal Agent application where a major revision has been introduced. The table name might be TES00 on the first revision and TES01 after some columns were added. When that happens the KPXATRGP file will have to be updated at the same time in order to specify the new table name.
High Rate of Event Sending
Events are also sometimes lost when they are sent rapidly. To avoid this consider implementing rate control at the TEMS with the BufferFlushRate=events_per_minute parameter defined in the om_tec.config file, located in:
This specifies the number of events that are sent per minute. The default value is 0 ; consequently all events are sent in one burst. This is particularly evident during a hub TEMS recycle, when all current events are transmitted. It can also be seen when there are pure situations and a flood of triggering conditions.
More support for:
ITM Tivoli Enterprise Mgmt Server V6
Software version: All Versions
Operating system(s): AIX, HP-UX, IBM i, Linux, Solaris, Windows, z/OS
Software edition: All Editions
Reference #: 1445309
Modified date: 15 October 2010