Simplifying ITM situations with time limits Using UNTIL

An ITM Situation formula defines when an alert should be generated. In most cases that means specifying a general condition, like a queue being INACTIVE. In many real cases, there are time schedules where an inactive queue is normal and should not generate an alert. For an example, consider a Linux OS situation with a formula that will alert if there are less then 100 processes.

For this class of server, less then 100 processes would be abnormal during production time but might occur during non-production time. Suppose production time is Monday 8am to Saturday 12 noon. One way to encode this logic in a formula is:



This does not appear complex on the surface, however it results in a complicated set of work units for TEMS for two reasons. First, each unit of work is limited to one attribute group at a time whereas the above formula is formed from two attribute groups: Linux_System_Statistics and Local Time. Second, situations which combine AND and ORs are reformulated before being sent to the SQL component of TEMS.

The formula above is compiled into "predicate" form when the situation is started and the results are stored in the TEMS SITDB table. Here is what the SITDB predicates look like for the above formula. There is a much simpler way to get this work done which is shown afterwards. The vertical bar is a separator between the rulename and the predicate.

IBM_SI_________________________0 | IBM_SI_________________________1 AND IBM_SI_________________________2

IBM_SI_________________________1 | SELECT DAYOFMONTH, DAYOFWEEK, HOURS, MINUTES, MONTHNUM, SECONDS, ORIGINNODE, SYSTIME FROM O4SRV.LOCALTIME WHERE SYSTEM.PARMA("SITNAME", "IBM_SI", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711081114001", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND LOCALTIME.DAYOFWEEK = '02' AND LOCALTIME.HOURS >= '08' ;

IBM_SI_________________________2 | SELECT CSWSEC, MAJFLTPS, CURUSRS, PGFLTPS, SWAPINPS, SWAPOUTPS, PGSWAPIN, PGSWAPOUT, PGPGIN, PGPGINPS, PGPGOUT, PGPGOUTPS, RATECSW, RATEPROC, ZOMBCNT, PROCSEC, LOAD15MIN, LOAD1MIN, LOAD5MIN, ORIGINNODE, SYSUPTIME, TIMESTAMP, TOTPROCS FROM KLZ.KLZSYS WHERE SYSTEM.PARMA("SITNAME", "IBM_SI", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711081114001", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND KLZSYS.TOTPROCS < 100 ;

IBM_SI_________________________3 | IBM_SI_________________________4 AND IBM_SI_________________________5

IBM_SI_________________________4 | SELECT DAYOFMONTH, DAYOFWEEK, HOURS, MINUTES, MONTHNUM, SECONDS, ORIGINNODE, SYSTIME FROM O4SRV.LOCALTIME WHERE SYSTEM.PARMA("SITNAME", "IBM_SI", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711081114001", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND LOCALTIME.DAYOFWEEK >= '03' AND LOCALTIME.DAYOFWEEK <= '06' ;

IBM_SI_________________________5 | SELECT CSWSEC, MAJFLTPS, CURUSRS, PGFLTPS, SWAPINPS, SWAPOUTPS, PGSWAPIN, PGSWAPOUT, PGPGIN, PGPGINPS, PGPGOUT, PGPGOUTPS, RATECSW, RATEPROC, ZOMBCNT, PROCSEC, LOAD15MIN, LOAD1MIN, LOAD5MIN, ORIGINNODE, SYSUPTIME, TIMESTAMP, TOTPROCS FROM KLZ.KLZSYS WHERE SYSTEM.PARMA("SITNAME", "IBM_SI", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711081114001", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND KLZSYS.TOTPROCS < 100 ;

IBM_SI_________________________6 | IBM_SI_________________________7 AND IBM_SI_________________________8

IBM_SI_________________________7 | SELECT DAYOFMONTH, DAYOFWEEK, HOURS, MINUTES, MONTHNUM, SECONDS, ORIGINNODE, SYSTIME FROM O4SRV.LOCALTIME WHERE SYSTEM.PARMA("SITNAME", "IBM_SI", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711081114001", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND LOCALTIME.DAYOFWEEK = '07' AND LOCALTIME.HOURS < '12' ;

IBM_SI_________________________8 | SELECT CSWSEC, MAJFLTPS, CURUSRS, PGFLTPS, SWAPINPS, SWAPOUTPS, PGSWAPIN, PGSWAPOUT, PGPGIN, PGPGINPS, PGPGOUT, PGPGOUTPS, RATECSW, RATEPROC, ZOMBCNT, PROCSEC, LOAD15MIN, LOAD1MIN, LOAD5MIN, ORIGINNODE, SYSUPTIME, TIMESTAMP, TOTPROCS FROM KLZ.KLZSYS WHERE SYSTEM.PARMA("SITNAME", "IBM_SI", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711081114001", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND KLZSYS.TOTPROCS < 100 ;

IBM_SI__________________________ | IBM_SI_________________________0 OR IBM_SI_________________________3 OR IBM_SI_________________________6

This last line is the base situation. So what appears to be a simple formula can in fact result in a complicated set of SQL predicates for the TEMS and agent to perform.

Another issue with the formula is that you must repeat the same logic in each situation, and when a schedule changes then many situations need to be changed. Thus it results in a lot of manual work and verification work. There is a better solution using the UNTIL clause.

Create a situation IBM_SN_schedule which defines the non-production time, like this.



It is the logical opposite of the production time schedule. This situation needs the same distribution as the base situation and for production should be autostarted. After development is complete, the situation can be dissociated from the navigation tree to avoid presenting helper situation alerts.

Next create the base situation, IBM_SN



with an UNTIL tab that looks like this


 

The SITDB rules now look like this:

IBM_SN__________________________ | SELECT CSWSEC, MAJFLTPS, CURUSRS, PGFLTPS, SWAPINPS, SWAPOUTPS, PGSWAPIN, PGSWAPOUT, PGPGIN, PGPGINPS, PGPGOUT, PGPGOUTPS, RATECSW, RATEPROC, ZOMBCNT, PROCSEC, LOAD15MIN, LOAD1MIN, LOAD5MIN, ORIGINNODE, SYSUPTIME, TIMESTAMP, TOTPROCS FROM KLZ.KLZSYS WHERE SYSTEM.PARMA("SITNAME", "IBM_SN", 6) AND SYSTEM.PARMA("NUM_VERSION", "8", 1) AND SYSTEM.PARMA("LSTDATE", "1090711093216000", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND KLZSYS.TOTPROCS < 100 ;

IBM_SN_schedule_________________ | SELECT DAYOFMONTH, DAYOFWEEK, HOURS, MINUTES, MONTHNUM, SECONDS, ORIGINNODE, SYSTIME FROM O4SRV.LOCALTIME WHERE SYSTEM.PARMA("SITNAME", "IBM_SN_schedule", 15) AND SYSTEM.PARMA("NUM_VERSION", "0", 1) AND SYSTEM.PARMA("LSTDATE", "1090711092927000", 16) AND SYSTEM.PARMA("SITINFO", "TFWD=N;OV=N;", 12) AND ( ( LOCALTIME.DAYOFWEEK = '02' AND LOCALTIME.HOURS < '08' ) OR ( LOCALTIME.DAYOFWEEK = '07' AND LOCALTIME.HOURS > '12' ) OR ( LOCALTIME.DAYOFWEEK = '01' ) ) ;

The predicates are fewer [two instead of nine] and simpler - with no generated AND and OR sub-situations.

When situation IBM_SN is started, the situation IBM_SN_schedule starts automatically with the same sampling interval and distribution as IBM_SN. The agent processes both of them independently and sends the results to the TEMS that it reports to. When the TEMS sees an IBM_SN true result, the UNTIL situation is checked and if it is true then the IBM_SN true is suppressed.

There is a bit more work for TEMS. However the time schedule situation will only fire twice a week, going true at Saturday 12 noon and false Monday 8am. The base routine will fire when there is a problem during production hours, will not be suppressed and an alert will be seen. If it fires outside production time, then TEMS logic will suppress it. The additional TEMS workload would likely never be measurable.

As an added benefit the time scheduler situation can be reused for any other situations that use the same time schedule.