When you collect data, it is important to distinguish between short and long-term historical data.
IBM® Tivoli® OMEGAMON® XE for Storage on z/OS® provides the following types of historical data collection:
Short-term historical data is stored in the persistent data store on z/OS systems or in files on distributed systems. Short-term historical data comprises data that is stored for 24 hours or less. However, the amount and age of short-term historical data that Tivoli OMEGAMON XE for Storage on z/OS keeps, depends on such factors as:
To find out how to configure the collection of short-term historical data for the persistent data store, see the appendix of the IBM Tivoli OMEGAMON XE for Storage on z/OS: Planning and Configuration Guide.
Short-term historical data is used to investigate and determine the nature of problems that arise.
Long-term historical data is stored in the IBM Tivoli Data Warehouse. The long-term history database can retain data collected by Tivoli OMEGAMON XE for Storage on z/OS monitoring agents for days, weeks, months, or years. See the following documents for detailed information about the collection of historical data:
You use long-term historical data to analyze trends and determine workload balance. Tivoli Data Warehouse is the primary tool for collecting long-term historical data that IBM Tivoli Monitoring gathers. The warehouse architecture requires a relational database. The warehouse installer provides an IBM DB2® relational database. However, the warehouse can use, instead of DB2, an Oracle, or Microsoft SQL relational databases. Long-term history also requires installation of the Warehouse Proxy Agent software (provided). See IBM Tivoli Monitoring: Installation and Setup Guide for information about setting up the collection of long-term historical data. Yo must activate and configure short-term historical data collection if you want to collect long-term historical data.
After you activate historical data collection, an icon is displayed in qualifying views in IBM Tivoli Enterprise Portal workspaces. To produce meaningful reports, you must wait until sufficient historical data is collected and stored. You can click the icon to extend any existing Tivoli Enterprise Portal view (also called a report) to include historical data. Tivoli Enterprise Portal reports automatically pull data from short-term and long-term history based upon the time period that you specify for the report.
You can configure different collection intervals for historical and real-time data. To avoid excessive demand on processor resources and to decrease storage consumption, historical data collection is typically performed less frequently than real-time data collection. You can configure a short-term historical data collection interval of 5, 15, 30, or 60 minutes.
Writing data to long-term history can be configured for 24 hours or 1 hour, or you can turn it off. If you configure long-term history, select a warehousing interval of 1 hour to avoid transferring 24 hours worth of historical data at one time. Shorter intervals reduce the duration of processor usage associated with writing data to the warehouse by distributing the writing of data across 24 time periods.
You set the following intervals for data collection:
| Type of interval | Duration in minutes |
|---|---|
| Real-time data collection | 5 |
| Short-term historical data collection | 15 |
| Long-term warehousing | 60 |
The real-time data collection is based on the intervals that you set:
| Time | Real-time data collection |
|---|---|
| 1:57 | First row of data, 798 bytes, collected |
| 2:02 | Second row of data collected (replaces first row of data which is discarded) |
| 2:07 | Third row of data collected (replaces second row of data which is discarded) |
| 2:12 | Fourth row of data collected (replaces third row of data which is discarded) |
| 2:17 | Fifth row of data collected (replaces fourth row of data which is discarded) |
| 2:22 | Sixth row of data collected (replaces fifth row of data which is discarded) |
| 2:27 | Seventh row of data collected (replaces sixth row of data which is discarded) |
| 2:32 | Eighth row of data collected (replaces seventh row of data which is discarded) |
| 2:37 | Ninth row of data collected (replaces eight row of data which is discarded) |
| 2:42 | Tenth row of data collected (replaces ninth row of data which is discarded) |
| 2:47 | Eleventh row of data collected (replaces tenth row of data which is discarded) |
| 2:52 | Twelfth row of data collected (replaces eleventh row of data which is discarded) |
| 2:57 | Start of next cycle of real-time data collection |
The short-term historical data collection is based on the intervals that you set:
| Time | Short-term historical data collection |
|---|---|
| 1:58 | Start collection |
| 2:00 | Use the most recent real-time collection to store the first row of data (816 bytes) |
| 2:15 | Use the most recent real-time collection to store the second row of data (816 bytes) |
| 2:30 | Use the most recent real-time collection to store the third row of data (816 bytes) |
| 2:45 | Use the most recent real-time collection to store the fourth row of data, (816 bytes) |
| 3:00 | Use the most recent real-time collection to store the fifth row of data (816 bytes) |
Every hour, the Warehouse Proxy Agent transfers all rows of data to the database as long-term historical data that is maintained by IBM Tivoli Data Warehouse. You can use the following formula to calculate the volume of data, in bytes, that is collected: 4 x 816 x n. In the formula, n stands for the total number of data sets in all data set groups.
You must decide which types of historical data to store in short-term and long-term history and you must also decide how long to store the data. Because data collection consumes processor cycles and disk space, there are, inevitably, trade-offs. Writing data to short-term history is cost-effective and typically much less costly than writing to long-term history.
If you choose to configure short-term historical data collection, an additional amount of storage is used on z/OS. If you also choose to configure long-term history (in the Tivoli Data Warehouse) an additional amount of storage is used on a distributed server (for example, on Windows ) for the historical data that your company needs to retain for days, weeks, months, or years.
Short-term history is written to disk. This operation is typically performed at the monitoring agent and it consumes processor cycles. Additional processor cycles are used when the Warehouse Proxy Agent receives data from short-term history and transfers it to the data warehouse. If you collect a large amount of data in short-term history, the extraction process significantly increases the use of processor resources by the monitoring agent.
For many companies, the following configurations and settings offer the best utilization of processor and storage resources:
Depending on your requirements, you can configure historical data collection for only a subset of attribute groups. This method is effective in limiting storage consumption, in particular, when you choose not to perform historical data collection for the following high-volume attribute groups:
You avoid the collection of data that you do not require for historical reports.
You can use this information as a basis for choosing which attribute groups to enable for historical collection. You can select individual attribute groups for historical collection, including specifying different historical collection intervals and warehouse intervals.
By default, historical reports retrieve up to 24 hours of data from short-term history. If your persistent data store is not allocated with sufficient space, you will not have 24 hours of short-term data to retrieve. Allocate your persistent data store to hold a full 24 hours of data or change the default of 24 hours. See the Configuring IBM Tivoli OMEGAMON for Storage on z/OS and OMEGAMON II for SMS manual for information about how to change the default of 24 hours.
Because historical data accumulates, you must also determine how long you want to keep the data.