IBM Support

QRadar: Accumulator_Rollup overview

Question & Answer


Question

What is an accumulation and what does QRadar do with accumulated data?

Answer

The Accumulator service is a QRadar process that counts and prepares both Events and Flows in data accumulations to assist with searches. These accumulations also enable dashboard charts and improve report performance.

Accumulated Data is used to draw Time Series graphs or run Scheduled Reports. We refer to the data created by accumulation as an aggregated data view (ADV) or Global View (GV). The Accumulator service runs on all appliances with local storage (Console, 16xx, 17xx, 18xx, 14xx) to create the minute by minute accumulations. Every hour, the accumulator_rollup service runs to create the hourly roll-up files. Each morning at 12:15 AM, accumulator_rollup creates the daily roll-up for the previous day's data.
Note: The accumulator_rollup service is started by hostcontext when it is time to perform an hourly or daily rollup. It is normal for this service to be in a "failed" state in the systemctl status accumulator_rollup output.

There are two ways in which accumulation can be enabled:

Capture Time Series data

The following steps outline how to capture Time Series data for accumulation:
  1. Access Log Activity or Network Activity.
     
  2. Conduct a search, ensuring that the search includes at least one "Group By" field and specifies a time range. Charts are not displayed when you view events or flows in Real Time (streaming) mode.


     
  3. In the Charts pane, click the Configure icon.
     
  4. In the Chart Type drop down, select Time Series.
  5. Enable the Capture Time Series Data option to enable time series data capture. When you enable this option, the chart feature begins accumulating data for time series charts. By default, this option is disabled. This option is only available on Time Series charts.


     
  6. Click the Save icon.
     
  7. If the search used was not a previously Saved Search Criteria, a window is presented in which a search criteria name needs to be provided.
     
  8. Click the OK icon.

Note: After you enable a time series data capture for a selected parameter, an asterisk (*) is displayed next to the parameter in the Value to Graph list box. Select the Capture Time Series Data check box and click the Save icon for each parameter for which you would like to accumulate data.

Scheduled Reports

If a Saved Search is used in a Daily, Weekly, or Monthly Report, then the data that matches the Saved Search criteria is accumulated. For more information on creating Reports, review the Creating custom reports documentation.


Overview

This data is accumulated, or "rolled up", into three different resolutions by time. This accumulation of data reduces the amount of data that is queried when the specified saved search is performed across that time period.

The accumulator service runs continuously and accumulates data every minute. Every minute, the normalized data collected in the previous minute accumulates based on the saved search criteria. This data is referred to as the minute roll-up

Hourly Roll-up

Every hour, the minute roll-up files accumulate into an hourly file. The 60 data points produced by the normal roll-up accumulate into 1 data point for the hour.

Daily Roll-up

Every day, the hourly roll-up files accumulate into a daily file. The 24 data points produced by the hourly roll-up accumulate into 1 data point for the day.

The different roll-up files store different time-based accumulations of the data. The accumulated data is stored in a flat file within the Ariel database. Each file is subdivided as Years, Months, Days, and Hours. The minute roll-up data are stored in the hours file.


Example of how the Accumulator rolls up data

The accumulator data for this example is based on the following search criteria:
Search criteria: Grouped by IP and Policy, Column = Risk Score (sum)

Example data:
  • 01/15/2013 01:01:02 IP=1.1.1.1, Policy A, Risk Score = 5
  • 01/15/2013 01:01:23 IP=1.1.1.1, Policy A, Risk Score = 7
  • 01/15/2013 01:01:12 IP=1.1.1.1, Policy B, Risk Score = 10
  • 01/15/2013 01:02:43 IP=1.1.1.1, Policy A, Risk Score = 5
  • 01/15/2013 02:05:14 IP=1.1.1.1, Policy A, Risk Score = 5

Minute Roll-up

Data is grouped by IP and Policy and Risk Scores are added.
In the example data, there are two events that occurred within the same minute. During the Minute Roll-up, these two events are combined into one record in the ADV, as the results of their grouped search parameters (IP and Policy) are the same. The Risk Scores for these two events are added:

  01/15/2013 01:01:02 IP=1.1.1.1, Policy A, Risk Score = 5
+ 01/15/2013 01:01:23 IP=1.1.1.1, Policy A, Risk Score = 7
= 01/15/2013 01:01 IP=1.1.1.1, Policy A, Risk Score = 12


The accumulated Minute Roll-up from the example data results in the following:
  • 01/15/2013 01:01 IP=1.1.1.1, Policy A, Risk Score = 12
  • 01/15/2013 01:01 IP=1.1.1.1, Policy B, Risk Score = 10
  • 01/15/2013 01:02 IP=1.1.1.1, Policy A, Risk Score = 5
  • 01/15/2013 02:05 IP=1.1.1.1, Policy A, Risk Score = 5

Hourly Roll-up

The Hourly Roll-up data is rolled up the resulting data from the Minute Roll-up each hour. There were three results for the first hour, all are from the same IP, but two are from Policy A, and one is from Policy B. The two results that have the same IP and Policy are rolled up together as the results of their grouped search parameters (IP and Policy) are the same. The Risk Scores for these two results are added:

  01/15/2013 01:01 IP=1.1.1.1, Policy A, Risk Score = 12
+ 01/15/2013 01:02 IP=1.1.1.1, Policy A, Risk Score = 5
= 01/15/2013 01 IP=1.1.1.1, Policy A, Risk Score = 17


The accumulated Hourly Roll-up from the results of the Minute Roll-up data results in the following:
  • 01/15/2013 01 IP=1.1.1.1, Policy A, Risk Score = 17
  • 01/15/2013 01 IP=1.1.1.1, Policy B, Risk Score = 10
  • 01/15/2013 02 IP=1.1.1.1, Policy A, Risk Score = 5

Daily Roll Up

Once a day, the Daily Roll-up again rolls up the resulting data from the Hourly Roll-up even further.
There are three for the day, but only two have both the same IP and Policy. Those two roll up together as the results of their grouped search parameters (IP and Policy) are the same. The Risk Scores for these two results are added:

  01/15/2013 01 IP=1.1.1.1, Policy A, Risk Score = 17
+ 01/15/2013 02 IP=1.1.1.1, Policy A, Risk Score = 5
= 01/15/2013 IP=1.1.1.1, Policy A, Risk Score = 22


The accumulated Daily Roll-up from the results of the Hourly Roll-up data results in the following:
  • 01/15/2013 IP=1.1.1.1, Policy A, Risk Score = 22
  • 01/15/2013 IP=1.1.1.1, Policy B, Risk Score = 10


Where do you find more information?

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwstAAA","label":"Accumulator"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
21 June 2023

UID

swg21677942