QRadar: How does coalescing work in QRadar?

Event Coalescing helps improve performance and reduce storage for non-critical security events where the full event payload does not need to be saved. As data comes in and is coalesced, a large burst of events can convert hundreds of thousands of events into only a few dozen records. This action is done while QRadar maintains the count of the number of actual events. Coalescing gives QRadar the ability to detect, enumerate, and track an attack on a huge scale. It also protects the performance of the pipeline by reducing the workload of the system, including storage requirements for those events. When events are received that match a specific criteria, QRadar can use coalescing to determine what to store from the event payload based on the log source settings in QRadar.

For example, a multitude of similar events created during a Denial Of Service attack can be converted from hundreds of thousands of events into only a few dozen records, while maintaining the count of the number of actual events received.

How does coalescing work?

Event data received by QRadar is processed in to normalized fields, along with the original payload. When coalescing is enabled the following five properties are evaluated to determine if a data source can be coalesced:

• QRadar Identifier (QID)
• Source IP
• Destination IP
• Destination port
• Username
   

Event coalescing starts after three events have been found with matching properties within a 10 second window. Additional events that occur within the 10 second period are coalesced together, with a count of the events noted. For each record containing coalesced events, only the payload of the first coalesced event is retained.

For example, if 1,005 events are received by QRadar within a 10 second window. Each of the 1,005 events has the same  QRadar Identifier (QID), Source IP, Destination IP, Destination port, and Username and coalescing is enabled for the log source. The Log Activity tab represents these 1,005 events as follows: 
 

• The first three events display in Log Activity as individual events with unique event payloads.
  Event 1: full event details saved.
  Event 2: full event details saved.
  Event 3: full event details saved.
• The fourth event contains a unique payload; however, the remaining 1,001 events that also arrived within that 10 second period are coalesced and only the original payload for the fourth event is retained.
  Event 4: full event details saved.
  Event 5-1,005: event details are NOT saved to disk for each individual event. The count for the fourth event displays in the user interface as Multiple (1001). The value within parenthesis (x) indicates to the user that this log source coalesced a number of events as the  with a duplicate QRadar Identifier (QID), Source IP, Destination IP, Destination port, and Username in to the fourth event as these all occurred within the 10 second window. Events 5-1005 do not have unique event payloads stored on disk.
  
  Note: Rules created by users intended to count events do update the event count even if the event is coalesced. For example, a user creates a rule with the test and when at least 5 events are seen with the same Username. The events that occur within the multiple field (x) update the count tracked in QRadar and coalesced data can trigger an offense or rule response.

System wide Event Coalescing Settings

QRadar provides the ability to disable coalescing if there is a requirement to retain all event payloads. This can be done either at system level, or per log source basis.

Procedure:

To disable coalescing at the system level:

1. on the Admin tab, click the System Settings icon.
2. Click Advanced.
3. Under the System Settings heading, find the Coalescing Events setting.
4. To disable Coalescing Events for all log sources, select No,
5. Click Save, and close the window.
6. From the Admin tab click Deploy Changes.

Event coalescing options for a specific log source
Coalescing can be enabled or disabled on each log source. Administrators can set coalescing when the log source is created or when editing an existing log source.

Procedure
To disable Coalescing for an existing log source:

1. Click the Admin tab.
2. Click the Log Sources icon.
3. Double click on a log source to edit the configuration.
4. Clear the Coalescing Events check box.
   

What types of log sources should I consider disabling coalescing?

Log sources for DNS systems, Proxy Servers, Anti-Virus systems, Windows servers and Endpoints can be good candidates for turning coalescing off. These log sources often include additional event payload information beyond QRadar's normalized fields which can be unique to the event payload that an administrator would want captured and searchable for an investigation.