IBM Support

Searching Your QRadar Data Efficiently: Part 2 - Leveraging Indexed Values

Troubleshooting


Problem

What are indexed values and how can they improve the speed of my searches in QRadar?

Resolving The Problem

How do I determine what properties QRadar indexes from events and flows?
QRadar indexes a number of properties when normalizing data, at the time it is received by QRadar and written to disk. Taking advantage of the default indexed properties is one of the best methods to improve the speed of searches in QRadar. Any search filter parameter with Indexed appended can be leveraged for more efficient searching. By adding indexed properties, the system is able to take advantage of having the property already extracted from the event or flow files. Searching indexed properties reduces the size of the data set required to find any remaining nonindexed search values.
 

Video 1: See the full series of videos on Searching your QRadar data efficiently in the IBM Security Learning Academy.

The following fields are indexed out-of-the-box in QRadar:
Event fields indexed by default Image
  • Custom Rule
  • Custom Rule Partially Matched
  • Destination IP
  • Destination Port
  • Event Name
  • Has Identity
  • Log Source
  • Log Source Type
  • Low-Level Category
  • Source IP
  • Username

Note: The parameter Category [Indexed] is an index of the Low-Level Category from the event data.

Note: The parameter Source or Destination IP is not defined as [Indexed], however, searches performed against this parameter use Source IP [Indexed] and Destination IP [Indexed] when selected. This field is a composite property, which is made up of two separate indexed properties.

Figure 1: Examples of default indexed event fields.
Flows fields indexed by default Image
  • Application
  • Custom Rule
  • Custom Rule Partially Matched
  • Destination IP
  • Destination Port
  • Source IP


Note: Any values that display (custom) in the name are associated to Custom Event Properties or Custom Flow Properties.

Figure 2: Examples of default indexed flow fields.


Example of how filters improve results
Let's say that a user wants to find all logs in the past six months where the payload contains the text, 'The operation is not allowed'. This means that to complete a payload search on the text 'The operation is not allowed' from the last six months, the system must test every payload from every event or flow from the last six months to find matches. Without any filters applied, the data sets can become large and impact performance when not required. QRadar is fully capable of returning this result, but the search is inefficient. If you can place an indexed value as a filter before you add an indexed filter (for example, a Log Source Type [Indexed], Username [Indexed], Event Name [Indexed], Source IP, or Destination IP) and then search for the payload containing 'The operation is not allowed', the search completes faster. Indexed filters assist the search by reducing the number of records for which ariel must apply the more expensive "payload contains" filter.


Tip: Start your searches by adding a filter that includes any relevant [indexed] properties to quickly locate results from large data sets or long time frames. Ariel uses index files to quickly find pointers to the data files that contain your results.



How do I leverage search statistics to improve the user experience?
The real power in indexing comes from understanding what users are searching for and helping them complete their searches by enabling indexes for the properties that users search frequently. Index Management allows administrators to enable or disable indexes on specific fields, and review the data behind what fields are used most often in searches.
 Index Management icon on the Admin Tab.

Figure 3: Example of the Index Management window from the QRadar Admin tab.
 

Search times can be reduced when you enable indexing for properties used frequently in searches. Administrators must be mindful when you enable indexing on large numbers of properties that can affect system performance. Specifically, during when the data is being written to the disk. Indexed properties do take more disk space as they have the extracted value from the record.

 AQL (Advanced Query Language) searches do not reorder based on index properties like default searches. The order of how filters are added impacts the search performance. Build the AQL query with indexed properties first for optimal search performance.

Index management allows administrators to control database indexing, which can optimize search performance for frequently searched values or criteria. As mentioned, QRadar enables indexes for some event and flow properties out-of-the-box. Administrators are encouraged to review what other properties are used frequently in searches to enable indexes and improve search performance. The Index Management feature in QRadar provides statistics for administrators to identify which properties are used in searches most often. Administrators can adjust the time range to understand how users are completing searches. The default statistics interval is 24 hours; however, statistics can be configured to display statistics from 1 hour ago to the last 30 days.


Index Management Columns:
  • % of Searches Using Property - This value indicates what percentage of searches use (across all users) a specific property in their search criteria within the selected time frame. High percentage values indicate properties that used in searches during the last interval.
  • % of Searches Hitting Index - This value represents the percentage of searches (across all users) that used the index property in a search value.
  • % of Searches Missing Index - A search that misses an index is counted each time a search includes a property, but either the property field has no index at any point during the search or when a search index is disabled.

    For example, a user creates a search for a property with a time frame of 8:00 to 9:00. If the search does not have any indexes (disabled), then a miss counted. When an index is disabled, this is why QRadar can show some searches missing 100% of the index.

    If an index is available for only part of the time interval that search counts as both an "index missed" and an "index hit" value. This situation can occur when an index is only available for part of the time period, such as an index being enabled for a property at 8:15. Since the index was not available for the entire time period, both the hit and the miss are counted, which can lead to values such as Searches using Property, 1.0%. Index hit, 4.5%. Index miss, 95.5%.
  • Data Written - The amount of disk space the index currently consumes.
  • Database - Some common properties overlap across both the event or flow data, such as IP addresses ports, etc. This column defines whether the property is part of the event or flow database.


Are there any general rules administrators can follow for indexing?
Yes, the following table provides some general advice on how to handle search indexes. Administrators can evaluate how users are leveraging searches over several time ranges (24 hours, 7 days, and 1 week) to make informed decisions about how users are searching in QRadar before making changes to the index configuration.
What to look for Time frame Action to consider Reason
Properties where the index is disabled and
the % of Searches Using Property is above 30% and the % of Searches Missing Index is above 30%
- 24 hours
- 7 days
- 30 days
Enable this index If administrators see search percentages above 30% across multiple time spans, then users are leveraging this search property often. Consider enabling indexing on this property. These values indicate that enabling an index can improve performance for users who search specific properties frequently.
Properties where the index is enabled and the % of Searches Using Property is zero. - 30 days Disable this index If after 30 days the statistics show that an enabled index is used in zero % of searches. Consider disabling indexing for this property.

Disabling preserves resources for more important and actively used searches.


 

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
22 December 2023

UID

swg21689802