QRadar: How to Modify Event Formats using Syslog, Forwarding, and Routing Rules

Quick links

• Part 1: Use cases for Modifying a Syslog Event Format
• Part 2: Selecting a Syslog Server
• Part 3: Configuration Overview
  • Part 3.1: How to configure a Forwarding Destination
  • Part 3.2: How to configure a Routing Rule
  • Part 3.3: How to configure Syslog-NG to filter and forward events
  • Part 3.4: Restarting services and verifying the listen port
• Part 4: Troubleshooting Syslog-NG

 

 

 

Part 1: Use cases for Modifying a Syslog Event Format

The purpose of this article is to take an existing event from QRadar, modify the event format by using Syslog and forward the event to another Syslog Target.

Modification of an event format may be required in the following use cases:

1. You need to forward events from QRadar to another log target expecting a format that differs from the payload received by the event collector process.
2. QRadar receives an event format that does not follow our DSM guide and cannot be changed at the source. To avoid writing an LSX the format can be changed to follow QRadar requirements. This makes sense if there are minor changes required.

Both use cases are different but can be addressed by using the same method: An intermediate Syslog server between QRadar and peer system that can interpret and modify the event forward, then forward the data to the target.

For example

Figure 1: Using an intermediate server to modify and forward a Syslog event.


The goal of this article is to inform administrators to convert
QRadar WinCollect forwards the DNS Debug events in a QRadar specific format. Thus, it is a good example for use case 1.

Original Event Format
WinCollect sends an event using the following format:
Mar 20 16:59:13 dns1wc AgentDevice=FileForwarder AgentLogFile=e:\dnslog\/dnslog.txt PluginVersion=7.2.1.984723 Payload=20/03/2017 16:58:10 dns1 PACKET 000000B2CFFCBB90 UDP Rcv 172.16.91.2 d51d Q [0001 D NOERROR] (13)dietersLaptop(4)home(6)office(10)heidelberg(3)com(0)

The log target requires a unique format. In this example, the Syslog header added by WinCollect must be removed for the remote system to be able to interpret and parse the Syslog event:
20/03/2017 16:58:10 dns1 PACKET 000000B2CFFCBB90 UDP Rcv 172.16.91.2 d51d Q [0001 D NOERROR] (13)dietersLaptop(4)home(6)office(10)heidelberg(3)com(0)

 

Part 2: Selecting a Syslog Server

There is a huge number of different Syslog servers out there that can be used for our purpose. Unfortunately, each of them must be configured in a completely different way. A popular Syslog server that is available on different kind of Linux flavors and on Windows is Balabit’s syslog-ng server (https://www.syslog-ng.com/products/log-management-software/). Syslog-NG is also used on the QRadar appliance itself and may be a convenient tool to evaluate configurations. Therefore, this document describes how to implement our use case by using Syslog-ng. We do not install a new Syslog-ng server. Instead we reuse the Syslog-ng instance running on QRadar.

 

Part 3: Configuration Overview

Configuration overview:

1. How to configure a forwarding destination in QRadar named “SyslogNG Server”
2. How to configure a Routing Rule named "To_SyslogNG”
3. Modifying the Syslog-NG configuration to forward a filtered event
4. Restart services and verify the listen port is open

Our test environment consists of a QRadar 3190 (virtual All-in-One Console) with QRadar 7.3.x installed. The IP address of the VM Console is 172.16.91.73.
A log source is already created with a log source of type WinCollectMicrosoftDNS that is receiving DNS Debug events.

Figure 2: An example event payload from the QRadar user interface Event Details screen.

 

Part 3.1: How to configure a forwarding destination in QRadar

In the Admin tab, click the Forwarding Destination icon to create a destination for your Log Target with the following properties:

• Name - A recognizable name should be used as your forwarding destination, such as SyslogNG Server.
• Destination Address - This field should contain the IP address or a DNS resolvable hostname for the Log Target server. This address should be the receiver of the Syslog forwarded event.
• Event Format - Select Payload from the Event Format drop-down. The other options include: JSON and RAW, which is the raw event payload as received by QRadar from the original source (not normalized).
• Destination Port - Type the IP address of the listen port of the Syslog-NG (Log Target) server.
• Protocol - Select the forwarding protocol, the options are TCP, UDP, or TCP over SSL. Note: The TCP over SSL option requires a certificate be imported to QRadar to establish communication to the remote log target.
• Prefix a Syslog header - In this example, leave the check box clear. This option adds a Syslog header to correct issues where the forwarder cannot determine the Syslog header format or if the event does not contain a Syslog header, the QRadar Console adds a header.
  
  
  Figure 3: A example of the Forwarding Destinations configuration in QRadar.

In this example, the Destination Address is the IP address of our QRadar Console. In a real environment or in lab testing, the administrator should specify the IP address or resolvable hostname of an external Syslog-NG server. Never use the local Syslog-NG server in a production environment, instead a non-QRadar appliance should be used to reformat and forward event data.

Important: QRadar Support recommends using Syslog-NG reformatting on QRadar appliances only be used in a lab or proof-of-concept test environment. As reformatting events can consume QRadar resources on busy appliances and a 3rd party Linux host should be used for this work, instead of the QRadar Console in production environments.



 

Part 3.2: How to configure a Routing Rule

Routing Rules allow QRadar to forward specific events received, such as, log source, events containing specific custom properties, forwarding in bulk by appliance and more. The routing rule defines what events and where to send the events by selecting the Forwarding destination that was already created in QRadar. In this example, QRadar should forward the DNS log events to the Syslog-NG server destination already created. In this routing rule, we use online forwarding. Online forwarding is considered 'real-time' as the event is forwarded immediately after parsing from the event collection process in QRadar. After the event is parsed by the appliance, it is forwarded. In case you need to ensure reliable transfer, you should select offline forwarding. Offline forwarding is considered 'guaranteed forwarding'.

In the Admin tab start the Routing Rules application and create a new routing rule by using the following properties:

• Name - A recognizable name should be used for your routing rule, such as To_SyslogNG or a descriptive term that informs the admins about the data being forwarded, like DNS_FormatFilter_ToSyslogNG. This allows admins to understand the routing rule without having potentially open the user interface.
• Description - Optional. This field should contain a description of the reason or use of the routing rule for other administrators.
• Mode - In this example, select Online. If you select Offline, this data is forwarded when written to disk.
• Forwarding - This user interface value is dynamic. If the admin selects Online Mode, the an Event Collector can be selected as the forwarder. If Offline Mode is selected, then an Event Processor on an appliance can be selected to forward the data.
• Data Source - Select if you want to forward Event data or Flow data. The user interface will change the filter options based on your selection. For example, if you select Flows, only flow custom properties are available in the filter options.
  NOTE: You must click Add Filter to add the value to the routing rule.
• Routing Options - Select the Forwarding Destination SyslogNG Server and select the Forward check box. The other options include, Drop, which will not store the data in QRadar (drops the event or flow) and Bypass Correlation. Bypass Correlation works like a False Positive in QRadar, where you can prevent the data from being evaluated against rules and prevent offenses from being generated.
  
  
  Figure 4: An example of a Routing Rule configuration in QRadar.

 

Part 3.3: How to configure Syslog-NG to filter and forward events

As a last configuration step, we define a payload modification filter on the Syslog-NG server. The Syslog server’s configuration file syslog-ng.conf on the QRadar console is in the directory /etc/syslog-ng.

This file contains an include statement:
@include "/etc/syslog-ng/conf.d"

The include statement ensures that the Syslog server incorporates all configuration functions defined in /etc/syslog-ng/conf.d. Therefore, we create a new file in this folder and name it: 02-dnsdebug.conf.

Note: Remember to save any changes made to 02-dnsdebug.conf.

The file contains the following settings:

Figure 5: An example of adding a filter to modify a Syslog event and forward the data to a target log server.



 

Part 3.4: Restarting services and verifying the listen port

After the Syslog-NG file is configured and saved, the administrator must restart the Syslog-NG service on the host to ensure that the file is loaded. To restart the Syslog-NG service, type the following command:

• In QRadar 7.3.x: systemctl restart syslog-ng.service

To verify that the Syslog-NG server is listening on port 1999, type the following command:

Figure 6: Verify that the listen port is open by using the netstat command.




 

Part 4: Troubleshooting Syslog-NG

Troubleshooting Syslog-Ng is a 2 step process. You need to verify Syslog-NG is working locally and then you need to check are you sending events over the correct port.

1. First step is to test Syslog-NG locally to determine if your configuration is working.
   1. Have you restarted Syslog-NG after making your configuration changes.
   2. Use the logger test, then verify you are seeing that event. To use logger use this command.
      logger "Hello World"
      This command will use Syslog-NG and log the event locally. This message will normally appear in /var/log/messages.
      Aug 29 12:58:19 Testserver2 root: Hello World
      Note: Syslog-NG must also have a local configuration in addition to the forwarding configuration for logger to work.
   3. Make sure your port is open for port 1999.
       iptables -L -n -v | grep 1999
          0     0 ACCEPT     tcp  --  *      *    0.0.0.0/0      0.0.0.0/0      tcp dpt:1999
          0     0 ACCEPT     tcp  --  *      *    0.0.0.0/0      0.0.0.0/0      state NEW tcp dpt:1999
          0     0 ACCEPT     udp  --  *      *    0.0.0.0/0      0.0.0.0/0      udp dpt:1999
   4. You can start Syslog-NG in verbose mode to see any error messages by using this command. This will display debug messages.
      syslog-ng -Fevd
      
      Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1'
      Global value changed; define='scl-root', value='/usr/share/syslog-ng/include/scl'
      Global value changed; define='include-path', value='/etc/syslog-ng:/usr/share/syslog-ng/include'
      Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2'
      Module loaded and initialized successfully; module='system-source'
   5. You can use the strace command to see debug messages.
         strace -s 256 -f syslog-ng
          …
         9835 open(“/etc/resolv.conf”, O_RDONLY) = ENOENT (No such file or directory)
         9835 uname({sys=”Linux”, node=”thor”, …}) = 0
         9835 stat(“/etc/resolv.conf”, 0x7fff612518a0) = -1 ENOENT (No such file or directory)
      …
      In this case resolv.conf is missing.
2. Check if QRadar is forwarding events to the Syslog-NG server. If you do not see events you may have firewall issues either at QRadar in your network.
   Using Tcpdump locally and the logger command from the QRadar appliance forwarding events to Syslog-Ng you can determine if events are being received.
   1. Log in to you Syslog-NG server.
   2. Enter the tcpdump command.
      tcpdump -nnAs0 -i <interface> port 1999
   3. Log in to the QRadar appliance were you are sending events and type this command.
      logger -n <IP or syslog-NG server> -P <port> "Message"
      Example # logger -n 192.168.0.23 -P 1999 "Hello World"
      Note: The -n and -P options for logger are only available on QRadar 7.3.0 and up.
   4. On the Syslog-NG server look for a message similar to this.
      tcpdump -nnAs0 -i eth0 host 192.168.0.23
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
      12:39:02.261022 IP 192.168.0.23.55820 > 192.168.0.45.1999: UDP, length 37
      E..Ao.@.@...    7..     7.......-N.<5>Aug 24 12:39:12 root: hello world.
      12:39:02.261144 IP 192.168.0.45 > 192.168.0.23: ICMP 192.168.0.45 udp port 1999 unreachable, length 73
      n..@... 7..     7..........E..Ao.@.@... 7..     7.......-N.<5>Aug 24 12:39:12 root: Hello World.
      12:39:07.277320 ARP, Request who-has 192.168.0.45 tell 192.168.0.23, length 46

For any further configuration options and installation tasks please refer to Balabit’s online documentation: https://www.balabit.com/support/documentation. For questions on QRadar or this article, see our forums: http://ibm.biz/qradarforums.