Troubleshooting
Problem
This document discusses some pitfalls of having a slower connection between the console and a managed host, with details on how to test the network speed.
Symptom
You might find that a particular host (or hosts) in your environment frequently fail to deploy changes due to insufficient network speed between the console and the host.
Diagnosing The Problem
One indication that the bandwidth between your console and your managed host is too slow is to look at how long replication takes to complete.
Note: To learn more about the impact of low bandwidth on replication, read QRadar: Effects of low bandwidth on replication.
To view how long replication takes, SSH to the console and then SSH to the managed host in question. On the managed host, run the following:
grep "Replication download" /var/log/qradar.log | tail
Example output:
[root@HOSTNAME ~]# grep "Replication download" /var/log/qradar.log | tail
HOSTNAME replication[13866]: Replication download timing: Downloading: 427 ms Overall: 453 ms DB transaction: 20 ms Transaction verification: 6 ms
HOSTNAME replication[724]: Replication download timing: Downloading: 399 ms Overall: 428 ms DB transaction: 21 ms Transaction verification: 6 ms
HOSTNAME replication[19562]: Replication download timing: Downloading: 403 ms Overall: 431 ms DB transaction: 22 ms Transaction verification: 6 ms
HOSTNAME replication[5264]: Replication download timing: Downloading: 399 ms Overall: 426 ms DB transaction: 20 ms Transaction verification: 6 ms
Your results display the 10 most recent replication downloads, occurring each minute. Find the "Downloading:" part of the messages and note how long it took to complete. Ideally, replication needs to be under 3000 ms. If these replications are taking longer than 5000 ms to complete, you likely have some bandwidth issues. In the last example, each replication all completing around 1/2 a second and indicate no problem.
If you do find that replication is taking longer than 5 seconds, you can test the network speed by transmitting some data between the console and the managed host. One way of testing speed is using the dd command and moving the data to /dev/null, so no hard disk space is used. An example of the command:
dd if=/dev/zero bs=1024 count=1048576 | ssh root@x,x,x,x 'cat > /dev/null'
This example transfers approximately 1GB of information over the network. Once completed, your output is similar to the following:
[root@HOSTNAME ~]# dd if=/dev/zero bs=1024 count=1048576 | ssh root@192.0.2.10 'cat > /dev/null'
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 6.6233 s, 162 MB/s
Note: If the transfer takes longer than 30 seconds to complete, enter Ctrl+c to end the transfer. It displays how much data transfers in that time and the average speed of the transfer.
This example shows that the transfer speed was 162 MB/s, which is sufficient. If your results are less than 12.5MB/s (roughly 100Mb/s after overhead), then this slower network link could impact the ability to stay in sync, causing replication or deployments to fail.
Resolving The Problem
If your network speed between the console and the managed hosts are consistently under the requirement of 100Mb/s, work with your network team on the possibility of increasing the speed to meet the requirements. Basic network troubleshooting needs done first to confirm there is not a configuration issue somewhere on the network potentially causing issues.
Here are some troubleshooting steps to help isolate the issue
- Check you network workflow. Refer to QRadar: Basic network troubleshooting workflow.
- Run from an SSH session the command traceroute between manage hosts switches and routers to isolate latency issues. An example of traceroute:
[root@HOSTNAME ~]# traceroute 192.168.0.84 traceroute to 192.168.0.84 (192.168.0.84), 30 hops max, 60 byte packets 732APPhost (192.168.0.84) 0.809 ms 0.708 ms 0.638 ms -
Use tools such as ifconfig, ip ethtool, or netstat. Refer to this article on how to use these tools: QRadar: Networking troubleshooting of interfaces and connections using the command line
-
Check all Switches and routers between QRadar console and managed hosts to ensure that they are working at network speeds compatible to QRadar.
-
If you are doing intensive searches, managing large numbers of assets, log sources, or Offenses, the preferred network requirement is 1 gigabit per second.
Results
Isolation of network issues ensures all QRadar components replicate and perform optimally.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
24 March 2023
UID
ibm10957897