Potential DB Server connections slowness on UNIX based systems.
Severity : High
We have identified a scenario where the S-TAP has a potential to run into the race condition, leading to high CPU utilization and DB connection slowness.
The Issue could be essentially triggered on the appliance side by high utilization of the internal appliance database which causes spikes in MySQL CPU consumption. In the observed cases those peaks lead to MySQL accessibility issues which in its turn causes rapid growth of the kernel buffers and eventually leads to the sniffer communication thread getting stalled in the SYN_RECV state.
At this point S-TAP gets into a loop, continuously trying to communicate with a non-responsive sniffer thread, which subsequently slows down DB server performance.
Additional Technical information:
Below shows a typical appliance netstat output for connected S-TAPs when the problem is observed:
tcp 0 0 0.0.0.0:16018 0.0.0.0:* LISTEN
tcp 0 0 111.222.333.92:16018 123.456.78.912:40682 SYN_RECV
tcp 0 0 111.222.333.92:16018 123.456.789.123:45045 SYN_RECV
tcp 0 0 111.222.333.92:16018 123.456.7.890:54140 SYN_RECV
tcp 0 0 111.222.333.92:16018 123.456.89.01:52147 SYN_RECV
During the normal operation the status should be ESTABLISHED.
We have made a combination of changes in the sniffer software on the Guardium Appliance and the STAP agent:
1. Sniffer patch v8.2p121 or later for the Guardium Appliance
2. Unix STAP r45364 or later
- Changes to validate network connections and ignore all invalid connections to prevent the sniffer getting into the unresponsive state.
- Change to detach sniffer interfaces with S-TAP and internal MySQL DB into separate asynchronous threads to provide high level of fail tolerance to database activity.
- Changes to prevent S-TAP from going into an endless loop while trying to communicate with an unresponsive sniffer process
- Performance improvement for creating TLS connections reducing S-TAP CPU load
|Affected Operating Systems||All Unix versions (AIX,SOLARIS,HP,Linux)|
|Affected Guardium versions||v8.2|
|Fixed in revision||All Unix platforms – r45364 or later|
The latest S-TAPs and other products can be found and downloaded at Fix Central
If you log into this site first - you can then select any of the following specific fixes one at a time
Fix Central Specific fixes for this (over time these might not be the latest S-TAPs)