Customer ALERT for Bug 31496

Content

Severity : High

Technical Description:

We have identified a scenario where the S-TAP has a potential to run into the race condition, leading to high CPU utilization and DB connection slowness.

The Issue could be essentially triggered on the appliance side by high utilization of the internal appliance database which causes spikes in MySQL CPU consumption. In the observed cases those peaks lead to MySQL accessibility issues which in its turn causes rapid growth of the kernel buffers and eventually leads to the sniffer communication thread getting stalled in the SYN_RECV state.

At this point S-TAP gets into a loop, continuously trying to communicate with a non-responsive sniffer thread, which subsequently slows down DB server performance.

Below shows a typical appliance netstat output for connected S-TAPs when the problem is observed:

tcp 0 0 0.0.0.0:16018 0.0.0.0:* LISTEN
tcp 0 0 111.222.333.92:16018 123.456.78.912:40682 SYN_RECV
tcp 0 0 111.222.333.92:16018 123.456.789.123:45045 SYN_RECV
tcp 0 0 111.222.333.92:16018 123.456.7.890:54140 SYN_RECV
tcp 0 0 111.222.333.92:16018 123.456.89.01:52147 SYN_RECV

During the normal operation the status should be ESTABLISHED.

Remediation:

We have made a combination of changes in the sniffer software on the Guardium Appliance and the STAP agent:

1. Sniffer patch v8.2p121 or later for the Guardium Appliance

2. Unix STAP r45364 or later


Sniffer changes:
1. Changes to validate network connections and ignore all invalid connections to prevent the sniffer getting into the unresponsive state.
2. Change to detach sniffer interfaces with S-TAP and internal MySQL DB into separate asynchronous threads to provide high level of fail tolerance to database activity.

S-TAP changes:

1. Changes to prevent S-TAP from going into an endless loop while trying to communicate with an unresponsive sniffer process
2. Performance improvement for creating TLS connections reducing S-TAP CPU load



Affected Operating Systems	All Unix versions (AIX,SOLARIS,HP,Linux)
Affected Databases	All
Affected Guardium versions	v8.2
Fixed in revision	All Unix platforms – r45364 or later


The latest S-TAPs and other products can be found and downloaded at Fix Central
If you log into this site first - you can then select any of the following specific fixes one at a time

Fix Central Specific fixes for this (over time these might not be the latest S-TAPs)
r45364 for suse 11
r45364 for rhel 5
r45364 for rhel 6
r45364 for rhel
r45364 for suse
r45364 for SOLARIS
r45364 for HPUX
r45364 for AIX