Customer ALERT for Bug 31496

Flash (Alert)


Abstract

Potential DB Server connections slowness on UNIX based systems.

Content

Severity : High

Technical Description:

    We have identified a scenario where the S-TAP has a potential to run into the race condition, leading to high CPU utilization and DB connection slowness.

    The Issue could be essentially triggered on the appliance side by high utilization of the internal appliance database which causes spikes in MySQL CPU consumption. In the observed cases those peaks lead to MySQL accessibility issues which in its turn causes rapid growth of the kernel buffers and eventually leads to the sniffer communication thread getting stalled in the SYN_RECV state.

    At this point S-TAP gets into a loop, continuously trying to communicate with a non-responsive sniffer thread, which subsequently slows down DB server performance.


Additional Technical information:

    Below shows a typical appliance netstat output for connected S-TAPs when the problem is observed:

      tcp 0 0 0.0.0.0:16018 0.0.0.0:* LISTEN
      tcp 0 0 111.222.333.92:16018 123.456.78.912:40682 SYN_RECV
      tcp 0 0 111.222.333.92:16018 123.456.789.123:45045 SYN_RECV
      tcp 0 0 111.222.333.92:16018 123.456.7.890:54140 SYN_RECV
      tcp 0 0 111.222.333.92:16018 123.456.89.01:52147 SYN_RECV

    During the normal operation the status should be ESTABLISHED.

Remediation:

    We have made a combination of changes in the sniffer software on the Guardium Appliance and the STAP agent:

      1. Sniffer patch v8.2p121 or later for the Guardium Appliance

      2. Unix STAP r45364 or later


    Sniffer changes:
      1. Changes to validate network connections and ignore all invalid connections to prevent the sniffer getting into the unresponsive state.
      2. Change to detach sniffer interfaces with S-TAP and internal MySQL DB into separate asynchronous threads to provide high level of fail tolerance to database activity.

    S-TAP changes:

      1. Changes to prevent S-TAP from going into an endless loop while trying to communicate with an unresponsive sniffer process
      2. Performance improvement for creating TLS connections reducing S-TAP CPU load

      Affected Operating Systems All Unix versions (AIX,SOLARIS,HP,Linux)
      Affected Databases All
      Affected Guardium versions v8.2
      Fixed in revision All Unix platformsr45364 or later

      The latest S-TAPs and other products can be found and downloaded at Fix Central
      If you log into this site first - you can then select any of the following specific fixes one at a time

      Fix Central Specific fixes for this (over time these might not be the latest S-TAPs)

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

InfoSphere Guardium

Software version:

8.2

Operating system(s):

AIX, HP-UX, Linux, Solaris

Reference #:

1621903

Modified date:

2013-06-25

Translate my page

Machine Translation

Content navigation