IBM Security Systems has discovered a critical issue that can cause the Gx7 series of appliances to induce high amounts of network latency.
The issue is caused when the analysis module on the Gx7 crashes more than three times in a 30 minute period. A technical description of the behavior is described below:
The Gx7 uses NPUs (Network Processing Unit) to speed up inspection of traffic. The NPUs have queues where packets are placed for inspection.
Typically, if the analysis module crashes, the engine will automatically restart. However, If the analysis module or inspection engine crashes more than 3 times in 30 minutes, the inspection engine will not restart.
If this occurs, even though the the inspection engine has stopped, the Gx7 driver will continue to queue packets to the NPU for inspection. The NPU will hold onto these packets for a maximum of 5 milliseconds while waiting for a response from the inspection engine. If the NPU does not receive a response within 5 milliseconds, it will either forward the unanalyzed packet or drop the unanalyzed packet based on the adapter policy settings. In addition to this, if the NPU inspection queue is full, new packets will also be handled according to the adapter policy settings.
When this situation arises, with both packets in queue waiting for 5 milliseconds and packets overflowing the queue being fast forwarded, it can lead to a large number of out of order packets with large time frames between the out of order packets. This results in high network latency.
Please note that the engine restart behavior is controlled by the following advanced parameters so your behavior may vary depending on these values. The default values are listed below:
engine.restart.count = 3
engine.restart.interval = 1800
This issue is addressed in hotfixes ( 22.214.171.124-ISS-ProvG-AllModels-Hotfix-FP0003 and 126.96.36.199-ISS-ProvG-GX7-Hotfix-FP0003) or later. Both the AllModels and GX7 hotfixes are required for this functionality to bypass the NPU queueing of packets when the maximum engine restart threshold is exceeded. For reference, this fix is included in firmware 4.6.
Our Gx series patches are built cumulatively. The fixes from the AllModels Hotfix FP0001 are included in AllModels Hotfix FP0002 and so on. The GX7 Hotifx Fix Pack delivers all GX7 specific component fixes to date and the AllModels hotfix Fix Pack is intended to deliver packages that are common to all models of the IPS. Please double check Technote articles and Fix Central for updated Fix Packs that may contain additional fixes, even if you are running a later firmware than firmware 4.5. Contained within each download is a readme file and a TGZ hotfix file. The readme file contains instructions for applying the hotfix as well as descriptions of the included fixes.
There are several critical issues resolved in these Fix Packs, including inspection engine related crashes. In addition, there are many other critical NPU, driver and inspection engine issues resolved between firmware versions. It is IBM's recommendation that all GX7 appliances be updated to the most current firmwasure and then patched to the latest GX7 and AllModels Fix Packs.
Fix Packs for firmware 4.5:
If the above information does not resolve your issue, please contact IBM Security Systems Technical Support.