Is a patch available to correct latency issues on the GX7 series appliances?
IBM Security Systems has discovered a critical issue that can cause the GX7 series of appliances to induce high amounts of network latency. The issue occurs when the analysis module on the GX7 crashes more than three times in a 30 minute period. A technical description of the behavior is as follows:
The GX7 uses NPUs (Network Processing Units) to speed up inspection of traffic. The NPUs have queues where packets are placed for inspection.
Typically, if the analysis module crashes, the engine will automatically restart. However, If the analysis module or inspection engine crashes more than 3 times in 30 minutes, the inspection engine will not restart.
If this occurs, the GX7 driver will continue to queue packets to the NPU for inspection even though the inspection engine has stopped. The NPU will hold these packets for a maximum of 5 milliseconds while waiting for a response from the inspection engine. If the NPU does not receive a response within 5 milliseconds, it will either forward the unanalyzed packet or drop the unanalyzed packet based on the adapter policy settings. In addition to this, if the NPU inspection queue is full, new packets will also be handled according to the adapter policy settings.
When this situation arises, with both packets in queue waiting for 5 milliseconds and packets overflowing the queue being fast forwarded, it can lead to a large number of out of order packets with large time frames between the out of order packets. This results in high network latency.
Please note that the engine restart behavior is controlled by the following advanced parameters so your behavior may vary depending on these values. The default values are listed below:
This issue is corrected in firmware 4.6 and higher. It is IBM's recommendation that all GX7 appliances be updated to the most current firmware and then patched to the latest "GX7" and "AllModels" fix packs.
If, for some reason, it is not possible to update the appliance to 4.6 or higher, this issue is addressed in the following fix packs for firmware 4.5:
Both the "AllModels" and "GX7" hotfixes are required for the functionality to bypass the NPU queueing of packets when the maximum engine restart threshold is exceeded. This fix is included in firmware 4.6 for the GX7.
GX series patches are built cumulatively. The fixes from the AllModels Hotfix FP0001 are included in AllModels Hotfix FP0002 and so on. The "GX7 hotfix" fix pack delivers all GX7 specific component fixes to date and the "AllModels hotfix" fix pack is intended to deliver packages that are common to all models of the IPS. There are several critical issues resolved in these Fix Packs, including inspection engine related crashes. In addition, there are many other critical NPU, driver and inspection engine issues resolved between firmware versions.
Fix Packs for firmware 4.5