After upgrading from AIX 6.1 to AIX 7.1, the Domino server reported intermittent unreachable errors such as 'Remote system no longer responding' when clients or other servers attempted to connect to it. The error was also reported from several operations, such as replication, sending emails, etc...
The issue was reproducible on any WAN or VPN connection but not on LAN. The network did respond to pings when the issue was occurring.
Blue Coat PacketShaper (Packeteer) configuration settings need to be adjusted.
Resolving the problem
Utilize wireshark sniffers to analyze the network. Debug can also be enabled on the Notes client and after the client was restarted, the issue was reproduced.
From the client clock debug, it was observed that the client was throwing the "remote system no longer responding" specifically for the GETOBJECT_RQST transaction at specific times. Looking at the Wireshark output for these times, there was no evidence of connection resets or dropped packets. In fact, the connections were clean. It was clear we were transferring a large number of bytes (440k+) in a very short period of time; however, the client possibly believed it did not receive the expected number of bytes, so it returned an error.
The customer disabled their Blue Coat's PacketShaper and the issue was no longer occurring. The customer worked with the 3rd party vendor, Blue Coat, and it was found that this was a Blue Coat's PacketShaper TCP shaping issue.
There is a problem with the way that the PacketShaper intercepts TCP window sizing negotiation between the client and server. TCP clipping is the way PacketShapers control TCP window sizing. Some servers might have a problem with this adjustment done on the PacketShaper.
Per Blue Coat support, there were 3 recommendations given. In this case, the customer implemented the 1st recommendation by making the CLI modification which disables TCP clipping and turned shaping back on.
1. Make the following CLI modification which will disable TCP clipping.
PacketShaper# sys set tcpclip 0
The tcpClipInitialWindow setting changes the way Blue Coat handles the initial window size that is negotiated between the TCP client and server. With this setting at '1', the PacketShaper will shape (clip) the initial window. Some TCP client/server have a problem with this initial change in the window size. Setting the tcpClipInitialWindow to '0' disables this initial shaping (clipping) of the window. As more packets come in, shaping will still occur and the window size will be adjusted per the rate controls of the PacketShaper.
If the above is performed and the issue still occurs then there are 2 additional options that can be performed on the PacketShaper.
2. Create a class of traffic based on each of the IP addresses of the servers. Put a policy against this class (try a priority policy first. 4 or higher).
3. Enable traffic discovery within the class and put individual policies on the child classes per your need.
For any questions on the above, please contact Blue Coat support.