Why is CICS Transaction Gateway running out of EXCI pipes and issuing message CTG6882E?
CICS Transaction Gateway running on z/OS is issuing CTG6882E messages like the following:
CTG6882E No pipe available. An attempt was made to open a pipe, but no free receive sessions available. CTG currently holds 86 pipes.
Under load, these messages repeat, with differences seen in the number of pipes held by the gateway.
In addition, requests fail with the following symptoms:
Error Code=ECI_ERR_RESOURCE_SHORTAGE, error code: -16
Note: prior to CICS TG v6, the error message reported would be CCL6882E
The following settings are assumed for the purpose of this document:
MAXWORKER is set to 100 in the CICS TG configuration file,
ReceiveCount is set to 100 in CICS.
LOGONLIM is set to 250.
These settings appear to be sufficient for the gateway to connect 100 EXCI pipes to CICS but the CICS TG appears to run out of pipes before reaching the 100 pipe limit.
This occurs because once a pipe is de-allocated by CICS TG, it is considered immediately available for reuse. This is not the case as CICS must perform some cleanup logic before the pipe becomes available again. This means that the CICS TG view of the number of available pipes and the CICS view of the number of available pipes is not in sync. This typically occurs when a system is heavily loaded and there are a lot of pipe allocates and de-allocates.
The CTG6882E message is issued by CICS TG when it requests an EXCI pipe and CICS returns a
202 NO_PIPE return code, indicating there are no free pipes from a CICS perspective.
Since MAXWORKER controls the maximum number of concurrent requests from a CICS TG perspective, and ReceiveCount controls the number of requests from a CICS perspective, this issue is related to the values set for these two parameters. Setting them to be the same can result in the NO_PIPE message when the system is under stress.
It is suggested to increase the ReceiveCount setting to a number higher than MAXWORKER to create a buffer to allow CICS time to cleanup pipes which have recently been de-allocated while still providing the capability to submit the maximum number of requests to CICS at once.
The specific value for ReceiveCount will depend on the type of work being processed and the transactions per second. Some scenarios have required ReceiveCount to be set to twice the value of MAXWORKER in order to handle the load of peak processing times.
It is possible to avoid these issues altogether. Setting CTG_PIPE_REUSE=ONE in the gateway's STDENV allows EXCI connections to be de-allocated and re-allocated to provide a new
connection to a different CICS region as and when needed. The performance overhead is typically about 0.25 ms cpu/txn on a z990 2084.
The use of this in combination with the setting for max worker threads, such that if MAXWORKER is less than LOGONLIM, will ensure that customers will not need to monitor pipes, as it will not be possible
for CICS TG to allocate more pipes than there are threads, and so the EXCI LOGONLIM can not be breached.
Monitoring EXCI pipe allocation failures with CICS TG statistics
From CICS TG v7.0 and onward it is possible to track the number of times than an EXCI pipes allocation failure has occurred. For EXCI pipe allocation it is the following statistics that need to be monitored:
The number of times the Gateway daemon has tried and failed to allocate a pipe to a CICS server.
The number of times communication with a CICS server has failed after a connection has already been established, or when there has been a failure with the link during communication with the server. In EXCI, a pipe was available but the request failed.
Once a worker thread has been allocated, the Gateway daemon will attempt to allocate an EXCI pipe. If this is unsuccessful for any reason the IALLOCFAIL/LALLOCFAIL statistics will be incremented. This might suggest that the settings for the number of Worker threads (MAXWORKER) is too large for the number of pipes available (as defined by LogonLim) .
If the pipe allocation is successful, the Gateway daemon will attempt to open the allocated pipe. if this is unsuccessful for any reason the ICOMMSFAIL/LCOMMSFAIL will be updated. The CTG6882E scenario explained above is one of the most common examples of a failure at this phase.