This problem can occur when all of the following conditions apply:
1. The maxQueuedTransfers property has been set on the source agent to a large number, for example, 2000.
2. A resource monitor has been created with a trigger condition which matches a large number of files.
For example, the trigger condition is ' -tr match,"*.txt" ' and the monitored directory contains 2000 .txt files.
In this situation, 2000 transfers will be initiated (one for each matching file).
Assuming the rest of the agent properties have not been changed, the source agent will attempt to start 25 of the transfers and queue the remaining 1975 transfers.
By the time the source agent has queued the remaining transfers, the requests to start the initial 25 have timed out at the destination agent and they move into the recovering state. Because of the way the agents handle a delay at the other side of the transfer, the 2000 transfers are completed successfully but only after being put into 'recovering' state first, resulting in a long delay in the transfer.
Although this scenario describes a problem caused by a resource monitor initiating a large number of transfers, it could also occur if a large number of transfer requests are manually submitted to an agent at the same time.
When the transfer requests arrive at the source agent, it sends command messages to the destination agent to negotiate the start of the first 25 (the default value of maxSourceTransfers) transfers. When the destination agent replies, reply messages are sent back to the source agent to notify it that the transfers can begin.
The replies from the destination agent are not dealt with immediately because the source agent is still queueing the remaining 1975 transfer requests. By the time all the transfer requests have been queued and the source agent has read the reply messages, the destination agent deems the current transfer requests to have timed out and puts them into recovering state. The agents are then delayed in negotiating any of the recovering or queued transfers and the transfers stay in queued or recovering state for a long time before eventually completing.
Diagnosing the problem
A large number of transfers will be submitted to an agent, but no file transfers will occur.
Resolving the problem
To allow the agents more time to negotiate the transfers, set the agent property maxTransferNegotiationTime in the agent.properties file of the source and destination agents. This prevents the initial transfers from going into recovery state so quickly and allows those transfers the opportunity to complete without going into recovery state at all. The scenario above has been run in a test environment, and a value of 300000 milliseconds was found to be sufficient to allow the source agent to queue all transfer requests and complete negotiation of the initial transfers without timing out. Each of the remaining queued transfers began when one of the previous transfers had completed, until all of the queued transfers had completed successfully.
The value you choose for maxTransferNegotiationTime will depend on your environment. It will depend on the number of transfers a source agent is queueing, but may also depend on the amount of other activity occurring at the source agent. For example, if a large number of resource monitors or scheduled transfers have been created on an agent, the value may need to be increased to allow for these other activities.
WMQ MQ WMQFTE FTE