Wait times for FCM communications

In a multi-partition database, or in an environment where there is intrapartition parallelism, the Fast Communications Manager (FCM) manages communication between different agents working on the same statement, whether those agents are on the same or on different members. All FCM communications involve the potential for wait times as one agent waits for work to be done by another, or for data to transferred from one to another.
FCM-related wait times do not necessarily indicate that processing is blocked across members; for a given statement, work might be proceeding in parallel or serially on sub-agents across members. The FCM-related wait times show the time that an agent is blocked on a single member waiting for another; however, work might very well be proceeding on the other member.

For example, Agent A on member 0 might be blocked waiting for Agent B on member 1 to read the data being sent to it. If Agent B is busy and does not receive the data from the table queue immediately, Agent A will only be allowed to send a limited amount of data before being forced to wait for an acknowledgement from Agent B before sending the remaining data. This wait time is counted as fcm_tq_send_wait_time by Agent A.

Figure 1. Wait times in FCM communications
FCM wait times across different agents
Another scenario might involve an agent on one member dispatching a request to an agent on another member. fcm_message_recv_wait_time will be incurred if one of the following situations takes place:
  • Agent A sends a lengthy request to agent B, and agent B is forced to wait for the full request to be received. In this case, agent B incurs fcm_message_recv_wait_time.
  • Agent A sends a request to agent B, and awaits a reply from agent B. In this case, agent A incurs fcm_message_recv_wait_time.
fcm_message_send_wait_time will be incurred if one of the following situations takes place:
  • Agent A sends a lengthy request to agent B, and is blocked for some reason. For example, Agent A might need to wait while the first portion of the request being sent is processed by the local FCM daemon. In this case, agent A incurs fcm_message_send_wait_time.
  • Agent B sends a reply to a request from agent A. If agent B is for some reason blocked before the entire message can be sent, then agent B incurs fcm_message_send_wait_time.

Depending on what it is that you want to measure, it might be appropriate to subtract FCM wait times from the total time spent if you are aggregating metrics for time spent across multiple partitions.