Summary of problems monitored and actions taken

When the sysplex monitor detects a problem, an operator message is issued. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement in the TCP/IP profile, the stack leaves the sysplex group, deletes all its dynamic VIPAs, and deactivates all its VIPADYNAMIC definitions. When the detected problem is relieved, the operator message is deleted. If AUTOREJOIN was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, the stack might or might not rejoin the sysplex group, depending on the severity of the problem.

Table 1 summarizes the various problems that are monitored and the specific actions that are taken for each detected problem.

Table 1. Sysplex problem monitoring
Type of check When problem is monitored Action taken when problem detected Action taken if AUTOREJOIN coded and problem is cleared
VTAM® address space is not active DVIPAs advertised and VTAM address space has been inactive greater than TIMERSECS interval
  1. Issue EZZ9671E, VTAM inactive for at least TIMERSECS interval.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when VTAM is started.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
Dynamic or static XCF route, or VIPAROUTE not active DVIPAs advertised, acting as forwarder for a DVIPA, two other MVS™ systems in the sysplex, and either XCF routes (when VIPAROUTE is not specified) or all IP routes (when VIPAROUTE is enabled) to all sysplex partners are not available for TIMERSECS interval
  1. If VIPAROUTE is not enabled, issue message EZZ9673E , dynamic XCF connectivity to all partners not available for at least TIMERSECS interval.
  2. If VIPAROUTE is enabled, issue message EZD1172E , dynamic XCF connectivity and IP routes to all partners are not available for at least TIMERSECS interval.
  3. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  4. Delete the operator message when any VIPAROUTE or XCF route becomes active.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
Network connectivity – critical interfaces are not active
  • MONINTERFACE is specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement and at least one interface is configured with the MONSYSPLEX keyword.
  • The TCP/IP stack participates in sysplex distribution (as a distributor or target) or acts as an owner or a backup for DVIPAs.
  • Another TCP/IP member exists in the sysplex.
  • All monitored interfaces become inactive for the TIMERSECS interval.
  1. Issue message EZD1209E when all monitored interfaces become inactive for at least the TIMERSECS interval.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, leave the sysplex group and delete all VIPADYNAMIC block definitions.
  3. Delete the operator message when any monitored interface becomes active.
  
Network connectivity – dynamic routes are not present over critical interfaces
  • MONINTERFACE or MONINTERFACE DYNROUTE is specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and at least one interface is configured with the MONSYSPLEX keyword.
  • The TCP/IP stack participates in sysplex distribution (as a distributor or target) or acts as an owner or a backup for DVIPAs.
  • Another TCP/IP member exists in the sysplex.
  • No dynamic routes are found over critical interfaces for the TIMERSECS interval.
  1. Issue message EZD1210E when no dynamic routes over critical interfaces are found for at least the TIMERSECS interval
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, leave the sysplex group and delete all VIPADYNAMIC block definitions.
  3. Delete the operator message when any dynamic route over a critical interface is found.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
OMPROUTE heartbeats are no longer being received First heartbeat received, and no heartbeat received for half of the TIMERSECS interval
  1. Issue message EZZ9672E, the OMPROUTE not responsive warning message.
  2. Delete the operator message when heartbeat received, or when message EZZ9678E is issued due to OMPROUTE not responding for at least TIMERSECS interval.
 Not applicable
OMPROUTE heartbeats are no longer being received First heartbeat received, DVIPAs advertised, and no heartbeat received for TIMERSECS interval
  1. Issue message EZZ9678E, OMPROUTE not responsive for at least TIMERSECS interval.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions. If a last heartbeat was not received (OMPROUTE did not terminate properly), abend and dump the TCP/IP and OMPROUTE address space.
  3. Delete the operator message when heartbeat received.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
TCP/IP private storage allocation failures, when POOLLIMIT is not coded on the GLOBALCONFIG statement DVIPAs advertised or this is a DVIPA target, and TCP private storage allocation fails
  1. Issue message EZD1170E, TCP/IP private storage allocation failed message.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when TCP/IP private allocation requests are successful for a full monitor interval (1/4 of the TIMERSECS interval or 1 minute, whichever is less), or delete the operator message when the stack leaves the TCP/IP sysplex group.
 Stack does not rejoin the sysplex group. Use of the VARY TCPIP,,SYSPLEX,JOINGROUP command is required to rejoin the sysplex and restore the saved VIPADYNAMIC configuration.
TCP/IP private critical storage limits reached, when POOLLIMIT values are specified on the GLOBALCONFIG statement DVIPAs advertised or this is a DVIPA target, and TCP/IP private pool is critical for TIMERSECS interval.

If TCP/IP private storage allocation request from MVS fails before the TCP/IP private pool is critical for TIMERSECS interval, the problem is treated as if no POOLLIMIT values were specified.
  1. Issue message EZD1187E, TCP/IP private storage is critical.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when TCP/IP private pool storage exits the critical state.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
TCP/IP ECSA critical storage limits reached, when TCP/IP ECSA storage limits are specified on the GLOBALCONFIG statement DVIPAs advertised or this is a DVIPA target, and TCP/IP ECSA is critical for TIMERSECS interval.

If TCP/IP ECSA storage allocation request from MVS fails before TCP/IP is critical for TIMERSECS interval, the problem is treated as if no TCP/IP limits were specified.
  1. Issue message EZD1187E, TCP/IP ECSA storage is critical.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when TCP/IP ECSA critical state is exited.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
TCP/IP ECSA storage allocation failures, when no TCP/IP ECSA storage limits are specified on the GLOBALCONFIG statement DVIPAs advertised or this is a DVIPA target, and TCP/IP ECSA storage allocation request fails
  1. Issue message EZD1170E, TCP/IP ECSA storage allocation failed message.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when TCP/IP ECSA allocation requests are successful for a full monitor interval (1/4 of the TIMERSECS interval or 1 minute, whichever is less), or delete the operator message when the stack leaves the TCP/IP sysplex group.
 Stack does not rejoin the sysplex group. Use of the VARY TCPIP,,SYSPLEX,JOINGROUP command is required to rejoin the sysplex and restore the saved VIPADYNAMIC configuration.
CSM storage critical DVIPAs advertised or this is a DVIPA target, and CSM critical for greater than TIMERSECS interval
  1. Issue message EZZ9679E, CSM critical message.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when CSM critical state exited.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
CSM storage constrained DVIPAs advertised or this is a DVIPA target, and CSM constrained for greater than three times the TIMERSECS interval
  1. Issue message EZD1974E, CSM constrained message.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when CSM constrained state exited.
 Stack rejoins the sysplex group and reprocesses the saved VIPADYNAMIC configuration
XCF status is not being updated Status not updated for TIMERSECS interval
  1. Issue message EZZ9674E, sysplex processing not responsive message.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave the sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when the monitor is able to run, or delete the operator message when the stack leaves the TCP/IP sysplex group.
 Stack does not rejoin the sysplex group. Use of the VARY TCPIP,,SYSPLEX,JOINGROUP command is required to rejoin the sysplex and restore the saved VIPADYNAMIC configuration.
Abend in sysplex code  
  1. Issue message EZZ9670E, sysplex processing encountered nonrecoverable error.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when the stack leaves the TCP/IP sysplex group.
 Stack does not rejoin the sysplex group. Use of the VARY TCPIP,,SYSPLEX,JOINGROUP command is required to rejoin the sysplex and restore the saved VIPADYNAMIC configuration.
Five abends in less than 1 minute  
  1. Issue message EZD1973E, multiple nonrecoverable errors are adversely affecting sysplex processing.
  2. If RECOVERY was specified on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement, and this TCP/IP stack is not the only member of the TCP/IP sysplex group, leave sysplex group, deactivate all DVIPAs, and save all VIPADYNAMIC block definitions.
  3. Delete the operator message when the stack leaves the TCP/IP sysplex group.
 Stack does not rejoin the sysplex group. Use of the VARY TCPIP,,SYSPLEX,JOINGROUP command is required to rejoin the sysplex and restore the saved VIPADYNAMIC configuration.
Parent topic: Sysplex problem detection and recovery