High Availability Setup for the Incremental Update Capture Engine

Content

Note that this document only describes a high availability setup for the incremental update function of IBM DB2 Analytics Accelerator for z/OS. For the general network setup of IBM DB2 Analytics Accelerator for z/OS, see the following documents:

• Network Requirements for IBM DB2 Analytics Accelerator for z/OS
• Network Configurations for IBM DB2 Analytics Accelerator for z/OS
• Network Connections for IBM DB2 Analytics Accelerator for z/OS

When an instance enters the active mode, it initializes its own TCP/IP task, which in turn starts a BIND operation that associates a TCP port with a TCP/IP stack. For the z/OS Communication Server, this BIND operation is a request to pass control to the TCP/IP stack of the LPAR whose IP address is specified in the BIND command. The entire operation is carried out in a Dynamic Virtual IP Addressing (DVIPA) environment.

If the CDC instance that is currently active becomes unavailable, the TCP/IP address of this instance is passed to one of the the Hot Standbyinstances. As a result, one of the Hot Standby instances goes into active mode.

The diagram below illustrates this setup with a DB2 data sharing group consisting of two members in different LPARs. The data sharing group is called DWCDBZA; its members are called SZA1 and SZA2. The member SZA1 runs in an LPAR called DWC1, whereas the other member, SZA2, runs in an LPAR called DWC2. Both LPARs belong to the same sysplex. Separate instances of the CDC replication agent are running in each LPAR. The TCP/IP stacks of both LPARs are named TCPIP. The connected accelerator hardware, an IBM PureData System for Analytics N1001 with the name dwatf0112, communicates with the active instance over DVIPA address 10.101.8.249.

Infosphere CDC for z/OS Installation

Follow the IBM InfoSphere Change Data Capture Installation Documentation to install and run the Infosphere CDC instance on each LPAR of your DB2 data sharing group. Make sure that the latest maintenance, especially for PM82593 and PM85740, has been applied.

Ensure that the VSAM Meta-data cluster (the name is defined by replacing the <MetaDatacluster> placeholder in the CHCDFMTD installation job) has the value (3 3) for the SHAREOPTIONS keyword. The VSAM Meta-data cluster is referenced by the CHCMTDTA DD name in the product startup JCL and must be shared across the active and all corresponding hot-standby instances. All instances using the same meta-data tables (identified by the owner) within a DB2 subsystem or Data Sharing Group must reference the same VSAM Meta-data cluster.

If a log cache is used, the VSAM cluster for the L2 cache that is created by the CHCCRCCH installation job (named <CACHE.QUALIFIER>.CHCCACHE) must be created with value (2) for the SHAREOPTIONS keyword and must also be shared across the active and all hot-standby instances.

Similarly, the PAL (product administration log) VSAM cluster that is created by the CHCDFPAL installation job (named <PALcluster>) must be created with the value (2 3) for the SHAREOPTIONS keyword (see APAR PI54370) and must also be shared across the active and all hot-standby instances.

You can use the ARMWRAP program to ensure that the address space is restarted in case it fails unexpectedly. See the following IBM z/OS Automatic Restart Manager Redpaper for more information .

The IP configuration for the Infosphere CDC for z/OS instances are defined in the CHCCMMxx member (xx is the suffix that you chose during the installation). In this example, the instances use TCP port 5999, which is defined by the SERVICENAME keyword, and a TCP/IP stack name TCPIP, which is defined by the SUBSYSTEM keyword.

TCP/IP SERVICENAME=5999,
      SUBSYSTEM=TCPIP

To verify this step, start Infosphere CDC and review the CHCPRINT data set in the SPOOLed output. The output must contain the following information:

CHC9512I Using TCP/IP Subsystem address space TCPIP
CHC9523I TCP/IP Communication Support is active
CHC5099I CMO Task was initialized successfully
CHC6804I TCP Listener was initialized on service name 5999, port  5999

z/OS Communications Server Setup

To reserve the port 5999 for the Infosphere CDC instances and to associate the port number with the dynamic VIPA address 10.101.8.249, use the following statement in the PROFILE TCPIP data set for the TCP/IP stacks of both LPARs. Use different names for the CDC address space in each stack (CDCSZA1 is the name of first address space in this example. CDCSZA2 is the name of the second address space. You would have to enter CDCSZA2 when defining the second stack).

Important: The DVIPA IP address must belong to the same subnet as the accelerator that you want to enable for incremental updates.

Example:

PORT                                                                            
  5999 TCP CDCSZA1 BIND 10.101.8.249; CDC AGENT BIND TO DVIPA IP ADDRESS

The VIPADYNAMIC keywords sets the operation mode to dynamic. Because you need only one DVIPA address (10.101.8.249), you can use the smallest possible subnet (mask 255.255.255.252). This subnet gives you two possible client addresses: 10.101.8.249 and 10.101.8.250. Only the first one is used.

VIPADYNAMIC     ; VIPA DEFINITIONS
VIPARANGE MOVEABLE DISRUPTIVE 255.255.255.252 10.101.8.248
ENDVIPADYNAMIC  

Note: 10.101.8.248 is the network address. For the definition of the VIPARANGE, this is the correct statement.

Verify the DVIPA configuration by entering the following command:

DISPLAY TCPIP,TCPIP,NETSTAT,VIPADCFG

The output of this command looks like this:

DWC1     12342 14:38:32.08 STC01367 00000010  EZD0101I NETSTAT CS V1R12 TCPIP 234
                                             DYNAMIC VIPA INFORMATION:
                                               VIPA RANGE:
                                                 IPADDR/PREFIXLEN: 10.101.8.248/30
                                                   MOVEABLE: DISRUPT
                                             END OF THE REPORT

Verify the PORT configuration by entering the following command:

DISPLAY TCPIP,TCPIP,NETSTAT,PORTL

The output of this command looks like this:
DWC1     12342 14:39:18.02 STC01367 00000010  EZD0101I NETSTAT CS V1R12 TCPIP 23
                                             PORT# PROT USER FLAGS RANGE SAF NAME
                                             5999 TCP CDCSZA1 DABU
                                                  BINDSPECIFIC: 10.101.8.249
                                             ...

You can display the current DVIPA status of a system by using the following command:

DISPLAY TCPIP,TCPIP,NETSTAT,VIPADYN

This output of this command looks like this:

DWC1     12342 14:41:04.12 STC01367 00000010  EZD0101I NETSTAT CS V1R12 TCPIP 252
                                             DYNAMIC VIPA:
                                               IPADDR/PREFIXLEN: 10.101.8.249/30
                                                 STATUS: ACTIVE ORIGIN: VIPARANGE BIND DISTSTAT:
                                                 ACTTIME: 12/07/2012 13:26:33 JOBNAME: CDCSZA1
                                             1 OF 1 RECORDS DISPLAYED
                                             END OF THE REPORT

This example shows that CDC Agent CDCSZA1 is running and uses the DVIPA address 10.101.8.249 .

Verify the current DVIPA status of a sysplex by entering the following command:

DISPLAY TCPIP,TCPIP,SYSPLEX,VIPADYN

This output of this command looks like this:             
DWC1     12342 14:43:52.32 STC01367 00000010  EZZ8260I SYSPLEX CS V1R12 105                      
                                             VIPA DYNAMIC DISPLAY FROM TCPIP    AT DWC1        
                                             LINKNAME: VIPL0A6508F9                            
                                             IPADDR/PREFIXLEN: 10.101.8.249/30                  
                                               ORIGIN: VIPARANGE BIND                          
                                               TCPNAME  MVSNAME  STATUS RANK DIST              
                                               -------- -------- ------ ---- ----              
                                               TCPIP    DWC1     ACTIVE                        
                                             1 OF 1 RECORDS DISPLAYED

The example shows that the DVIPA address 10.101.8.249 is used by an application in LPAR DWC1.

Testing failover

When you start the first instance of the Infosphere CDC replication agent, which will become the active instance, verify that the dynamic VIPA address is created correctly. This is indicated by the following message, which is displayed during the start procedure:

EZD1205I DYNAMIC VIPA 10.101.8.249 WAS CREATED USING BIND BY CDCSZA1 ON TCPIP

All subsequently started instances will enter the Hot Standby mode. This is indicated by the following message, which is shown at the start of each additional instance:

CHC9212I The meta data owned by <owner> is being used by
        another instance of InfoSphere CDC for z/OS, this
        instance is entering Hot Standby mode

If the active instance ends abnormally or the system on which it is running becomes unavailable, one of the Hot Standby instances takes over. This is indicated by the following message:

CHC9211I Termination of Active instance of InfoSphere CDC for
        z/OS using meta data owned by <owner> detected, this
        instance is leaving Hot Standby mode and will become
        Active

To initiate a failover manually, you can use the HANDOVER parameter of the SHUTDOWN command. The syntax of the command is as follows:

MODIFY <A/SName>,SHUTDOWN,[SCHED=<DateTime>|NORM|IMMED|ABORT],[HANDOVER]

where <A/SName> is the name of the instance of the Infosphere CDC replication agent.

• If you shut down the active instance with the HANDOVER parameter, one of the Hot Standby instances reacts as though the active instance had been abnormally terminated, and becomes the new active instance.
• If you use this command for the active instance without the HANDOVER parameter, or issue a z/OS STOP command for this instance, the active instance and all Hot Standby instances are shut down.
• If you use this command for a Hot Standby instance or a issue a z/OS STOP command for such an instance, only the specified Hot Standby instance is shut down.

The following example shows the output of a SHUTDOWN command invocation with the IMMED and HANDOVER parameters for the CDCSZA1 instance.

Command:

MODIFY CDCSZA1,SHUTDOWN,IMMED,HANDOVER

Message output:     
DWC1     12342 15:01:37.08 STC01367 00000010  EZD1298I DYNAMIC VIPA 10.101.8.249 DELETED FROM TCPIP                
DWC1     12342 15:01:37.08 STC01367 00000010  EZD1207I DYNAMIC VIPA 10.101.8.249 WAS DELETED USING CLOSE API BY    
                                             CDCSZA1 ON TCPIP                                                      

DWC2     12342 15:01:37.14 STC01855 00000010  +CHC9211I (CDCSZA2) Termination of Active instance of IBM InfoSphere
                                             CDC for z/OS using meta data owned by CD ...                        
DWC2     12342 15:01:37.14 STC01855 00000010  +CHC9211I (CDCSZA2) ... CUSER detected, this instance is leaving Hot
                                             Standby mode and will become Active                                  
DWC2     12342 15:01:37.49 STC01459 00000010  EZD1205I DYNAMIC VIPA 10.101.8.249 WAS CREATED USING BIND BY CDCSZA2 ON
                                             TCPIP                      

Note: The goal of this setup is to keep replication active. If the Active instance's DB2 subsystem is lost, FailOver occurs automatically in order to continue replication. If the Active instance itself is lost, a standby instance will automatically become the new active instance. The purpose of the HANDOVER keyword is only to trigger a manual FailOver, i.e. FailOver not caused by an unexpected loss of DB2 or the CDC address space.