Case study

This is a case study that illustrates some of the items discussed.

An application was invoked that was changed to use coupling facility signaling. When the workload was increased XCF delays increased. This was evident from messages like ERB463I which indicated the RMF™ Sysplex Data Server was not able to communicate with another system because the XCF signaling function was busy.

Looking at RMF Monitor III, it showed the information in Figure 1.

Figure 1. Example: RMF Report - XCF Delays

                   RMF 1.3.0  XCF Delays
Samples: 120   System: J90   Date: 02/07/97   Time: 13.03.00
                                                                 
              Service    DLY    ----------Main Delay Path(s)
Jobname   C   Class       %     %  Path    %  Path   %   Path
WLM       S   SYSTEM     87    87  -CF-         
*MASTER*  S   SYSTEM     10    10  -CF-    
RMFGAT    S   SYSTEM      3     3  -CF-
JESXCF    S   SYSTEM      1     1  C601

Comparing the RMF XCF reports to some other reports, it was noticed the amount of XCF traffic had quadrupled and the increase was in the class with the larger CLASSLEN (DEFAULT on this system).

In order to protect other XCF users and to investigate what was happening, a decision was made to separate these messages into their own transport class. A new transport class, NEWXCF, was defined using the GROUP keyword to specifically assign messages from the new application to this class. Since it was known the messages were bigger than the transport class with the smaller CLASSLEN (DEFSMALL), using guess work, it was decided the messages might fit into a 4K(-68) buffer. The report shown in Figure 2 was generated.

Figure 2. Example: RMF Report - Defining New Transport Class

                                              ---- BUFFER ----        ALL
TO      TRANSPORT   BUFFER      REQ     %      %      %      %      PATHS    REQ    
SYSTEM  CLASS       LENGTH       OUT    SML    FIT    BIG    OVR   UNAVAIL  REJECT    
JA0     DEFAULT     20,412     2,167     92      8     <1    100         0       0
        DEFSMALL       956    29,730      0    100      0      0         0       0 
        NEWXCF       4,028   106,018      0      0    100      0         0       0
JB0     DEFAULT     20,412     6,132     97      3     <1    100         0       0
        DEFSMALL       956    82,687      0    100      0      0         0       0 
        NEWXCF       4,028    18,085      0      0    100      0         0       0

Since all the NEWXCF messages were too big, the CLASSLEN was increased. Figure 3 shows the results.

Figure 3. Example: RMF Report - Increasing CLASSLEN

                                              ---- BUFFER ----        ALL
TO      TRANSPORT   BUFFER      REQ     %      %      %      %      PATHS    REQ    
SYSTEM  CLASS       LENGTH       OUT    SML    FIT    BIG    OVR   UNAVAIL  REJECT    
JA0     DEFAULT     20,412     1,715     90     10      0      0         0       0
        DEFSMALL       956    37,687      0    100      0      0         0       0 
        NEWXCF       8,124   103,063      0    100      0      0         0   3,460
JB0     DEFAULT     20,412     2,075     92      8      0      0         0       0
        DEFSMALL       956    38,985      0    100      0      0         0       0 
        NEWXCF       8,124   117,727      0    100      0      0         0     195

Now all the messages fit, but some are being REJECTed. This suggests message buffer space for the outbound path is no longer large enough. As Figure 4 shows, the XCF path statistics confirm outbound messages are queuing up.

Figure 4. Example: Insufficient Message Buffer Space for Outbound Path

TO      Y DEVICE, OR      TRANSPORT     REQ   AVG Q
SYSTEM  P STRUCTURE       CLASS         OUT   LNGTH    AVAIL   BUSY    
JA0     S IXCPLEX_PATH1   DEFAULT     1,715    0.00    1,715      0      
        S IXCPLEX_PATH2   DEFSMALL      486    0.00      486      0        
        S IXCPLEX_PATH3   NEWXCF    103,063    1.42  102,818    245         
        C C600 TO C584    DEFSMALL   13,684    0.00   13,644      0      
        C C601 TO C585    DEFSMALL   13,603    0.00   13,603      0
        C C602 TO C586    DEFSMALL   12,610    0.00   12,610      0
JB0     S IXCPLEX_PATH1   DEFAULT     2,075    0.00    2,075      0      
        S IXCPLEX_PATH2   DEFSMALL      737    0.00      737      0        
        S IXCPLEX_PATH3   NEWXCF    117,727    1.26  117,445    282         
        C C610 TO C584    DEFSMALL   16,391    0.00   16,391      0      
        C C611 TO C585    DEFSMALL   12,131    0.00   12,131      0
        C C612 TO C586    DEFSMALL   12,294    0.00   12,294      0

Increasing the MAXMSG on the PATHOUT for the NEWXCF transport class from 1000 tp 2000 clears up the queuing delays. See Figure 5.

Figure 5. Example: Increasing the MAXMSG on the PATHOUT

                                              ---- BUFFER ----        ALL
TO      TRANSPORT   BUFFER      REQ     %      %      %      %      PATHS    REQ    
SYSTEM  CLASS       LENGTH       OUT    SML    FIT    BIG    OVR   UNAVAIL  REJECT    
JA0     DEFAULT     20,412     2,420     93      7      0      0         0       0
        DEFSMALL       956    41,215      0    100      0      0         0       0 
        NEWXCF       8,124   133,289      0    100      0      0         0       0
JB0     DEFAULT     20,412     2,362     93      7      0      0         0       0
        DEFSMALL       956    39,302      0    100      0      0         0       0 
        NEWXCF       8,124   143,382      0    100      0      0         0     195

The BUSY conditions are reduced, and more importantly the AVG Q LNGTH has been greatly reduced. Since the pathout with the contention is a coupling facility structure AVG Q LNGTH is an appropriate metric to use when tuning. See Figure 6.

Figure 6. Example: BUSY Conditions Reduced

TO      Y DEVICE, OR      TRANSPORT     REQ   AVG Q
SYSTEM  P STRUCTURE       CLASS         OUT   LNGTH    AVAIL   BUSY    
JA0     S IXCPLEX_PATH1   DEFAULT     2,420    0.00    2,420      0      
        S IXCPLEX_PATH2   DEFSMALL      361    0.00      361      0        
        S IXCPLEX_PATH3   NEWXCF    133,289    0.08  133,117      2         
        C C600 TO C584    DEFSMALL   12,700    0.00   12,700      0      
        C C601 TO C585    DEFSMALL   16,421    0.00   16,421      0
        C C602 TO C586    DEFSMALL   14,173    0.00   14,173      0
JB0     S IXCPLEX_PATH1   DEFAULT     2,362    0.00    2,362      0      
        S IXCPLEX_PATH2   DEFSMALL    1,035    0.00    1,033      2        
        S IXCPLEX_PATH3   NEWXCF    143,383    0.09  143,087    296         
        C C610 TO C584    DEFSMALL   12,647    0.00   12,646      1      
        C C611 TO C585    DEFSMALL   15,944    0.00   15,944      0
        C C612 TO C586    DEFSMALL   12,183    0.00   12,182      1

When determining how to tune the application to limit the number of XCF messages, a DEF8K transport class for UNDESIG messages was created and the NEWXCF class assigned to this application was eliminated.

Note: In this case study, the messages were being queued because the message buffer space was too small. If, instead of REJECTS, there was a high percentage of messages marked as BUSY, then increasing the number of signaling paths would have been appropriate.

Incidentally the path associated with the NEWXCF was a coupling facility structure that used the new HiPerLinks available on the G3 server. The structure was chosen since it was quicker and easier to implement. Since the structure was receiving over 500 req/sec, it was unclear if the structure could handle the traffic. As can be seen from the queue lengths, it was capable of handling this rate.