Blocking, Sequential Only, and the Effect on a Program

Resolving The Problem

This is a two-part document. Part 1 deals with general considerations on blocking or record buffering and programs. Part 2 deals with enhancements to the Force End Of Data function in IBM® RPG/400® at R520 and later releases.

Part 1

All high-level language programs (HLLs) use blocking at certain times and use single record I/O at other times, based on program specifications. Because blocking takes less system resources to perform a single I/O, a program that blocks performs better and uses less system resources. The default for the HLL uses record blocking if opening a file for output only (write) or input only (read). The programmer is expected to override the file or use larger blocks if the default is not appropriate. The HLLs do single I/O if files are opened for both input/output or update/delete. Blocking works well only if one reads many records sequentially because the block of records read are from some position in the file and the next N records. If one is going to position randomly within the file (CHAIN or set lower limit (SETLL) in RPG) and then read only two records, blocking might actually be a waste of system resources because the read after the random position reads a single record; however, the next read reads a block of records and the program reads only one.

The parameter on the override command that expresses the intent to block is sequential only (SEQONLY) to emphasize that the intent is to read some number of records in a row (you choose the number). Sequential only does not preclude the program from doing random positioning; however, remember the system reads an entire block of records when the second READ is run after the position (the first read accesses a single record only).

Blocking is controlled by the defaults that are established at compile time and by overrides that are run at run time. The compiler places an information message into the compiler listing if a file will be opened in blocking (sequential only) mode. Blocking is established at full open for the file that is being opened, and all reads (READ verbs, not random positioning verbs such as CHAIN) get an entire block of records. Blocking is performed when the first I/O is performed after the open. The system reads a block of records into the system buffer (located in the Open Data Path (ODP) for that file) when the first read in the user's program is run and then moves the first of these records into the program buffer so the program can process that data. When the program next performs a read, the next record in the block is moved into the program buffer by the program (no system calls are made). When the block no longer contains records that the program has not processed, another call to the system is made, and another block of records is placed into the system buffer, the first of these is placed into the program buffer, and the cycle continues until the file is closed or the end of file is reached. If the program positions in the file and then does a read, the old block of records is discarded, and a new block of records is obtained.

Writes are essentially the same. The program fills the system buffer with some number of writes before a call to the system to write the block and the final block being written at file close time (if the last write did not happen to occur at a block boundary).

The most common problem reported with programs that block records is the timeliness of the data in the block. For example, if a program reads a block at 8 a.m. with 8 records in the buffer and the program reads one record an hour, the data that the program sees at 4 p.m. is really the data as it looked at 8 a.m. rather than the state of the data at 4 p.m. A similar program that writes a record an hour would hold all 8 records in its buffer until the 5 p.m. write. Someone might complain that they know that the record was written in the morning. For example, a program written to wait for records to arrive using the EOFDLY parameter on the OVRDBF command will not see the 8 new records until the block is written at 5 p.m. In fact, no jobs will have access to the 8 new records until the write at 5 p.m. While the timeframe in the example is somewhat extreme, the blocks of data represent the state of a system at block boundaries rather than at read/write boundaries (as most programmers have been taught to believe).

Traditional batch processing, which usually has no long waits and is often run when there are no others accessing the data, is ideal for blocking while interactive real time database access often requires a real time view of data and should closely control blocking if it is done at all.

Note: Blocking is not out of the question in this case. In an application that presents an entire screen of data, the all at one time nature of the blocked read might be exactly what is required.

The use of the blocking can greatly improve performance, and most applications are tolerant to minor timing problems. Batch program defaults that systems use are often too small because they are often a compromise between what is best for batch and keeping blocks within reasonable size to minimize timing problems.

If the compiler has defaulted the open for sequential only, the system chooses a block size such that the number of records fits in a block of 4K. This default relates to the IBM® System/38™ implementation and has never been changed. Making this blocking factor larger typically greatly improves batch performance but it might make some interactive application worse; therefore, it was not changed.

The override database file (OVRDBF) command is used to control both the size of the block and to turn blocking on and off.

The sequential only (SEQONLY) parameter can override the program specification of blocking to not block (SEQONLY *NO) or change the size of the block to match the program intent:

Interactive - To match the size of an interactive screen (presumes the application presents eight records on a screen) one set of data presented to a user interactively (SEQONLY *YES 8). Most interactive applications' writes are probably best coded as immediate writes (SEQONLY *NO). An alternative commonly mentioned is to use the FEOD operation to force the buffer (and while this does force the writes to the database, it also causes the data to be forced to secondary storage (DASD) and makes it likely that the page on which the data resides will be stolen). FEOD is almost always less efficient from a system-wide perspective than using single writes.

Some programmers choose to prevent blocking by changing the physical file force write ratio to 1. This is typically done because it forces the system to put every write to disk. If force write ratio of 1 was chosen for data recovery, journaling the file is preferred.

Batch - To change the number of records in the block to match the maximum size of a single system I/O operation. The value that will usually result in the best system performance, has proven to be the size that is closest to the maximum size of a single system disk I/O. The size for CISC system is 32,767 (32K), and for RISC systems is 131,072 (128 K). The SEQONLY parameter is expressed in the number of records. So one must calculate number of records from the ideal block size by knowing the physical record length and dividing that into the I/O size. The calculation of physical record length can be somewhat difficult. The length of the data, at least an additional database status byte, and the record might have additional information on variable length fields, or null value indicators. A reasonable approximation is 32000(128000)/rcdlen+1 if no nulls or variable length, else 28000(110000)/reclen. The best length is one that is less than the machine maximum.

There is a second parameter on the OVRDBF command called number of records (NBRRCDS). This parameter does not control blocking and unblocking at the program level. Rather, it is a database directive that instructs the database to initiate an asynchronous read of the next block of storage in the physical file (in physical file arrival order). In essence, this is a directive to start double buffering of the physical block of records off of DASD, and the I/O is performed while the application is processing the current block (for example, asynchronously). If the data is being processed in physical order, this parameter can improve the efficiency of the processing of that data. If the file is opened with sequential only and processing is arrival order, database open uses the same value for (default) NBRRCDS as specified in the sequential-only parameter. If the file being processed is keyed, the NBRRCDS parameter is not defaulted. This parameter can be specified if the physical order matches the key order or the file is opened in arrival order but the options do not allow sequential only (for example, file is opened for update, but the file will be processed via sequential reads). If the file is processed by a keyed order file and the physical order of the data are not the same as the keyed order, coding the NBRRCDS parameter will cause the system to bring (physically buffer) data that has very little likelihood of being referenced; in other words, causes worse performance than if the parameter were not coded.

Part 2

ILE RPG V5R2 Enhancements

Performance option for FEOD: The FEOD operation is enhanced by offering an option to write out the blocked buffers locally without forcing a costly write to disk.

Use of the new option for FEOD is in the manual IBM® WebSphere® Development Studio ILE RPG Reference, Version 5, SC09-2508-04.