Enhanced I/O Error Handling Kernel Services

Enhanced I/O Error Handling (EEH) kernel services is an error recovery strategy for errors that occur during I/O operations on a Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCI-X), or PCI Express (PCIe) bus.

Bridges and switches, PCI-to-PCI or PCI-X-to-PCI-X PCIe switches, that enable each slot to be on its own bus provide a form of electrical and logical isolation of slots. The smallest PCI unit (device or function) that can be assigned to a logical partition (LPAR) is called a Partitionable Endpoint (PE).

For example, if each port of a 4-port Ethernet adapter can be assigned to a different LPAR, and each port is a PCI function, then each port would be a PE. In this case, granularity of a PE is a PCI function. A PE is not necessarily the same as a PCIe Endpoint, and the two should not be confused. A bridge or switch above the PE where the EEH state is maintained forms a PE domain. This bridge or switch is called the Top of PE domain. EEH recovery is performed according to the PE domain and is carried out by the Top of PE domain as directed by the software (operating system and device drivers).

Several PCI functions in one or more adapters that belong to the same EEH recovery domain are called a shared EEH domain. This has been typically limited to a multifunction adapter, in which the functions on the adapter are recovered together. Because a shared EEH domain supports any number of PCI functions to be included in it, including the functions on different adapters, its function is more general than a multifunction adapter. For present purposes, the multifunction model are referred to as the shared EEH model.

In the LPAR environment, a PE domain is the same as a shared EEH domain and includes all PCI functions in the PE domain. In other words, if multiple functions belong to a shared EEH domain, they cannot be individual PEs because the EEH recovery can only affect the LPAR to which the PE belongs.

The bridges can be of different types, such as PCI-to-PCI or PCIX-to-PCI, and so on. A switch is a PCIe switch, which is logically a collection of bridges. The exact type of the bridge or switch is not important to this programming model. These details are handled by the hardware and firmware. A bridged single-function adapter is treated like a bridged multifunction adapter for the purposes of EEH programming model.

The device drivers for all these types of adapters use the same EEH kernel services to drive the error recovery except for the registration service. A nonbridged single-function adapter calls the eeh_init() registration service function. Adapters in a shared EEH domain call the eeh_init_multifunc() function. This includes any bridged or nonbridged multifunction adapters and bridged single-function adapters. Although a nonbridged single-function adapter typically calls eeh_init(), it can choose the shared EEH model and call eeh_init_multifunc() instead. Regardless of the number of functions and bridges, the device drivers should always use the shared EEH model and call eeh_init_multifunc(). The PCIe device drivers are required to use the shared EEH model. Although the same services are used by the single and shared EEH adapter drivers, the error recovery models are different. In addition, any time there are intermediate bridges between the Top of PE domain and the PCI functions in the PE domain, those bridges have to be recovered as well. For example, a multifunction bridged adapter requires that the bridge on the adapter also be recovered.

The key difference in the single-function and shared EEH models is that in the shared EEH model, there is a need for coordination among different driver instances controlling the same PE domain. For example, a PE domain can include a physical device on a single slot. The driver instances controlling each function on the device require coordination. In addition, there are steps in the recovery process that need to be carried out only once. So among all registered drivers in a shared EEH domain, one is chosen to be the master. The drivers follow a state machine. The EEH kernel services are implemented so that they present an EEH recovery state machine to the device drivers. It is the master driver's responsibility to drive the state machine. The section titled Shared EEH Programming Model, which follows, contains more details on how a master driver is determined. Many details are hidden from the device drivers for simplicity. Because the shared EEH model is more flexible and extensible, it is recommended for the new device drivers.

In the single-function model, the device drivers are responsible for driving their own error recovery. In other words, they are responsible for implementing their own state machine. Every time EEH recovery is extended in some way at the hardware or firmware level, there is probably a code and testing impact on the single-function implementations. As previously described, a single-function adapter should still use the shared EEH model. In that case, all the messages from the EEH kernel services are sent to just one driver instance.