PCIe3 1.92 TB CAPI NVMe Flash Accelerator Adapter (FC EJ1K; CCIN 58CD)

Learn about the specifications and operating system requirements for the feature code (FC) EJ1K adapter.

Overview

The PCIe3 CAPI NVMe Flash Accelerator Adapter with 1.92 TB Flash Storage is a PCI Express (PCIe) generation 3 (Gen3) x8 adapter. The adapter must be placed in a POWER8® CAPI-capable PCIe3 x16 slot in the system. The PCIe3 CAPI NVMe Flash Accelerator Adapter acts as a co-processor for the POWER8® processor, and is designed to offload CPU access to built-in NVMe-attached flash storage. If the maximum write capability is achieved, the adapter's replacement or any NVMe M.2 module's replacement is not covered under IBM® warranty or maintenance. In order for the adapter to function properly, all of the administration prerequisites must be satisfied and critical activities completed.
Important: Graphics processing units (GPUs) and Coherent Accelerator Processor Interface (CAPI) adapters cannot be replaced with the power turned on.
Additional information about the adapter:
  • The adapter contains a CAPI accelerator and is connected to two NVMe drives.
  • Each NVMe drive is presented as a separate volume to the operating system.
  • Each volume is 960 GB totaling 1920 GB (1.92 TB) of flash per adapter.
  • The adapter requires volumes to be formatted as 4096-byte sector drives.
  • The adapter and its accompanying software libraries will not function with 512-byte sector drives.
  • The adapter must be plugged into a direct host-bus PCIe CAPI-capable slot to function.
  • Functionality of the adapter depends on operating system and software versions.
  • Each of the two NVMe drives on the adapter has a separate drive life or durability meter.
  • Linux kernel 4.4 or later is required to enable the adapter.
  • Enabled applications must extend or leverage the IBM CAPI Flash libraries. For more information, programming guides, or sample code, see https://github.com/open-power/capiflash/
Figure 1. CAPI NVMe Flash Accelerator Adapter with 1.92 TB Flash Storage
CAPI NVMe Flash Accelerator Adapter with 1.92 TB Flash Storage

Specifications

Item
Description
Adapter FRU number
01DH749 (Designed to comply with RoHS requirement)
I/O bus architecture
PCIe3 x8
Slot requirement
One CAPI-capable PCIe3 x16 slot
Supported Systems
POWER8® processor-based system
  • 8335-GCA
  • 8335-GTB
  • 8247-21L and 8247-22L
Voltage
3.3 V, 12 V
Form factor
Short, low-profile
Maximum number
For details about the maximum number of adapters that are supported, see PCIe adapter placement rules and slot priorities and select the system you are working on.
Attributes provided
Delivers high IOP and low CPU access to flash memory
Designed for read intensive workloads with light write activity
Two 960 GB M.2 NVMe modules with a write capacity of approximately 1,370 TB

Administering the adapter

Administration of the CAPI NVMe Flash accelerator adapters uses common software utilities with the CAPI Fibre Channel Accelerator (FC EJ16).

In order for the adapter to function properly, the following prerequisites must be met:
  • The system firmware and operating system are installed and the virtualization configuration is complete.
  • The IBM Data Engine for NoSQL User Libraries and AFU microcode are installed in the system default location /opt/ibm/capikv.
  • One or more CAPI NVMe Flash accelerator adapters are plugged into CAPI-capable slots.
  • The CAPI NVMe Flash accelerator adapter microcode is updated.

Checking Drive Write Durability or “Drive Endurance"

Each adapter contains two volumes of 960 GB. Each volume is a separate drive, with its own dedicated controller logic and write endurance. Each drive is warranted to a specific number of write cycles during its lifetime. Administrators must check the drive endurance meters built-in to each drive periodically.
Important: If any single drive on the CAPI NVMe Flash accelerator adapter write endurance indicator reaches 100%, the CAPI NVMe Flash accelerator adapter has exceeded its warranted life, and should be replaced.
To read the endurance indicator built-in to each drive on the adapter:
  1. Find the drive’s SCSI sgX identifier from cxlfstatus:
    • username@hostname:~$ sudo /opt/ibm/capikv/bin/cxlfstatus
    • [sudo] password for ubuntu:
    • CXL Flash Device Status

    Output similar to the following example is displayed:

    Found 0601 0000:01:00.0 U78C9.001.RST0013-P1-C7

    Device SCSI Block Mode LUN WWID
    sg2: 1:0:0:0 sdb legacy 60025380025382463300116000000000
    sg3: 1:1:0:0 sdc legacy 60025380025382463300116000000000

    Found 0601 0002:01:00.0 U78C9.001.RST0013-P1-C6

    Device SCSI Block Mode LUN WWID
    sg4: 2:0:0:0 sdd legacy 60025380025382462300037000000000
    sg5: 2:1:0:0 sde legacy 60025380025382462300036000000000
  2. To read the drive’s endurance indicator, use the sg utilities sg_logs:
    • uusername@hostname:/opt/ibm/capikv/afu$ sudo sg_logs -a /dev/sg2
    • [sudo] password for ubuntu:
    • IBM SAMSUNG MZ1LV960 301Q
    Output similar to the following example is displayed:
    Supported log pages [0x0]:
    0x00 Supported log pages
    0x0d Temperature
    0x11 Solid state media
    0x2f Informational exceptions (SMART)
    0x19 General statistics and performance
    • Temperature page [0xd]:
      • Current temperature = 38 °C
      • Reference temperature = 84 °C
    • Solid state media page [0x11]:
      • Percentage used endurance indicator: 12%
Important: The replacement of depleted NVMe Flash adapters is covered during the system warranty period. After the warranty period has expired, the cost that is incurred for the replacement of depleted CAPI NVMe Flash accelerator adapters is not covered under maintenance contracts, and is the responsibility of the customer.

Control access to the CAPI NVMe Flash accelerator

Use this command to enable an account to control access to the accelerator.
Note: To run the commands in this section, you must have Linux Kernel 4.2.x, or later. For support details, see the Linux Alert website.
  1. To control access to the CAPI NVMe Flash accelerator, complete the following step:
    Note: By default, installation of the IBM Data Engine for NoSQL software libraries creates a cxl user and adds Udev rules to restrict read and write access to the accelerator's volumes to members of the cxl system.
    1. To enable an account to read and write to the accelerator, add the account to the cxl by entering the following command:

      sudo usermod -a -G cxl userid

View the status of the CAPI NVMe flash accelerator

Use this command to view the status of each CAPI NVMe Flash accelerator adapter's logical unit numbers (LUN). The libraries of the IBM DataEngine for NoSQL software include scripts that display the status of the volumes mapped to the accelerator.
Note: To run the commands in this section, you must have Linux Kernel 4.2.x, or later. For support details, see the Linux Alert website.
  1. To view the status of each accelerator adapter's LUN, enter the following command:
    sudo /opt/ibm/capikv/bin/cxlfstatus
    Notes:
    • Volumes may be in either legacy or superpipe mode. The default mode is legacy.
    • Volumes must be in superpipe mode for exploitation by the CAPI Flash block or arkdb APIs.
    Table 1. Coherent Accelerator Interface (CXL) Flash device
    Device SCSI Block Mode LUN WWID
    sg9: 33:0:0:0 sdc legacy 60050768218b0818200000000400006e
    sg10: 33:1:0:0 sdd superpipe 60050768218b0818200000000600006f
    sg11: 34:0:0:0 sde superpipe 60050768218b08182000000007000070
    sg12: 34:1:0:0 sdf superpipe 60050768218b0818200000000300006d

    Table 1 shows two accelerators, each with two ports, and a single volume mapped to each port's WWPN. One volume is in legacy mode, and three volumes are in superpipe mode. The WWID for each volume in the IBM FlashSystem® are displayed for convenience of administration. This matches the WWID shown in the IBM FlashSystem GUI or CLI (ibm@power8:~$ sudo /opt/ibm/capikv/bin/cxlfstatus)

Set the mode for a volume

Use this command to set the mode for a volume.
Note: To run the commands in this section, you must have Linux Kernel 4.2.x, or later. For support details, see the Linux Alert website.
  1. To set the mode for a volume, enter the following command:

    sudo /opt/ibm/capikv/bin/cxlfsetlunmode <LUN> <Mode>

    • An output similar to the following example is displayed:

      • ibm@power8:~$ /opt/ibm/capikv/bin/cxlfsetlunmode
      • 60050768218b0818200000000400006e 1
      • INFO: Adding LUN 60050768218b0818200000000400006e to Super IO table. SUCCESS
      • LUN modes might be 0 for legacy mode, or 1 for superpipe mode.
      • After a LUN is set to superpipe mode, all paths to that volume will also be set to superpipe mode.
      • As shown in Table 1, each volume might be in either legacy or superpipe mode.

Operating system or partition requirements

The adapter is supported on the following versions of the operating systems, depending on the system you are using:
  • Linux
    • Ubuntu 16.04.1 or later
    • Red Hat Enterprise Linux (RHEL) 7.3, or later
    • For support details, see the Linux Alert website.
    • The latest version of enabling libraries and utilities may be downloaded from Fix Central.
Parent topic: PCIe adapters
Related information:
IBM Prerequisite website
Installing, removing, or replacing PCIe adapters
PCIe adapter placement rules and slot priorities
PCIe adapter information by feature type


Last updated: Thu, June 27, 2019