IBM Support

Recovery from Reference Code 551, 555, or 557 in PowerVM VIOS

Troubleshooting


Problem

This document discusses the most common causes for reference codes 551, 555, or 557 during boot up of a VIOS partition and common recovery procedure.

Symptom

VIOS fails to boot up with reference code 551, 555, or 557

Cause

  1. A corrupted filesystem in VIOS rootvg
  2. A corrupted journaled file system (JFS) log device in rootvg
  3. Hardware component failure (e.g. disk, adapter, etc) related to VIOS rootvg

Environment

PowerVM VIOS 3.1

Diagnosing The Problem

Login to the VIOS management GUI, commonly an HMC.  Select the VIOS partition and look at the Reference Code.
  • 551 indicates VIOS rootvg activation (varyon)
  • 555 means fsck error, fsck failure of / (root) filesystem, /dev/hd4
  • 557 means failure to mount / (root) filesystem

Resolving The Problem

To diagnose and attempt to fix any of these reference codes, you must boot the VIOS partition into maintenance mode to run a thorough file system check (fsck) and reformat the jfs2log, /dev/hd8.

1. Boot VIOS partition to maintenance mode and choose the following option:

Access this volume group and start a shell before mounting the filesystems (option 2)

This option will attempt to import the VIOS rootvg in a limited shell (with no filesystems mounted). Pay close attention to the output as this is the point where I/O errors are commonly seen when the issue is due to an underlying hardware problem related to rootvg, i.e.

Fatal: I/O error
If you get errors at this point indicating that a physical volume is missing from the rootvg or errors suggesting a hardware component failure, such as the "I/O error", do not continue with the rest of the steps in this document. Run diagnostics or contact your local Hardware Support Representative.

2. Format the default jfs2log for the VIOS rootvg if the volume group is successfully imported:
/usr/sbin/logform /dev/hd8

Answer yes when asked if you want to destroy the log.

3. Next, run fsck to check and repair file system corruption. (The -y option gives fsck permission to repair file systems when necessary.)
fsck -y /dev/hd4           / (this is the root filesysem)
fsck -y /dev/hd2           /usr
fsck -y /dev/hd9var        /var
fsck -y /dev/hd3           /tmp
fsck -y /dev/hd1           /home

fsck -y /dev/hd10opt       /opt
fsck -y /dev/hd11admin     /admin
fsck -y /dev/livedump      /var/adm/ras/livedump

4. Type exit. The file systems will automatically mount after you type exit.

5. If the VIOS is hosting AIX clients' rootvg on virtual SCSI disk backed by an hdisk, you can verify at this point that the volume group selected earlier (based on Volume Group ID) corresponds to the VIOS rather than a AIX client rootvg. A good way to do this, is by running:
/usr/ios/cli/ioscli ioslevel

The command should return the expected ioslevel installed on disk.

6. At this point, proceed to reboot the VIOS
shutdown -Fr
Other Probable Cause
Reference code 557 is known to occur after rebooting a VIOS following a VIOS update that was done incorrectly. In such case, consider restoring back to the original level. 
Then, verify the VIOS update procedure is done correctly.  For how-to details:
   > Go to Fix Central
   > Review the Readme instructions for the VIOS fix package you plan to update to.
   > Retry the VIOS update as per the Installation Information details on the Readme.

[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSPHKW","label":"PowerVM Virtual I\/O Server"},"ARM Category":[{"code":"a8m50000000L0KDAA0","label":"PowerVM VIOS-\u003EBOOT"}],"ARM Case Number":"TS013898456","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"3.1.2;3.1.3;3.1.4"}]

Document Information

Modified date:
17 August 2023

UID

isg3T1022728