Flashes (Alerts)
Abstract
IBM has identified an issue in IBM Spectrum Scale (GPFS) V4.2 through 5.0, in which mounting a file system created on GPFS 2.2 or earlier releases may result in mmfsd daemon crashing when performing basic file system operations including creating files and writing to them. This is an old file system format, as that release reached end-of-service in 2007.
Content
Problem Summary:
The issue affects IBM Spectrum Scale (GPFS) V4.2 and later releases, where file systems created at format level 2.2 (format number 7.00) or older levels are not properly handled. The result is that operating on files created on such file systems causes internal failures, including mmfsd daemon crashes. Note that some of these older file systems created with format level 2.2 (or older) may have been updated with command
mmchfs <file system> -V full Updating those file system formats in this way does not avoid the issue, though, since the file system structure remains essentially the same. A file system created with GPFS 2.2 or older releases uses a "narrow" disk address format (severely limiting the size of the file system which may be supported), and that structure remains intact even as the file system format is updated.
Since GPFS 2.2 reached end-of-service in 2007, only a small subset of existing file systems should be affected by the issue.
Users Affected:
This issue affects customers running IBM Spectrum Scale (GPFS) V4.2 through 5.0, when the following condition is met:
The file system has been created at format level 2.2 (format number 7.00) or older levels, even if the current file system format level is higher.
How to determine if a 2.2 file system format is being used
Two methods are provided:
1.) Issue command
mmlsfs <file system name> -VThe output will either include a line such as
<format number> (<format level>) File system versionor two lines:
<format number 1> (<format level 1>) Current file system version <format number 2> (<format level 2>) Original file system versionIn the case of the single line containing "File system version", use the format number / level as the version. In the case of the two lines above, use the format number / level from the "Original file system version" line (the second line). If the format number / level is
-V 7.00 (2.2.0.0) File system version
or earlier, this is a narrow-address file system, which is impacted by this issue.
2.) On a node which is part of the cluster and has the file system mounted, issue command
mmfsadm dump stripe | grep 'disk address size'If the output includes the following character string:
disk address size: 6then at least one of the file systems is a narrow-address file system, which is impacted by this issue. If the output only includes lines
disk address size: 12 then all file systems are of the more modern wide-address type, and these are not affected by the issue.
If the output includes lines with both disk address sizes then it is possible to find out which file systems are of the narrow-address type, by redirecting the output of the command:
mmfsadm dump stripe > /tmp/dump_stripe and then examining the output file (/tmp/dump_stripe in the example above) to locate the name of the file system which corresponds to "disk address size: 6":
State of StripeGroup "myfs99" at 0xF10000130296DA98, uid C0A873CF:52543702, local id 8: homeCluster myAddr <c0n0> ipAddr 192.168.140.92 [...] disk address size: 6In the example above, file system "myfs99" uses a "narrow-address" format.
Problem Determination:
Updating a cluster which is running V4.1 or earlier levels to V4.2 or later levels may result in errors being encountered.
When files are created on the affected file system formats, the mmfsd daemon may terminate abnormally with errors similar to the following:
When files are created on the affected file system formats, the mmfsd daemon may terminate abnormally with errors similar to the following:
Assertion failed: subblocksPerFileBlock == (1 << (tinodeP->getFblockSize())), file [...] metadata.C, line 5222 Other possible failures include mmfsd daemon asserts such as the following:
Assertion failed: (((newNHashTabEntries) & ((newNHashTabEntries)-1)) == 0) && newNHashTabEntries <= dlP->unqMask + 1, file [...] direct-common.C, line 205 Assertion failed: startRange * rangeSize >= blockOffset && stopRange <= theBitmap.getRangesNum() && stopRange * rangeSize <= (blockOffset + dataBufP->getDataLen()), file [...] bufdesc.C, line 3668 Recommendations:
1.) Users with file systems created at Release 2.2 level or earlier should not update (say, from 4.1) to V4.2 or later.
Contact IBM Service for further guidance, including support arrangements on the V4.1.1 release.
2.) Users affected by the issue should not attempt to use any of the following functions, even on V4.1:
(that is, do not use any of the commands below on such file systems)
A) "Rapid repair", as activated via
mmchfs <file system> --rapid-repairB) Immutable files:
mmchattr -iC) Append-only files:
mmchattr -a
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STXKQY","label":"IBM Spectrum Scale"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"4.2, 5.0","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]
Was this topic helpful?
Document Information
Modified date:
26 September 2022
UID
ibm10881468