IBM Support

IBM Spectrum Scale support: 'blk_cloned_rq_check_limits: over max size limit' errors

Troubleshooting


Problem

If you are using a GPFS block size larger than 512 Kbytes, you may encounter 'blk_cloned_rq_check_limits: over max size limit' errors which lead to dm-multipath path failure. GPFS then will not be able to access the underlying block device.

Symptom

You may encounter the following error messages and dm-multipath path failures:

Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: device-mapper: multipath: Failing path 70:64.
Jun 14 09:33:38 testhost kernel: device-mapper: multipath: Failing path 71:80.
Jun 14 09:33:38 testhost multipathd: 70:64: mark as failed
Jun 14 09:33:38 testhost multipathd: nsd_crs_04: remaining active paths: 1
Jun 14 09:33:38 testhost multipathd: 71:80: mark as failed
[...]

Cause

This is because the following upstream kernel commit is in the Linux distributions:

Commit bf4e6b4e757488dee1b6a581f49c7ac34cd217f8
Author: Hannes Reinecke <hare@suse.de>
Date: Thu Nov 26 08:46:57 2015 +0100

block: Always check queue limits for cloned requests
[...]

Environment

Red Hat Enterprise Linux 6.8, 7.2, and 7.3

Diagnosing The Problem

Check dmesg to see if there are 'blk_cloned_rq_check_limits: over max size limit' error messages.

Resolving The Problem


By default Linux devices are configured for a maximum 512Kbyte IO size. If you are using a GPFS blocksize larger than 512Kbytes, you should upgrade your level of code:

Note: IBM Spectrum Scale V4.2.3 is not affected by this issue.

- For IBM Spectrum Scale V4.2.0 thru V4.2.2.1, upgrade to V4.2.2.2 or later available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.2&platform=All&function=all

- For IBM Spectrum Scale V4.1.0.0 thru V4.1.1.12, upgrade to V4.1.1.13 or later available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=All&function=all

- For IBM GPFS V3.5.0.0 thru V3.5.0.33, upgrade to V3.5.0.34 available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%20software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=All&function=all

If you are unable to upgrade, contact IBM Service to obtain an efix referencing:

IBM GPFS V3.5.0.34, APAR IV93649, or
IBM Spectrum Scale V4.1.1.13, APAR IV93650, or
IBM Spectrum Scale V4.2.2.2, APAR IV92104

To contact IBM Service, see http://www.ibm.com/planetwide/


Workaround

Note: With the Fix/APARs above, the rules in /etc/udev/ are no longer needed.

If you are using a GPFS blocksize larger than 512Kbytes, and you do not apply the fix, you need to add an udev rule to increase the max_sectors_kb parameter.

For example:

If the GPFS file system block size is 4MB, and the storage is from IBM, you need to create a new rule in /etc/udev/rules.d/54-custom.rules with the following content:


ACTION=="add|change", SUBSYSTEM=="block", ATTRS{vendor}=="IBM*", RUN+="/bin/sh -c '/bin/echo 4096 > /sys/block/%k/queue/max_sectors_kb'"

You need to change the block size in the 'echo' command to match the file system block size. You also need to provide the correct storage vendor in the "ATTRS{vendor}" statement.

Document information

More support for: IBM Spectrum Scale

Software version: 4.1.1, 4.2.0, 4.2.1, 4.2.2

Operating system(s): Linux

Reference #: S1009622

Modified date: 01 August 2018