ACL Corruption/Loss when using Independent Filesets on a SONAS System

Problem Environment / Exposure:

SONAS 1.3.0.0 or higher system may be exposed to this issue in each of the following instances:
1) If the filesystem contains independent filesets.
2) If the filesystem has had independent filesets which have been deleted.
3) If you have used the GUI to create filesets, and you have not changed the default to choose dependent filesets instead.
4) You have not created filesets yet and wish to create independent filesets now.

PLEASE CALL IBM SUPPORT IMMEDIATELY IF YOU MEET ANY OF THE ABOVE CRITERIA.

Problem Diagnosis:
Notes:
1. This procedure requires root privileges to execute.
2. This data/output should be collected and provided to IBM support, if IBM support is required.
3. Steps (a) & (b) should be repeated for each filesystem configured on the system.

(a) To determine if your system may be affected, issue the following command on the management node.
echo desc | /usr/lpp/mmfs/bin/tsdbfs <filesystem-name> | grep inodeSpaceMask

System at Risk:
If the values reported for the inodeSpaceMask are non-zero the filesystem is at risk for this problem.
If so, proceed to next step (b) below.
Example (System at Risk):
------------------
# echo desc | /usr/lpp/mmfs/bin/tsdbfs pk1 | grep inodeSpaceMask

   inodeSpaceMask  0000000000000100
0000000000000000000000000000000000000000000000000000000100000000
------------------
Note: In the above example, filesystem pk1 is at risk (independent filesets were created).

System NOT at Risk:
If the values reported for the inodeSpaceMask are all zero, then you are currently not at risk. The problem will be corrected in a future SONAS.

Example (System NOT at Risk):
------------------
# echo desc | /usr/lpp/mmfs/bin/tsdbfs pk2 | grep inodeSpaceMask

   inodeSpaceMask  0000000000000000
0000000000000000000000000000000000000000000000000000000000000000
------------------
Note: In the above example, filesystem pk2 is safe (no independent filesets were created).



(b) To determine if the ACL reclaim background job may have run, issue the following command on the management node.
echo inode 4| /usr/lpp/mmfs/bin/tsdbfs <filesystem-name>|grep -e fileSize -e currentMetadataRepl

If the value of nFullBlocks equals currentMetadataReplicas, then the ACL garbage collection background job will not run until at least another whole block of ACLs are created. This usually provides a significant margin of safety. You should still contact IBM support to further evaluate the risk factors and seek guidance as this could fill up depending on the usage pattern.
Example:
------------------
# echo inode 4| /usr/lpp/mmfs/bin/tsdbfs pk1|grep -e fileSize -e currentMetadataRepl

   fileSize=262144 nFullBlocks=2
   currentMetadataReplicas=2 maxMetadataReplicas=2
------------------
Note: In the above example, the ACL garbage collection background job has not been run yet (nFullBlocks equals currentMetadataReplicas). The files in filesystem pk1 with ACLs are at risk in future, though there is some margin of safety at this moment.

However, if the value of nFullBlocks is greater than currentMetadataReplicas, you may or may not have already hit the issue, losing access to some files. You should contact IBM support immediately to analyze further.


Problem Resolution / Prevention:
To avoid potential access / data loss, it is critical to prevent the ACL garbage collection process from getting triggered.
Customers are advised to contact IBM Support immediately, with the diagnostic data explained above, for guidance on determining and obtaining the appropriate fix.