IJ44899: SOME RECALLED FILES WERE NOT DELETED PERMANENTLY.

APAR status

• Closed as program error.

Error description

• All files in the active file system are deleted and also
  the snapshots along with files deletions. The data of
  some
  files had been migrated to external storage, so when
  deleting these files, recall was needed to preserve the
  data into the not yet deleted snapshots. However, the
  recall failed with an E_DM_ABORT error and these files
  were left as to-be-deleted states and still consumed the
  disk space. Later, after all files deletions and snapshot
  deletions were done, even though there are no files from
  the users' point of view, the mmrepquota command still
  reported some files and disk space consumption. This is
  working as designed because the to-be-deleted files are
  still there and indeed occupied the space.
  
  The essential issue for this case is the recall failure
  which would be caused by some issue on the DM management
  application side. As long as that DM issue has been
  fixed,
  This problem wouldn't happen. In practice, users still
  could hit such DM issue and result in recall failure,
  then
  cause some files as to-be-deleted states.
  
  To completely remove these to-be-deleted files, you can
  take one of following actions:
  
  1) Migrate the file system manager to another node, which
     will start the full scan on these to-be-deleted inodes
     and proceed to remove them.
  
  2) There is an undocumented 'mmfsadm' command that can
     manually start a full inode map scan to trigger
     deletion on these to-be-deleted inodes.
  
  Note that this retry won't hit the recall failure again
  because the data won't need to be preserved to the
  snapshots as all snapshots have already been deleted.
  

Local fix

Problem summary

• File usage quota is effective and some files in the file system
  have been migrated through DMAPI application, then delete all
  files in the file system. However, the mmrepquota consistently
  shows some files still in-use.
  

Problem conclusion

• This problem is fixed in 5.1.2.10
  To see all Spectrum Scale APARs and their respective
  Fix solutions refer to page:
  https://public.dhe.ibm.com/storage/spectrumscale/spectrum_scale_
  apars.html
  
  Benefits of the solution:
  The mmrepquota can correctly report the file usage in the file
  system.
  
  Work around:
  Since this problem only happens when there are snapshots, so
  deleting snapshots can workaround the problem.
  
  Problem trigger:
  Migrate files to external storage through DMAPI function, and
  create snapshots for the file system, then delete these migrated
  files.
  
  Symptom:
  Error output
  
  Platforms affected:
  Linux Only
  
  Functional Area affected:
  DMAPI
  
  Customer Impact:
  High Importance
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPEC SCALE DME

• Fixed component ID

  5737F34AP

Applicable component levels