IJ41651: GPFS NODE OS CRASH: PANIC: "BUG: UNABLE TO HANDLE KERNEL NULL PO

APAR status

• Closed as program error.

Error description

• GPFS member node os crashed unexpectedly with below
  error:
  
  PANIC: "BUG: unable to handle kernel NULL pointer
  dereference at 000000000000000c"
  
  This issue was found on GPFS 5.1.2.2 version
  

Local fix

Problem summary

• Linux kernel 4.2 added a new field to the Linux inode
  data structure. When an inode is reused under heavy
  workload, this field might not be initialized correctly,
  leading to a kernel crash when accessing the symlink.
  

Problem conclusion

• This problem is fixed in 5.1.2 PTF 7
  To see all Spectrum Scale APARs and
  their respective fix solutions refer to page
  https://public.dhe.ibm.com/storage/spectrumscale/spectrum_scale_
  apars.html
  
  Benefits of the solution:
  The solution is to initialize the
  field correct to avoid the crash.
  
  Work Around:
  It is possible to manually patch the GPl layer:
  Edit the file /usr/lpp/mmfs/src/gpl-linux/inode.c
  In function cxiSetOSNode after line: "case S_IFLNK:"
  insert a new line with:
  inodeP->i_link = NULL;
  Run mmbuildgpl again and restart GPFS on the node.
  Problem trigger:
  This problem is highly depended on the workload.
  If there is a workload creating directories, creating
  files underneath directories, deleting directories and
  also creating symlinks, there is a chance that this
  problem is hit. Build systems are a type of software
  that can exhibit this pattern.
  Symptom: Abend/Crash
  Platforms affected: ALL Linux OS environments
  (excluding RHEL7, since that does not include the
  mentioned Linux kernel change)
  Functional Area affected: All Scale Users
  Customer Impact: Critical
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPEC SCALE STD

• Fixed component ID

  5737F33AP

Applicable component levels