IJ39449: ESS3200 PEMS HANG DUE TO NO IPMI RECV SLOTS

APAR status

• Closed as program error.

Error description

• Pems hang due to no ipmi recv slots, so no new
  ipmi command will be sent to BMC.
  When pems hangs, it will generate these
  lines at dmesg or /var/log/messages
  ERROR: no open ipmi recv slots
  pems_mod:[E]:0136:0581:failed to enq cmd rc=0xfffffff0
  pemsIpmiEnqueueCmd failed to enq setting QUEUE_FULL
  pems_mod:[E]:0136:0315:failed to send
  cmd to backend interface rc=-16
  pems_mod:[E]:0136:0581:failed to enq cmd rc=0xfffffff0
  You will see the last 2 prints over and over.
  

Local fix

• Restart pems module and restart the ess3200_pemscfg service.
  

Problem summary

• Pems hang due to no ipmi recv slots, so no new
  ipmi command will be sent to BMC.
  When pems hangs, it will generate these
  lines at dmesg or /var/log/messages
  ERROR: no open ipmi recv slots
  pems_mod:[E]:0136:0581:failed to enq cmd rc=0xfffffff0
  pemsIpmiEnqueueCmd failed to enq setting QUEUE_FULL
  pems_mod:[E]:0136:0315:failed to send
  cmd to backend interface rc=-16
  pems_mod:[E]:0136:0581:failed to enq cmd rc=0xfffffff0
  You will see the last 2 prints over and over.
  

Problem conclusion

• This problem is fixed in 5.1.2 PTF 4
  To see all Spectrum Scale APARs and
  their respective fix solutions refer to page
  https://public.dhe.ibm.com/storage/spectrumscale/spectrum_scale_
  apars.html
  
  Benefits of the solution:
  Pems will continue sending ipmi commands to
  BMC and it should not fill dmesg with
  pems_mod:[E]:0136:0315:failed to send cmd
  to backend interface rc=-16
  pems_mod:[E]:0136:0581:failed to enq cmd rc=0xfffffff0
  Work around:
  Restart pems module and restart the ess3200_pemscfg service.
  Problem trigger:
  It is a small hole at pems ipmi receive
  handler that it can happen at any time in ESS3200.
  Symptom:  pems hang generating a lot messages at dmesg.
  Platforms affected:   x86_64-linux only (ESS3200 only)
  Functional Area affected: ESS/GNR
  Customer Impact: High Importance
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPEC SCALE STD

• Fixed component ID

  5737F33AP

Applicable component levels