IBM Support

IJ41254: NFS-GANESHA CRASHED AT LOCK_ENTRY_DEC_REF()

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • NFS-ganesha crashed with below stack
during file lock request from nfs client.

(gdb) bt
#0  0x00007f889ae809bf in raise () from /lib64/libpthread.so.0
#1  0x00000000004427b8 in crash_handler
(signo=11, info=0x7f86fa1debb0, ctx=0x7f86fa1dea80)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_init.c:239
#2  <signal handler called>
#3  lock_entry_dec_ref (lock_entry=0x7f86386bd2c0) at
/usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
SAL/state_lock.c:708
#4  0x00000000004ae6fa in free_cookie
(cookie_entry=0x7f85c02144a0, unblock=true)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
SAL/state_lock.c:1371
#5  0x00000000004af31e in state_complete_grant
(cookie_entry=0x7f85c02144a0)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
SAL/state_lock.c:1717
#6  0x0000000000499098 in nlm4_Granted_Res
(args=0x7f84fc602e38, req=0x7f84fc602730, res=0x7f84fc309a00)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
Protocols/NLM/nlm_Granted_Res.c:101
#7  0x000000000045a0ab in nfs_rpc_process_request
(reqdata=0x7f84fc602730)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_worker_thread.c:1331
#8  0x000000000045a97e in nfs_rpc_valid_NLM
(req=0x7f84fc602730)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_worker_thread.c:1593
#9  0x00007f889c8e6538 in svc_vc_decode (req=0x7f84fc602730) at
/usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_vc.c:834
#10 0x000000000044d1aa in nfs_rpc_decode_request
(xprt=0x7f866c7d4500, xdrs=0x7f84fc557f70)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_rpc_dispatcher_thread.c:1349
#11 0x00007f889c8e6449 in svc_vc_recv (xprt=0x7f866c7d4500) at
/usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_vc.c:807
#12 0x00007f889c8e2b91 in svc_rqst_xprt_task
(wpe=0x7f866c7d4758)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:779
#13 0x00007f889c8e3050 in svc_rqst_epoll_events
(sr_rec=0x5572860, n_events=1)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:956
#14 0x00007f889c8e32e9 in svc_rqst_epoll_loop (sr_rec=0x5572860)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:1029
#15 0x00007f889c8e339f in svc_rqst_run_task (wpe=0x5572860)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:1065
#16 0x00007f889c8ebc2d in work_pool_thread (arg=0x7f84c0006270)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/work_pool.c:181
#17 0x00007f889ae7614a in start_thread () from
/lib64/libpthread.so.0
#18 0x00007f889a783dc3 in clone () from /lib64/libc.so.6

Local fix

  • 



Problem summary


    

  • NFS-ganesha crashed with below stack
during file lock request from nfs client.

(gdb) bt
#0  0x00007f889ae809bf in raise () from /lib64/libpthread.so.0
#1  0x00000000004427b8 in crash_handler
(signo=11, info=0x7f86fa1debb0, ctx=0x7f86fa1dea80)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_init.c:239
#2  <signal handler called>
#3  lock_entry_dec_ref (lock_entry=0x7f86386bd2c0) at
/usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
SAL/state_lock.c:708
#4  0x00000000004ae6fa in free_cookie
(cookie_entry=0x7f85c02144a0, unblock=true)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
SAL/state_lock.c:1371
#5  0x00000000004af31e in state_complete_grant
(cookie_entry=0x7f85c02144a0)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
SAL/state_lock.c:1717
#6  0x0000000000499098 in nlm4_Granted_Res
(args=0x7f84fc602e38, req=0x7f84fc602730, res=0x7f84fc309a00)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
Protocols/NLM/nlm_Granted_Res.c:101
#7  0x000000000045a0ab in nfs_rpc_process_request
(reqdata=0x7f84fc602730)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_worker_thread.c:1331
#8  0x000000000045a97e in nfs_rpc_valid_NLM
(req=0x7f84fc602730)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_worker_thread.c:1593
#9  0x00007f889c8e6538 in svc_vc_decode (req=0x7f84fc602730) at
/usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_vc.c:834
#10 0x000000000044d1aa in nfs_rpc_decode_request
(xprt=0x7f866c7d4500, xdrs=0x7f84fc557f70)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
MainNFSD/nfs_rpc_dispatcher_thread.c:1349
#11 0x00007f889c8e6449 in svc_vc_recv (xprt=0x7f866c7d4500) at
/usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_vc.c:807
#12 0x00007f889c8e2b91 in svc_rqst_xprt_task
(wpe=0x7f866c7d4758)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:779
#13 0x00007f889c8e3050 in svc_rqst_epoll_events
(sr_rec=0x5572860, n_events=1)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:956
#14 0x00007f889c8e32e9 in svc_rqst_epoll_loop (sr_rec=0x5572860)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:1029
#15 0x00007f889c8e339f in svc_rqst_run_task (wpe=0x5572860)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/svc_rqst.c:1065
#16 0x00007f889c8ebc2d in work_pool_thread (arg=0x7f84c0006270)
at /usr/src/debug/
gpfs.nfs-ganesha-2.7.5-ibm067.04.295048.el8.x86_64/
libntirpc/src/work_pool.c:181
#17 0x00007f889ae7614a in start_thread () from
/lib64/libpthread.so.0
#18 0x00007f889a783dc3 in clone () from /lib64/libc.so.6

    



Problem conclusion


    

  • This problem is fixed in 5.1.2 PTF 6
To see all Spectrum Scale APARs and
their respective fix solutions refer to page
https://public.dhe.ibm.com/storage/spectrumscale/spectrum_scale_
apars.html

Benefits of the solution:
Corrected code to avoid crash.

Work around: None
Problem trigger:
The users may hit the crash if the same file is accessed
by multiple clients and lot of overlapping
file operations(create/delete/lock).
Symptom: Crash
Platforms affected: ALL Operating System environments
Functional Area affected: cNFS/ NFS-ganesha
Customer Impact: Suggested

    



Temporary fix


    

  • 



Comments


    

  • 



APAR Information




    

  • APAR number
    IJ41254
    • 

  • Reported component name
    SPEC SCALE STD
    • 

  • Reported component ID
    5737F33AP
    • 

  • Reported release
    512
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Special Attention
    NoSpecatt / Xsystem
    • 

  • Submitted date
    2022-07-27
    • 

  • Closed date
    2022-07-27
    • 

  • Last modified date
    2022-09-08
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    SPEC SCALE STD
    • 

  • Fixed component ID
    5737F33AP
    • 



Applicable component levels


    








      
              
                            

              

              [{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STXKQY"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"512","Line of Business":{"code":"LOB26","label":"Storage"}}]
              

                          

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            08 September 2022
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback