IBM Support

IZ70724: MACHINE CRASHED @GXKQPFREE APPLIES TO AIX 5300-12

A fix is available

Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • =============

machine crashed at abend_trap while executing bos tests.

invalid command KDB(6)> where pvthread+100500 STACK:
￭0001BEE0abend_trap+000000 () ￭0011CC88xmdbg_error+000088
(??, ??, ??, ??) ￭0038769Ccheck_free+00025C (??, ??, ??)
￭003D029Cvalidfault+00015C (??, ??, ??)
￭003CEEC8v_pfget+000328 (??, ??, ??, ??, ??, ??)
￭003FD878state_save_ret+000528 () ____ Exception
(F000000030033600) ____ iar : 000000000461F3D4 msr :
8000000000009032 cr : 24000881 lr : 000000000461F384 ctr :
000000000036B200 xer : 20000008 mq : 00000000 asr :
000000033614B001 r0 : 0000000000000007 r1 : 0FFFFFFFF007F580 r2
: 00000000046376C8 r3 : F100010051999880 r4 : 0000000004629B40
r5 : 00000000046349A8 r6 : 0000000000000000 r7 :
F100013809D3EB42 r8 : 0000000000000000 r9 : 0000000000000000
r10 : 000000000000006A r11 : 0000000000000000 r12 :
0000000044000881 r13 : F100014007FAAC00 r14 : 0000000000000002
r15 : 0000000000000010 r16 : 0000000000000002 r17 :
0000000000000010 r18 : F100010053073838 r19 : F100050011EBD000
r20 : FFFFFFFFFFFFFFFF r21 : F100050011EBD1D0 r22 :
000000000000000B r23 : 0000000000000000 r24 : 000000000000000B
r25 : F10001480F721800 r26 : 0FFFFFFFF007F820 r27 :
000000000474AA40 r28 : 0000000004637560 r29 : F100013809D3EA00
r30 : 0000000004637C48 r31 : 0FFFFFFFF007F688 (6)> more (ﾪC to
quit) ?

r30 : 0000000004637C48 r31 : 0FFFFFFFF007F688 (6)> more (ﾪC to
quit) ?

prev 0000000000000000 stackfix 0000000000000000 int_ticks 0000
kjmpbuf 0000000000000000 excbranch 0000000000000000 no_pfault
00 intpri 0B backt 00 flags 00 hw_fru_id 00000000 hw_cpu_id
00000008 fpscr 0000000000000000 fpscrx 00000000 fpowner 00 fpeu
00 fpinfo 00 alloc F000 o_iar 0000000000000000 o_toc
0000000000000000 o_arg1 0000000000000000 o_vaddr
0000000000000000 krlockp 0000000000000000 rmgrwa
0000000000000000 amrstackhigh F00000003001BFF0 amrstacklow
F00000003001B000 amrstackcur F00000003001BFF0 amrstackfix
0000000000000000 kstackhigh 0000000000000000 kstacksize
00000000 frrstart 700DFEED00000000 frrend 700DFEED00000000
frrcur 700DFEED00000000 frrstatic 0000 kjmpfrroff 0000 frrovcnt
0000 frrbarrcnt 0000 frrmask 00 callrmgr 00 Except :  excp_type
00000106 EXCEPT_DSI orgea F100013809D3EB42 dsisr
0000000040000000 bit set: DSISR_PFT vmh 00007FFFFFFFD000 curea
F100013809D3EB42 pftyp 0000000000000106
￭0461F3D4GxkQpFree+000134 (F100010051999880 ￭??)
￭047420A4IbQpDestroy+0007C4 (??, ??) (6)> more (ﾪC to quit) ?
fpeu 00 fpinfo 00 alloc F000 o_iar 0000000000000000 o_toc
0000000000000000 o_arg1 0000000000000000 o_vaddr
0000000000000000 krlockp 0000000000000000 rmgrwa
0000000000000000 amrstackhigh F00000003001BFF0 amrstacklow
F00000003001B000 amrstackcur F00000003001BFF0 amrstackfix
0000000000000000 kstackhigh 0000000000000000 kstacksize
00000000 frrstart 700DFEED00000000 frrend 700DFEED00000000
frrcur 700DFEED00000000 frrstatic 0000 kjmpfrroff 0000 frrovcnt
0000 frrbarrcnt 0000 frrmask 00 callrmgr 00 Except :  excp_type
00000106 EXCEPT_DSI orgea F100013809D3EB42 dsisr
0000000040000000 bit set: DSISR_PFT vmh 00007FFFFFFFD000 curea
F100013809D3EB42 pftyp 0000000000000106
￭0461F3D4GxkQpFree+000134 (F100010051999880 ￭??)
￭047420A4IbQpDestroy+0007C4 (??, ??) (6)> more (ﾪC to quit) ?
￭0573DB84kIbQpDestroy+000064 (??)
￭05728478rds_ib_term_ibc+000D98 (??)
￭05729828rds_ib_failover+000108 (??)
￭0572FF04rds_timer+000844 () ￭0048EE38Netintr+0002F8 ()
￭0048EB00netisr_thread+000020 () ￭0025F174threadentry+000094
(??, ??, ??, ??) KDB(6)>

￭0025F174threadentry+000094 (??, ??, ??, ??) KDB(6)> stat
SYSTEM_CONFIGURATION:  POWER_PC POWER_5 machine with 8
available CPU(s) (64-bit registers)

SYSTEM STATUS:  sysname... AIX nodename.. rds6ih release... 1
version... 6 build date Jan 22 2010 build time 12:08:21
label..... 1003A_61J machine... 00C92AFF4C00 nid.......
C92AFF4C Kernel Abend age of system: 2 day, 18 hr., 41 min., 17
sec.  xmalloc debug: enabled FRRs active... 0 FRRs started.. 0

CRASH INFORMATION:  CPU 6 CSA F1000815B01B1D00 at time of
crash, error code for LEDs: 70000000

(6)> more (ﾪC to quit) ?

(6)> more (ﾪC to quit) ?  SYSTEM MESSAGES:

AIX Version 6.1 Starting NODE#000 physical CPU#001 as logical
CPU#001... done.  Starting NODE#001 physical CPU#002 as logical
CPU#002... done.  Starting NODE#001 physical CPU#003 as logical
CPU#003... done.  Starting NODE#002 physical CPU#004 as logical
CPU#004... done.  Starting NODE#002 physical CPU#005 as logical
CPU#005... done.  Starting NODE#003 physical CPU#006 as logical
CPU#006... done.  Starting NODE#003 physical CPU#007 as logical
CPU#007... done.  PALCMD_HA_REPORT: B002AF30 fail_io_cfg:
save_dev=800000110000000B, save_offset=125C, save_overunder=2
fail_io_uncfg: save_dev=800000110000000B fail_io_cfg:
save_dev=800000110000000A, save_offset=125C, save_overunder=2
fail_io_uncfg: save_dev=800000110000000A fail_io_cfg:
save_dev=8000001100000009, save_offset=125C, save_overunder=2
fail_io_uncfg: save_dev=8000001100000009 fail_io_cfg:
save_dev=800000110000000B, save_offset=125B, save_overunder=1
fail_io_uncfg: save_dev=800000110000000B

￭mustfix template continued in the next note

==== State: Working by: shouchun on 16 February 2010 16:15:24
====

fail_io_cfg: save_dev=800000110000000A, save_offset=125B,
save_overunder=1 fail_io_uncfg: save_dev=800000110000000A
fail_io_cfg: save_dev=8000001100000009, save_offset=125B,
save_overunder=1 (6)> more (ﾪC to quit) ?

_id=6 cke_errno=0 cke_Sy_error=0x6008 cke_Xtnd_error=0x393
Interal-err-string:"" cke_pid=0"chk_write_inst failed" iss_id=6
cke_errno=22 cke_Sy_error=0x6012 cke_ Xtnd_error=0x5FC
Interal-err-string:"" cke_pid=-1080847417735367648"chk_write_dt
ype failed" iss_id=6 cke_errno=22 cke_Sy_error=0x6011
cke_Xtnd_error=0x51E Inter al-err-string:""
cke_pid=-1080862536036519936"Check/restart file failed" iss_id=
6 cke_errno=22 cke_Sy_error=0x6058 cke_Xtnd_error=0x2A6
Interal-err-string:"fd = 1" cke_pid=-1080862536179383952if_ib:
SIOCIF_IB_DEL_ARP failed, err=6 <- end_of_buffer KDB(6)> errpt
ERRORS NOT READ BY ERRDEMON (ORDERED CHRONOLOGICALLY):

Error Record:  erec_flags .............. 1 erec_len
................ 58 erec_timestamp .......... 4B64EAEA
erec_rec_len ............ 34 erec_cid ................ 0
erec_dupcount ........... 0 erec_duptime1 ........... 0
erec_duptime2 ........... 0 erec_rec.error_id ....... 975DD6C
erec_rec.resource_name .. ABEND EEEE0000 DA51B023 F1000138
09D3EB42 .....Q.#...8...B 00000000 0011CC8C 80000000 00029032
...............2 KDB(6)>

The machine is still in KDB mode.You can access the vterm
through the HMC given above.

See MACHINE ACCESS section for login information.

Local fix

  • 



Problem summary


    

  • =============

machine crashed at abend_trap while executing bos tests.

invalid command KDB(6)> where pvthread+100500 STACK:
 0001BEE0 abend_trap+000000 ()  0011CC88 xmdbg_error+000088
(??, ??, ??, ??)  0038769C check_free+00025C (??, ??, ??)
 003D029C validfault+00015C (??, ??, ??)
 003CEEC8 v_pfget+000328 (??, ??, ??, ??, ??, ??)
 003FD878 state_save_ret+000528 () ____ Exception
(F000000030033600) ____ iar : 000000000461F3D4 msr :
8000000000009032 cr : 24000881 lr : 000000000461F384 ctr :
000000000036B200 xer : 20000008 mq : 00000000 asr :
000000033614B001 r0 : 0000000000000007 r1 : 0FFFFFFFF007F580 r2
: 00000000046376C8 r3 : F100010051999880 r4 : 0000000004629B40
r5 : 00000000046349A8 r6 : 0000000000000000 r7 :
F100013809D3EB42 r8 : 0000000000000000 r9 : 0000000000000000
r10 : 000000000000006A r11 : 0000000000000000 r12 :
0000000044000881 r13 : F100014007FAAC00 r14 : 0000000000000002
r15 : 0000000000000010 r16 : 0000000000000002 r17 :
0000000000000010 r18 : F100010053073838 r19 : F100050011EBD000
r20 : FFFFFFFFFFFFFFFF r21 : F100050011EBD1D0 r22 :
000000000000000B r23 : 0000000000000000 r24 : 000000000000000B
r25 : F10001480F721800 r26 : 0FFFFFFFF007F820 r27 :
000000000474AA40 r28 : 0000000004637560 r29 : F100013809D3EA00
r30 : 0000000004637C48 r31 : 0FFFFFFFF007F688 (6)> more (ªC to
quit) ?

r30 : 0000000004637C48 r31 : 0FFFFFFFF007F688 (6)> more (ªC to
quit) ?

prev 0000000000000000 stackfix 0000000000000000 int_ticks 0000
kjmpbuf 0000000000000000 excbranch 0000000000000000 no_pfault
00 intpri 0B backt 00 flags 00 hw_fru_id 00000000 hw_cpu_id
00000008 fpscr 0000000000000000 fpscrx 00000000 fpowner 00 fpeu
00 fpinfo 00 alloc F000 o_iar 0000000000000000 o_toc
0000000000000000 o_arg1 0000000000000000 o_vaddr
0000000000000000 krlockp 0000000000000000 rmgrwa
0000000000000000 amrstackhigh F00000003001BFF0 amrstacklow
F00000003001B000 amrstackcur F00000003001BFF0 amrstackfix
0000000000000000 kstackhigh 0000000000000000 kstacksize
00000000 frrstart 700DFEED00000000 frrend 700DFEED00000000
frrcur 700DFEED00000000 frrstatic 0000 kjmpfrroff 0000 frrovcnt
0000 frrbarrcnt 0000 frrmask 00 callrmgr 00 Except :  excp_type
00000106 EXCEPT_DSI orgea F100013809D3EB42 dsisr
0000000040000000 bit set: DSISR_PFT vmh 00007FFFFFFFD000 curea
F100013809D3EB42 pftyp 0000000000000106
 0461F3D4 GxkQpFree+000134 (F100010051999880  ?? )
 047420A4 IbQpDestroy+0007C4 (??, ??) (6)> more (ªC to quit) ?
fpeu 00 fpinfo 00 alloc F000 o_iar 0000000000000000 o_toc
0000000000000000 o_arg1 0000000000000000 o_vaddr
0000000000000000 krlockp 0000000000000000 rmgrwa
0000000000000000 amrstackhigh F00000003001BFF0 amrstacklow
F00000003001B000 amrstackcur F00000003001BFF0 amrstackfix
0000000000000000 kstackhigh 0000000000000000 kstacksize
00000000 frrstart 700DFEED00000000 frrend 700DFEED00000000
frrcur 700DFEED00000000 frrstatic 0000 kjmpfrroff 0000 frrovcnt
0000 frrbarrcnt 0000 frrmask 00 callrmgr 00 Except :  excp_type
00000106 EXCEPT_DSI orgea F100013809D3EB42 dsisr
0000000040000000 bit set: DSISR_PFT vmh 00007FFFFFFFD000 curea
F100013809D3EB42 pftyp 0000000000000106
 0461F3D4 GxkQpFree+000134 (F100010051999880  ?? )
 047420A4 IbQpDestroy+0007C4 (??, ??) (6)> more (ªC to quit) ?
 0573DB84 kIbQpDestroy+000064 (??)
 05728478 rds_ib_term_ibc+000D98 (??)
 05729828 rds_ib_failover+000108 (??)
 0572FF04 rds_timer+000844 ()  0048EE38 Netintr+0002F8 ()
 0048EB00 netisr_thread+000020 ()  0025F174 threadentry+000094
(??, ??, ??, ??) KDB(6)>

 0025F174 threadentry+000094 (??, ??, ??, ??) KDB(6)> stat
SYSTEM_CONFIGURATION:  POWER_PC POWER_5 machine with 8
available CPU(s) (64-bit registers)

SYSTEM STATUS:  sysname... AIX nodename.. rds6ih release... 1
version... 6 build date Jan 22 2010 build time 12:08:21
label..... 1003A_61J machine... 00C92AFF4C00 nid.......
C92AFF4C Kernel Abend age of system: 2 day, 18 hr., 41 min., 17
sec.  xmalloc debug: enabled FRRs active... 0 FRRs started.. 0

CRASH INFORMATION:  CPU 6 CSA F1000815B01B1D00 at time of
crash, error code for LEDs: 70000000

(6)> more (ªC to quit) ?

(6)> more (ªC to quit) ?  SYSTEM MESSAGES:

AIX Version 6.1 Starting NODE#000 physical CPU#001 as logical
CPU#001... done.  Starting NODE#001 physical CPU#002 as logical
CPU#002... done.  Starting NODE#001 physical CPU#003 as logical
CPU#003... done.  Starting NODE#002 physical CPU#004 as logical
CPU#004... done.  Starting NODE#002 physical CPU#005 as logical
CPU#005... done.  Starting NODE#003 physical CPU#006 as logical
CPU#006... done.  Starting NODE#003 physical CPU#007 as logical
CPU#007... done.  PALCMD_HA_REPORT: B002AF30 fail_io_cfg:
save_dev=800000110000000B, save_offset=125C, save_overunder=2
fail_io_uncfg: save_dev=800000110000000B fail_io_cfg:
save_dev=800000110000000A, save_offset=125C, save_overunder=2
fail_io_uncfg: save_dev=800000110000000A fail_io_cfg:
save_dev=8000001100000009, save_offset=125C, save_overunder=2
fail_io_uncfg: save_dev=8000001100000009 fail_io_cfg:
save_dev=800000110000000B, save_offset=125B, save_overunder=1
fail_io_uncfg: save_dev=800000110000000B

 mustfix template continued in the next note

==== State: Working by: shouchun on 16 February 2010 16:15:24
====

fail_io_cfg: save_dev=800000110000000A, save_offset=125B,
save_overunder=1 fail_io_uncfg: save_dev=800000110000000A
fail_io_cfg: save_dev=8000001100000009, save_offset=125B,
save_overunder=1 (6)> more (ªC to quit) ?

_id=6 cke_errno=0 cke_Sy_error=0x6008 cke_Xtnd_error=0x393
Interal-err-string:"" cke_pid=0"chk_write_inst failed" iss_id=6
cke_errno=22 cke_Sy_error=0x6012 cke_ Xtnd_error=0x5FC
Interal-err-string:"" cke_pid=-1080847417735367648"chk_write_dt
ype failed" iss_id=6 cke_errno=22 cke_Sy_error=0x6011
cke_Xtnd_error=0x51E Inter al-err-string:""
cke_pid=-1080862536036519936"Check/restart file failed" iss_id=
6 cke_errno=22 cke_Sy_error=0x6058 cke_Xtnd_error=0x2A6
Interal-err-string:"fd = 1" cke_pid=-1080862536179383952if_ib:
SIOCIF_IB_DEL_ARP failed, err=6 <- end_of_buffer KDB(6)> errpt
ERRORS NOT READ BY ERRDEMON (ORDERED CHRONOLOGICALLY):

Error Record:  erec_flags .............. 1 erec_len
................ 58 erec_timestamp .......... 4B64EAEA
erec_rec_len ............ 34 erec_cid ................ 0
erec_dupcount ........... 0 erec_duptime1 ........... 0
erec_duptime2 ........... 0 erec_rec.error_id ....... 975DD6C
erec_rec.resource_name .. ABEND EEEE0000 DA51B023 F1000138
09D3EB42 .....Q.#...8...B 00000000 0011CC8C 80000000 00029032
...............2 KDB(6)>

The machine is still in KDB mode.You can access the vterm
through the HMC given above.

See MACHINE ACCESS section for login information.

    



Problem conclusion


    

  • Stop tracing the control block after it is freed.

    



Temporary fix


    

  • 



Comments


    

  • 5300-12 - use AIX APAR IZ70724
6100-05 - use AIX APAR IZ70690

    



APAR Information




    

  • APAR number
    IZ70724
    • 

  • Reported component name
    AIX  5.3
    • 

  • Reported component ID
    5765G0300
    • 

  • Reported release
    530
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Submitted date
    2010-02-19
    • 

  • Closed date
    2010-02-19
    • 

  • Last modified date
    2013-03-29
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    
IZ70690
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    AIX  5.3
    • 

  • Fixed component ID
    5765G0300
    • 



Applicable component levels


    

  • R530  PSY  U827015
       UP10/04/14  I  1000
    • 



PTF to Fileset Mapping








      
              
                            

              

              [{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG11P","label":"APARs - AIX 5.3 environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"530","Edition":"","Line of Business":{"code":"","label":""}}]
              

                          

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            29 March 2013
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback