PH11595: BIGSQL CRASH DUE TO MEMORY CORRUPTION IN THE PACKAGE CACHE

APAR status

• Closed as fixed if next.

Error description

• A timing issue in package cache management may result in a cache
  entry being reused or freed by one agent while still in use by
  another.  This may result in memory corruption, causing various
  symptoms.
  .
  The instance may panic when memory corruption is detected. Some
  possible stacks include:
  .
  Stack:
  0x00007FFFEE5095DE _Z10sqle_panici + 0x021e
  0x00007FFFF02CE07E
  _ZN13SQLO_MEM_POOL32diagnoseMemoryCorruptionAndCrashEmPKcb +
  0x028e
  0x00007FFFF02C9953
  _ZN13SQLO_MEM_POOL10MemTreePutEP8SMemNodemP17SqloChunkSubgroup +
  0x0403
  0x00007FFFF02AF6C7 sqlofmblkEx + 0x0477
  .
  or
  .
  0x00007FFFEE5095DE _Z10sqle_panici + 0x021e
  0x00007FFFF02CE07E
  _ZN13SQLO_MEM_POOL32diagnoseMemoryCorruptionAndCrashEmPKcb +
  0x028e
  0x00007FFFF02B2170 address: 0x00007FFFF02B2170 ; dladdress:
  0x00007FFFE9497000 ; offset in lib: 0x0000000006E1B170 ;
  0x00007FFFF02AF9EE sqlofmblkEx + 0x079e
  0x00007FFFF10F7412 _Z8sqlristmP8sqlrr_cbP9sqlri_shdi + 0x0232
  0x00007FFFF0C7ADC0
  _Z32sqlra_sqlW_mem_free_sibling_listP8sqlrr_cbP23sqlra_workspace
  _siblingPm + 0x1680
  0x00007FFFF0C7E23F _Z24sqlra_sqlW_mem_freespaceP8sqlrr_cbmPmi +
  0x02ff
  0x00007FFFF0C725E1
  _Z28sqlra_sqlW_check_for_cleanupP8sqlrr_cbmP7OSSTick + 0x0661
  .
  which may be associated with a message in the db2diag.log
  similar to the following:
  .
  2019-04-25-11.00.47.102960-240 I246426E3260          LEVEL:
  Severe
  PID     : 2202152              TID : 140554599917312 PROC :
  db2sysc 0
  INSTANCE: bigsql               NODE : 000            DB   :
  BIGSQL
  APPHDL  : 0-3592               APPID: **.**.**.***.**.****
  AUTHID  : xxx                  HOSTNAME: xxxx.xxx.com
  EDUID   : 390                  EDUNAME: db2agent (BIGSQL) 0
  FUNCTION: DB2 UDB, SQO Memory Management,
  sqloDiagnoseFreeBlockFailure, probe:10
  MESSAGE : Possible memory corruption detected.
  DATA #1 : ZRC, PD_TYPE_ZRC, 4 bytes
  0x820F0002
  DATA #2 : Corrupt block address, PD_TYPE_CORRUPT_BLK_PTR, 8
  bytes
  0x00007fff7286f280
  DATA #3 : Block header, PD_TYPE_BLK_HEADER, 24 bytes
  0x00007FFF7286F268 : 68ED 8672 FF7F 0000 0000 0000 0000 0000
  h..r............
  0x00007FFF7286F278 : 48ED 8672 FF7F 0000
  H..r....
  code: 0x7286ed68
  curSize: 32767
  SMemBlkFileId: 0
  SMemBlkLOC: 0
  SMemBlkNum: 0
  CodedCSG: 7fff7286ed48
  DATA #4 : Data header, PD_TYPE_BLK_DATA_HEAD, 48 bytes
  0x00007FFF7286F280 : 0000 0000 0000 0000 5441 4F42 0000 0000
  ........TAOB....
  0x00007FFF7286F290 : 801B 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FFF7286F2A0 : 0000 7500 0002 0001 0800 2200 0000 0F00
  ..u.......".....
  CALLSTCK: (Static functions may not be resolved correctly, as
  they are resolved to the nearest symbol)
    [0] 0x00007FFFF02B1C40 /home/bigsql/sqllib/lib64/libdb2e.so.1
  + 0x6E1AC40
    [1] 0x00007FFFF02AF9EE sqlofmblkEx + 0x79E
    [2] 0x00007FFFF10F7412 _Z8sqlristmP8sqlrr_cbP9sqlri_shdi +
  0x232
    [3] 0x00007FFFF0C7ADC0
  _Z32sqlra_sqlW_mem_free_sibling_listP8sqlrr_cbP23sqlra_workspace
  _siblingPm + 0x1680
    [4] 0x00007FFFF0C7E23F
  _Z24sqlra_sqlW_mem_freespaceP8sqlrr_cbmPmi + 0x2FF
    [5] 0x00007FFFF0C725E1
  _Z28sqlra_sqlW_check_for_cleanupP8sqlrr_cbmP7OSSTick + 0x661
    [6] 0x00007FFFF0C7195C
  _Z34sqlra_sqlW_get_dynamic_section_newP8sqlrr_cbP16sqlra_cached_
  var + 0xDEC
    [7] 0x00007FFFF0BF127A
  _Z14sqlra_load_varP8sqlrr_cbP16sqlra_cached_varP14SQLP_LOCK_INFO
  Pb + 0x8A
    [8] 0x00007FFFF0C8B841 _Z13sqlra_get_varP8sqlrr_cbiibPbS1_ +
  0x19D1
    [9] 0x00007FFFF091A3D2
  _Z13sqlrr_prepareP14db2UCinterfaceP16db2UCprepareInfo + 0x452
    [10] 0x00007FFFF0B60294
  _Z27sqlrrProcessDeferredPrepareP8sqlrr_cbP14db2UCinterface +
  0x274
    [11] 0x00007FFFF08D35AB
  _Z21sqlrr_sql_request_preP14db2UCinterfacejiP16db2UCprepareInfoP
  15db2UCCursorInfo + 0x210B
    [12] 0x00007FFFF0916942
  _Z10sqlrr_openP14db2UCinterfaceP15db2UCCursorInfo + 0x212
    [13] 0x00007FFFEEAF3511
  _Z16sqljs_ddm_opnqryP14db2UCinterfaceP13sqljDDMObject + 0x431
  .
  The memory corruption may instead trigger a trap, due to a null
  or invalid pointer.  Stacks may vary, but will often include
  these functions:
  .
  0x00007FFFF0C7E12E _Z24sqlra_sqlW_mem_freespaceP8sqlrr_cbmPmi +
  0x01ee
  0x00007FFFF0C725E1
  _Z28sqlra_sqlW_check_for_cleanupP8sqlrr_cbmP7OSSTick + 0x0661
  .
  or
  .
  0x00007FFFF0C7A24B
  _Z32sqlra_sqlW_mem_free_sibling_listP8sqlrr_cbP23sqlra_workspace
  _siblingPm + 0x0b0b
  0x00007FFFF0C7C4F4
  _Z42sqlra_sqlW_mem_free_variation_siblings_newP8sqlrr_cbP16sqlra
  _cached_var + 0x0254
  .
  Other symptoms include (but are not limited to) traps due to
  corruption in the memory pool free tree:
  .
  0x00007FFFF02C9055
  _ZN13SQLO_MEM_POOL10MemTreeGetEmmPP17SqloChunkSubgroupPj +
  0x0155
  0x00007FFFF02CE22C
  _ZN13SQLO_MEM_POOL19allocateMemoryBlockEmmjmPP17SqloChunkSubgrou
  pPjP12SMemLogEvent + 0x006c
  0x00007FFFF02AB570 sqloGetMemoryBlockExtended + 0x0ff0
  0x00007FFFEF1B7723
  _Z20sqlnn_new_invocationP8sqlrr_cbP19sqlrr_invocation_cbiPP19sql
  nn_invocation_cb + 0x01a3
  0x00007FFFEF1C53F9
  _Z10sqlnn_cmplP8sqeAgentP11sqlrrstrings17sqlnn_compileModesP14sq
  lrr_cmpl_enviiPP9sqlnq_qur + 0x88e9
  0x00007FFFEF1BCB01
  _Z10sqlnn_cmplP8sqeAgentP11sqlrrstrings17sqlnn_compileModesP14sq
  lrr_cmpl_env + 0x0021
  0x00007FFFF0BCD174
  _Z17sqlra_compile_varP8sqlrr_cbP14sqlra_cmpl_envPhitiiiiiP14SQLP
  _LOCK_INFOP16sqlra_cached_varPiS5_Pb + 0x10f4
  0x00007FFFF0BE997E
  _Z14sqlra_find_varP8sqlrr_cbP17sqlra_cached_stmt13sqlra_stmt_idj
  jPhjthP14sqlra_cmpl_env15sqlra_fill_modePiiS8_iiiS8_P14SQLP_LOCK
  _INFOPP16sqlra_cached_varS8_bSA_Pb + 0x096e
  0x00007FFFF0C8AD81 _Z13sqlra_get_varP8sqlrr_cbiibPbS1_ + 0x0f11
  

Local fix

Problem summary

• Please see problem description.
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IT27178

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels