PH15540: BIG SQL WORKER NODE CRASH DUE TO MEMORY CORRUPTION WHEN CALLING STORED PROCEDURE

APAR status

• Closed as fixed if next.

Error description

• A Big SQL worker node may crash when invoking a stored
  procedure.
  .
  The problem will manifest in the form of a BigSQL worker node
  crash showing symptoms of a memory corruption in the
  communication areas between the db2fmp process and the engine.
  Since the problem is a memory corruption, the symptoms can
  vary, but the main flavour of this problem that as been
  observed is an error when trying to free memory.
  .
  Messages similar to the following may be seen in the
  db2diag.log:
  .
  2019-08-09-12.42.08.946286+120 I36527589E450         LEVEL:
  Error
  PID     : 23634                TID : 140281560938240 PROC :
  db2fmp (
  INSTANCE: bigsql               NODE : 000
  HOSTNAME: sample.host.name
  FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpListener,
  probe:1999
  MESSAGE : FMP exiting with error. Last agent registered:
  DATA #1 : Hexdump, 4 bytes
  0x00007F95D89BAA9C : 0000 0000
  ....
  .
  2019-08-09-12.42.08.952797+120 I36528040E464         LEVEL:
  Warning
  PID     : 23634                TID : 140281560938240 PROC :
  db2fmp (
  INSTANCE: bigsql               NODE : 000
  HOSTNAME: sample.host.name
  FUNCTION: DB2 UDB, routine_infrastructure,
  sqlerWorkerThreadEntry, probe:1022
  MESSAGE : ZRC=0xFFFFFB95=-1131
            SQL1131N  A stored procedure process has been
  terminated abnormally.
            Routine name: "". Specific name: "".
  .
  2019-08-09-12.42.09.641007+120 E36529404E1759        LEVEL:
  Severe
  PID     : 13848                TID : 140097148151552 PROC :
  db2sysc 0
  INSTANCE: bigsql               NODE : 000
  HOSTNAME: sample.host.name
  EDUID   : 11                   EDUNAME: db2sysc 0
  FUNCTION: DB2 UDB, SQO Memory Management,
  sqloDiagnoseFreeBlockFailure, probe:99
  9
  MESSAGE : Memory validation failure, diagnostic file dumped.
  DATA #1 : String, 28 bytes
  Corrupt pool free tree node.
  DATA #2 : File name, 41 bytes
  13848.140097148151552.mem_diagnostics.txt
  CALLSTCK: (Static functions may not be resolved correctly, as
  they are resolved to the nearest symbol)
    [0] 0x00007F6B0A7C74B5
  _ZN13SQLO_MEM_POOL32diagnoseMemoryCorruptionAndCrashEmPKcb +
  0x285
    [1] 0x00007F6B0A7A8B07 sqlofmblkEx + 0x477
    [2] 0x00007F6B0589372C
  _Z13sqlccFreeIPCsP18SQLCC_INITSTRUCT_Tcc + 0x9C
    [3] 0x00007F6B08D5FD77
  _Z18sqlerDeallocFmpIPCPP18SQLCC_INITSTRUCT_TP8sqeAgentb + 0x57
    [4] 0x00007F6B08D5A2F3
  _Z24sqlerCleanThreadResourceP18sqlerFmpThreadListbcc + 0x423
    [5] 0x00007F6B08D59CF3 /home/bigsql/sqllib/lib64/libdb2e.so.1
  + 0x53BECF3
    [6] 0x00007F6B08D615A5
  _Z20sqlerReturnFmpToPoolccP14sqlerFmpHandleP8sqeAgent + 0x14A5
    [7] 0x00000000004165CA _Z14sqleCleanupFmpii + 0x41A
    [8] 0x0000000000414D2B
  _Z25sqleSyscRequestProcessingjP18SQL_SYSCON_REQUESTPi + 0x1AB
    [9] 0x0000000000414425 _Z14sqleRunSysCtlrv + 0x695
    [10] 0x0000000000413C36 _Z11sqleSysCtlrv + 0x1736
    [11] 0x00007F6B0A7EC01A /home/bigsql/sqllib/lib64/libdb2e.so.1
  + 0x6E5101A
    [12] 0x00007F6B0A7EAA68 sqloRunInstance + 0x928
    [13] 0x000000000040CF76 DB2main + 0x1236
    [14] 0x00007F6B0A7F05AB sqloEDUEntry + 0x11B
    [15] 0x00007F6B120B8DD5 /lib64/libpthread.so.0 + 0x7DD5
    [16] 0x00007F6B02463B3D clone + 0x6D
  

Local fix

• NA
  

Problem summary

• Please see the problem description.
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IT27993

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels