IBM Support

FAQ - Enabling coredumps for DB2 diagnostics for DB2 support

Question & Answer


Question

How are coredumps enabled if requested by DB2 Support ?

Answer

These instructions apply to Linux/Unix systems. To gather the Windows equivalent which is called a user dump, please see links section for "Configuring Windows to produce a user-mode dump".

 

(A) Enable Full Core (AIX only) - Skip this step for other operating systems

Enable Full CORE dump as root

 

chdev -l sys0 -a fullcore=’true’

 


Check Full CORE dump has been enabled. Two methods below:

 

 

lsconf|grep "Full Core"
Full Core: false

 

OR

 

 

lsattr -D -c sys -a fullcore -H, attribute deflt description user_settable fullcore false Enable full CORE dump True

 

 

 

 

(B) Ensure the soft and hard limits for "coredump" of the DB2 instance owner is set to unlimited

 

If it is not set to unlimited be sure to change the .profile so that it is set to unlimited by default.

Note:

  • Hard limits can be modified by changing /etc/security/limit as the root id.
  • For older releases, the db2fmp process's ulimit is inherited from the id which starts the DB2 instance. This is typically the DB2 instance owner, but in some customer configurations they may use a different id such as root. For newer releases the ulimits are not inherited.

 

$ ulimit -c unlimited

 

Soft Limits

 

 

$ ulimit -aS time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2000000
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) unlimited
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited


Hard Limits

 

 

 

$ ulimit -aH
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user) unlimited

 

 

 

 

(C) Optional step if diagnosing db2wdog crash

Corefiles for db2wdog are written to /core instead of the FODC DUMPDIR location because this process is started by root.

Create a symbolic link for the corefile so that it is output to another filesystem with more space (optional). In my example the corefile is created in /tmp/core. Core files can be several gigabytes in size.

 

 

$ ln -s /tmp/core /core

 

Note we link to the filename /tmp/core, there is no directory called "core" under /tmp.

 

 

$ ls -ld /core
lrwxrwxrwx    1 root     system      24 Apr 19 07:37 core -> /tmp/core

 


(D) Optional step if diagnosing db2fmp crash

 

 

db2set DB2_ROUTINE_DUMPCORE=ON
db2stop
db2start


(E) Enable DB2's FODC and check configuration

Enable coredumps. The hard and soft core limit should be unlimited, if not recheck the ulimit settings. The ulimits will only take effect after DB2 instance is restarted.

 

 

 

$ db2pdcfg -fodc dumpshm=on

Database Member 0
FODC (First Occurrence Data Capture) options:
Dump directory for large objects (DUMPDIR)= /home/db2inst1/sqllib/db2dump/
Dump Core files (DUMPCORE)= ON
Current hard core file size limit = Unlimited
Current soft core file size limit = Unlimited
Dump shared memory (DUMPSHM)= ON
Changes to FODC Options have been applied. To make these changes permanent use db2set DB2FODC.

 

 

 

DB2's core files will be written to the location specified by DUMPDIR. Use the db2pdcfg command to check or modify this.

 

 

 

 

$ db2pdcfg -fodc

FODC (First Occurrence Data Capture) options:
   Dump directory for large objects (DUMPDIR)= /home/db2inst1/sqllib/db2dump
   Dump Core files (DUMPCORE)= AUTO
   Current hard core file size limit = Unlimited
   Current soft core file size limit = Unlimited

 

To change this to a different directory which as more space use:

 

 

$ db2pdcfg -fodc dumpdir=/data1

Database Member 0

FODC (First Occurrence Data Capture) options:
Dump directory for large objects (DUMPDIR)= /data1
Service level (SERVICELEVEL)= AUTOMATIC (Effective: FULL)
Dump Core files (DUMPCORE)= AUTO (Effective: ON)

 

 

 

 

 

(F) Restart DB2 instance for new settings to take effect.

db2stop

db2start

 


Note: On Redhat Enterprise Linux 6.x and 7.x, the ABRT (Automatic Bug Reporting Tool) will prevent DB2 from dumping a core file.

Sample to stop abrtd

# service abrtd status
Redirecting to /bin/systemctl status abrtd.service
abrtd.service - ABRT Automated Bug Reporting Tool
Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled)
Active: active (running) since Sun 2017-05-07 08:54:04 EDT; 1 weeks 2 days ago
Main PID: 679 (abrtd)
CGroup: /system.slice/abrtd.service
+-679 /usr/sbin/abrtd -d -s

May 07 08:54:04 test systemd[1]: Starting ABRT Automated Bug Reporting Tool...
May 07 08:54:04 test systemd[1]: Started ABRT Automated Bug Reporting Tool.
May 07 08:54:05 test abrtd[679]: Init complete, entering main loop

# service abrtd stop
Redirecting to /bin/systemctl stop abrtd.service

# service abrtd status
Redirecting to /bin/systemctl status abrtd.service
abrtd.service - ABRT Automated Bug Reporting Tool
Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled)
Active: inactive (dead) since Tue 2017-05-16 11:56:29 EDT; 4s ago
Process: 27795 ExecStart=/usr/sbin/abrtd -d -s (code=killed, signal=TERM)
Main PID: 27795 (code=killed, signal=TERM)


Check with your system administrator if help is needed. It may be possible to stop ABRT for only specific Process ID.

Reference: Chapter 28: Automatic Bug Reporting Tool (ABRT)

(G) Collecting the corefile generated after a crash

AIX:
snapcore -d /tmp /home/db2inst1/sqllib/db2dump/50855994.000.core/core /home/db2inst1/sqllib/adm/db2sysc

where
/tmp = The location of the output file snapcore*.pax is created
/home/db2inst1/sqllib/adm/db2sysc = The executable which created the corefile
/home/db2inst1/sqllib/db2dump/50855994.000.core/core = the corefile itself

For db2fmp crashes the executable would be /home/db2inst1/sqllib/adm/db2fmp

snapcore will create *.pax file in /tmp.

Linux/Solaris:

DB2 includes is own db2snapcore command, if this is not available use attached db2snapcore.tar script. Untar before running the Perl script.

Running ./db2snapcore will provide the usage.

Example this collects all the libraries used by the corefile into output directory /tmp.

db2snapcore -outdir /tmp -corefile $DB2_HOME/db2dump/5085.000.core/core.5085 -trapfile $DB2_HOME/db2dump/FODC_Trap_2017-01-01-08.56.38.458307_0000/5085.000.000.trap.txt -e $DB2_HOME/adm/db2sysc

Notes:

 

  • $DB2_HOME points to /home/db2inst1/sqllib
  • -trapfile is optional on Solaris
  • -e is the executable that crashed this is normally a thread in db2sysc process. See trapfile to confirm the process.
  • This script requires /usr/bin/pax to be installed.

 


db2snapcore.tardb2snapcore.tar


(H) Collect db2support.zip

db2support /tmp -d <db name> -g -s

This creates db2support.zip in /tmp

(I) Send the corefile db2support.zip to DB2 Support.

AIX: Send the *.pax file created by snapcore
Linux/Solaris: db2snapcore.tar


Please refer to the link "Submitting diagnostic information to IBM Technical Support..."

 

[{"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"DB2 Tools - db2support","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"}],"Version":"9.8;9.7;9.5;9.1;10.1;10.5;11.1","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
08 November 2018

UID

swg21499458