Question & Answer
Question
How are coredumps enabled if requested by DB2 Support ?
Answer
These instructions apply to Linux/Unix systems. To gather the Windows equivalent which is called a user dump, please see links section for "Configuring Windows to produce a user-mode dump".
(A) Enable Full Core (AIX only) - Skip this step for other operating systems
Enable Full CORE dump as root
chdev -l sys0 -a fullcore=’true’
Check Full CORE dump has been enabled. Two methods below:
lsconf|grep "Full Core"
Full Core: false
OR
lsattr -D -c sys -a fullcore -H, attribute deflt description user_settable fullcore false Enable full CORE dump True
(B) Ensure the soft and hard limits for "coredump" of the DB2 instance owner is set to unlimited
If it is not set to unlimited be sure to change the .profile so that it is set to unlimited by default.
Note:
- Hard limits can be modified by changing /etc/security/limit as the root id.
- For older releases, the db2fmp process's ulimit is inherited from the id which starts the DB2 instance. This is typically the DB2 instance owner, but in some customer configurations they may use a different id such as root. For newer releases the ulimits are not inherited.
$ ulimit -c unlimited
Soft Limits
$ ulimit -aS time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2000000
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) unlimited
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited
Hard Limits
$ ulimit -aH
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user) unlimited
(C) Optional step if diagnosing db2wdog crash
Corefiles for db2wdog are written to /core instead of the FODC DUMPDIR location because this process is started by root.
Create a symbolic link for the corefile so that it is output to another filesystem with more space (optional). In my example the corefile is created in /tmp/core. Core files can be several gigabytes in size.
$ ln -s /tmp/core /core
Note we link to the filename /tmp/core, there is no directory called "core" under /tmp.
$ ls -ld /core
lrwxrwxrwx 1 root system 24 Apr 19 07:37 core -> /tmp/core
(D) Optional step if diagnosing db2fmp crash
db2set DB2_ROUTINE_DUMPCORE=ON
db2stop
db2start
(E) Enable DB2's FODC and check configuration
Enable coredumps. The hard and soft core limit should be unlimited, if not recheck the ulimit settings. The ulimits will only take effect after DB2 instance is restarted.
$ db2pdcfg -fodc dumpshm=on
Database Member 0
FODC (First Occurrence Data Capture) options:
Dump directory for large objects (DUMPDIR)= /home/db2inst1/sqllib/db2dump/
Dump Core files (DUMPCORE)= ON
Current hard core file size limit = Unlimited
Current soft core file size limit = Unlimited
Dump shared memory (DUMPSHM)= ON
Changes to FODC Options have been applied. To make these changes permanent use db2set DB2FODC.
DB2's core files will be written to the location specified by DUMPDIR. Use the db2pdcfg command to check or modify this.
$ db2pdcfg -fodc
FODC (First Occurrence Data Capture) options:
Dump directory for large objects (DUMPDIR)= /home/db2inst1/sqllib/db2dump
Dump Core files (DUMPCORE)= AUTO
Current hard core file size limit = Unlimited
Current soft core file size limit = Unlimited
To change this to a different directory which as more space use:
$ db2pdcfg -fodc dumpdir=/data1
Database Member 0
FODC (First Occurrence Data Capture) options:
Dump directory for large objects (DUMPDIR)= /data1
Service level (SERVICELEVEL)= AUTOMATIC (Effective: FULL)
Dump Core files (DUMPCORE)= AUTO (Effective: ON)
(F) Restart DB2 instance for new settings to take effect.
db2stop
db2start
Note: On Redhat Enterprise Linux 6.x and 7.x, the ABRT (Automatic Bug Reporting Tool) will prevent DB2 from dumping a core file.
Sample to stop abrtd
# service abrtd status
Redirecting to /bin/systemctl status abrtd.service
abrtd.service - ABRT Automated Bug Reporting Tool
Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled)
Active: active (running) since Sun 2017-05-07 08:54:04 EDT; 1 weeks 2 days ago
Main PID: 679 (abrtd)
CGroup: /system.slice/abrtd.service
+-679 /usr/sbin/abrtd -d -s
May 07 08:54:04 test systemd[1]: Starting ABRT Automated Bug Reporting Tool...
May 07 08:54:04 test systemd[1]: Started ABRT Automated Bug Reporting Tool.
May 07 08:54:05 test abrtd[679]: Init complete, entering main loop
# service abrtd stop
Redirecting to /bin/systemctl stop abrtd.service
# service abrtd status
Redirecting to /bin/systemctl status abrtd.service
abrtd.service - ABRT Automated Bug Reporting Tool
Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled)
Active: inactive (dead) since Tue 2017-05-16 11:56:29 EDT; 4s ago
Process: 27795 ExecStart=/usr/sbin/abrtd -d -s (code=killed, signal=TERM)
Main PID: 27795 (code=killed, signal=TERM)
Check with your system administrator if help is needed. It may be possible to stop ABRT for only specific Process ID.
Reference: Chapter 28: Automatic Bug Reporting Tool (ABRT)
(G) Collecting the corefile generated after a crash
AIX:
snapcore -d /tmp /home/db2inst1/sqllib/db2dump/50855994.000.core/core /home/db2inst1/sqllib/adm/db2sysc
where
/tmp = The location of the output file snapcore*.pax is created
/home/db2inst1/sqllib/adm/db2sysc = The executable which created the corefile
/home/db2inst1/sqllib/db2dump/50855994.000.core/core = the corefile itself
For db2fmp crashes the executable would be /home/db2inst1/sqllib/adm/db2fmp
snapcore will create *.pax file in /tmp.
Linux/Solaris:
DB2 includes is own db2snapcore command, if this is not available use attached db2snapcore.tar script. Untar before running the Perl script.
Running ./db2snapcore will provide the usage.
Example this collects all the libraries used by the corefile into output directory /tmp.
db2snapcore -outdir /tmp -corefile $DB2_HOME/db2dump/5085.000.core/core.5085 -trapfile $DB2_HOME/db2dump/FODC_Trap_2017-01-01-08.56.38.458307_0000/5085.000.000.trap.txt -e $DB2_HOME/adm/db2sysc
Notes:
- $DB2_HOME points to /home/db2inst1/sqllib
- -trapfile is optional on Solaris
- -e is the executable that crashed this is normally a thread in db2sysc process. See trapfile to confirm the process.
- This script requires /usr/bin/pax to be installed.
(H) Collect db2support.zip
db2support /tmp -d <db name> -g -s
This creates db2support.zip in /tmp
(I) Send the corefile db2support.zip to DB2 Support.
AIX: Send the *.pax file created by snapcore
Linux/Solaris: db2snapcore.tar
Please refer to the link "Submitting diagnostic information to IBM Technical Support..."
Related Information
Was this topic helpful?
Document Information
Modified date:
08 November 2018
UID
swg21499458