Collecting a thread dump on AIX.

Technote (troubleshooting)


Problem(Abstract)

If the ITDS server appears hung or has generated a core file, it is important to collect a thread dump in order to identify the reason for the hang or crash condition.

Resolving the problem

Purpose: Gather ibmslapd (or any) processes thread stacks. Usually used when the ibmslapd server process is hung but it can also be used for other processes as well.

Requirements:


    AIX with the bos.adt.debug (dbx) installed

    ulimit -c unlimited for the process we are collecting the thread dump for

    The dumpthreads.sh script should be copied to the system and must have execute permission.


Usage: dumpthreads.sh [ PID | program corefile ]

To verify you have the bos.adt.debug installed on your system, issue the following command:

# lslpp -l | grep bos.adt.debug

**Note: if this package is not installed, you can install from the AIX operating system media.

1) To get a thread dump from a hung server:

In the case of a hung process, you will want to collect multiple thread dumps in succession. Typically 2-3 thread dumps about a minute apart is helpful. This will show whether or not the processes' threads are truly hung or if there is progress being made.

1.1) Find the process ID (PID) of the ibmslapd server

From a command prompt, type:

# ps -ef | grep slapd

**Note: if you are trying to collect a thread dump from a process other than ibmslapd, you will need to grep for that process.

1.2) Collect the thread dump

From a command prompt, invoke the dumpthreads.sh script with the process ID.

# <path>/dumpthreads.sh PID > /tmp/threaddump1.out

where PID is the process ID obtained in step 1.1

e.g.: Assuming dumpthreads.sh is in /tmp folder

# /tmp/dumpthreads.sh 12345 > /tmp/threaddump1.out

Repeat the above command multiple times at one minute intervals redirecting the output into different files.

2) To get a thread dump from a core file:

Unlike a process hang condition as described above, collecting a thread dump from a core file is a single collection. From the thread dump we are trying to identify the signature of the core file so that we can attempt to identify root cause.

Usage:

# <path>/dumpthreads.sh PROGRAM CORE > /tmp/threaddump.out

where PROGRAM is the fully qualified path to the ibmslapd executable or the executable which generated the core,

and CORE is the generated core file.

ibmslapd executable PROGRAM locations:

ITDS 6.3: /opt/IBM/ldap/V6.3/sbin/64/ibmslapd
ITDS 6.2: /opt/IBM/ldap/V6.2/sbin/64/ibmslapd
ITDS 6.1: /opt/IBM/ldap/V6.1/sbin/64/ibmslapd
ITDS 6.0: /opt/IBM/ldap/V6.0/sbin/64/ibmslapd
ITDS 5.2: /usr/ldap/bin/ibmslapd

Other Examples of executable program locations:

ITDS 6.3: /opt/IBM/ldap/V6.3/sbin/64/bulkload
ITDS 6.2: /opt/IBM/ldap/V6.2/sbin/64/ldif2db
ITDS 6.1: /opt/IBM/ldap/V6.1/sbin/64/db2ldif
ITDS 6.0: /opt/IBM/ldap/V6.0/sbin/64/bulkload

CORE file locations:

ITDS 6.x: <Location>/idsslapd-<instanceName>/workdir/
ITDS 5.2: Location from where ibmslapd process is started.

Verify the the program which generated the CORE:
# file core
core: AIX core file fulldump 64-bit, ibmslapd

Run the dumpthreads script to collect the threads dump into a file:
e.g.: Assuming the dumpthreads.sh is kept in /tmp

# /tmp/dumpthreads.sh /opt/IBM/ldap/V6.3/sbin/64/ibmslapd core > /tmp/threaddump.out

dumpthreads.sh -> dumpthreads.shdumpthreads.sh

Rate this page:

(0 users)Average rating

Document information


More support for:

IBM Security Directory Server
General

Software version:

5.2, 6.0, 6.1, 6.2, 6.3

Operating system(s):

AIX

Reference #:

1268413

Modified date:

2010-03-04

Translate my page

Machine Translation

Content navigation