Troubleshooting
Problem
Resolution for a specific ibmslapd crash condition in a environment with custom OS password plugin.
Symptom
ibmslapd startup failed with a Memory Fault:
GLPSRV041I Server starting.
GLPCTL113I Largest core file size creation limit for the process (in bytes): '0'(Soft limit) and '-1'(Hard limit).
GLPCTL119I Maximum Data Segment(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 262144.
GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is 2097152.
GLPCTL122I Maximum Open Files soft ulimit for the process is 1024 and the prescribed minimum is 500.
GLPCTL122I Maximum Stack Size(Kbytes) soft ulimit for the process is 10240 and the prescribed minimum is 10240.
GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 1048576.
GLPCOM024I The extended Operation plugin is successfully loaded from libevent.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libtranext.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libldaprepl.so.
GLPSRV155I The DIGEST-MD5 SASL Bind mechanism is enabled in the configuration file.
GLPCOM021I The preoperation plugin is successfully loaded from libDigest.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libevent.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libtranext.so.
GLPCOM023I The postoperation plugin is successfully loaded from libpsearch.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libpsearch.so.
GLPCOM025I The audit plugin is successfully loaded from libldapaudit.so.
GLPCOM021I The preoperation plugin is successfully loaded from libibmldapcl.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libevent.so.
GLPCOM023I The postoperation plugin is successfully loaded from libpsearch.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libpsearch.so.
GLPCOM022I The database plugin is successfully loaded from libback-config.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libevent.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libtranext.so.
GLPCOM023I The postoperation plugin is successfully loaded from libpsearch.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libpsearch.so.
GLPCOM022I The database plugin is successfully loaded from libback-rdbm.so.
GLPCOM010I Replication plugin is successfully loaded from libldaprepl.so.
GLPCOM021I The preoperation plugin is successfully loaded from libibmldapcl.so.
GLPCOM021I The preoperation plugin is successfully loaded from libpta.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libevent.so.
GLPCOM023I The postoperation plugin is successfully loaded from libpsearch.so.
GLPCOM024I The extended Operation plugin is successfully loaded from libpsearch.so.
GLPCOM021I The preoperation plugin is successfully loaded from libibmldapcl.so.
GLPCOM022I The database plugin is successfully loaded from libback-rdbm.so.
Memory fault
Cause
Custom OS password (user/group) plugin configured in operating system caused this crash condition. The custom OS password plugin library is not visible in ibmslapd process space. Symbol resolution problem arises during ibmslapd start up when libdb2 verifies the user with operating system. This causes ibmslapd to crash.
Environment
Linux - RHEL
Tivoli Directory Server 6.*
Diagnosing The Problem
#cat /proc/meminfo showed that there is enough memory on the system:
3895408 KB ( ~3.8 gig) real mem total.
~2.8 gig free.
idsilist -a showed ldapdb2 on default ports location is /home/ldapdb2.
There are no indications in ibmslapd.log and db2cli.log files.
Enabled the ulimit for core file using the following:
ulimit -c unlimited
ulimit -Sc unlimited
Started ibmslapd after this and a core file got generated, then used
gdb /opt/ibm/ldap/V6.1/sbin/32/ibmslapd core.19317
where core file showed the following stack trace:
#0 0x00890a1e in readFile () from /opt/ibm/ldap/V6.1/lib/libidsosutils.so
#1 0x00d8e4e4 in _passwd_load () from /lib/libnss_cwa.so.2
#2 0x00d8eba4 in _nss_cwa_getpwnam_r () from /lib/libnss_cwa.so.2
#3 0x007867b2 in getpwnam_r@@GLIBC_2.1.2 () from /lib/tls/libc.so.6
#4 0x0014ead59 in sqloGetUserAttribByName () from /home/ldapdb2/idsslapd-ldapdb2/db2instance/lib/libdb2.so
... (a bunch of calls in libdb2.so)
Reviewed the /etc/nsswitch.conf and found that the passwd and group are using some custom password/user/group plugin cwa.
passwd: files cwa
group: cwa
cwa is a custom password (user/group) plugin.
Resolving The Problem
First verify if the following libs are present on the system and then export LD_PRELOAD as below:
export LD_PRELOAD="/lib/libnss_cwa.so.2 /lib/libnss_compat.so.2"
Start ibmslapd from the same shell where the above export is done.
The above export command can be put into the profile for the root and instance owner so that ibmslapd can be aware of the custom plugin and load it for proper symbol resolution.
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21499819