IBM JAVA FOR AIX MustGather: Data collection procedure for UnsatisfiedLinkError

Exceptions similar to ones listed below will be written to standard output (stdout), standard error (stderr), or to an application log file, depending on the application's configuration:




Exception in thread "main" java.lang.UnsatisfiedLinkError: helloworld (A file or directory in the path name does not exist.)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1008)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:972)
at java.lang.System.loadLibrary(System.java:470)
at JavaNativeHelloWorld.(JavaNativeHelloWorld.java:17)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) Could not find the main class: JavaNativeHelloWorld. Program will exit.



Exception in thread "main" java.lang.UnsatisfiedLinkError: /usr/java71_64/jre/lib/ppc64/xawt/libmawt.so (Could not load module /usr/java71_64/jre/lib/ppc64/xawt/liblibmawt.so.so. System error: No such file or directory)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1542)
at java.lang.System.load(System.java:549)
at java.lang.ClassLoader.loadLibraryWithPath(Native Method)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1534)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1506)
at java.lang.System.loadLibrary(System.java:562)
at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:79)



java.lang.UnsatisfiedLinkError: no nlsxbe in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at lotus.domino.NotesThread.load(NotesThread.java:299)
at lotus.domino.NotesThread.checkLoaded(NotesThread.java:320)
at lotus.domino.NotesThread.initThread(NotesThread.java:140)
at lotus.domino.NotesThread.run(NotesThread.java:202)

java.lang.UnsatisfiedLinkError: Native Library "foo" already loaded in another classloader
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1715)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1646)
at java.lang.Runtime.load0(Runtime.java:787)
at java.lang.System.load(System.java:1022)
..........



Exception in thread "main" java.lang.UnsatisfiedLinkError: bt3 (0509-022 Cannot load module /path/to/the/library/libbt3.a.
0509-103 The module has an invalid magic number.)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1006)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:970)
at java.lang.System.loadLibrary(System.java:501)
.........



java.lang.UnsatisfiedLinkError: Native library not found. Tried to load tibrvnative64 and tibrvnative
at com.tibco.tibrv.Tibrv.loadNativeLibrary(Tibrv.java:392)
at com.tibco.tibrv.Tibrv.(Tibrv.java:79)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at com.tibco.tibrv.TibrvMsg.(TibrvMsg.java:25)



UnsatisfiedLinkError exception loading native library: njni12
java.lang.UnsatisfiedLinkError: njni12 (rtld: 0712-001 Symbol CreateIoCompletionPort was referenced
from module /oravl01/oracle/app/product/12.1.0.2/dbhome_1/lib/libnjni12.so(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol GetMultipleCompletionStatus was referenced
from module /oravl01/oracle/app/product/12.1.0.2/dbhome_1/lib/libnjni12.so(), but a runtime definition
of the symbol was not found.
rtld: 0712-002 fatal error: exiting.)



java.lang.UnsatisfiedLinkError: jnilib (No such file or directory)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1018)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:982)
at java.lang.System.loadLibrary(System.java:506)
at com.betasystems.jdmc.jni.JDMConnector.(JDMConnector.java:626)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
at com.betasystems.jdmc.jni.JDMC.(JDMC.java:80)
at com.betasystems.jdmc.rmi.JDMCServer.main(JDMCServer.java:813)


The above exception stacks are examples only. The exception stack trace and class names may be different depending on the application and situation.

To search for the failing Native library, prefix the library name with "lib" and suffix with ".so".

Example:

For the failing native library "helloworld" in the exception :

java.lang.UnsatisfiedLinkError:helloworld ....

..............



search for the library by prefixing with "lib" and suffixing with ".so" :

libhelloworld.so

Before proceeding with the setup and data collection instructions to collect complete diagnostic data for analysis, click Step 7. Upload Data for instructions to upload any existing generated logs and data collected.

To prepare for these data collection procedures, the process environment needs to be configured to collect complete data and save the additional debug information to a log file.



A. Set the user ulimits:

From a command prompt and while logged in as the root user, execute following commands to:

{ set file and data ulimit sizes to unlimited }

# chuser fsize=-1 data=-1 USERID

B. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".

C. Perform the following actions in order for the changes to take effect:

- Stop the application

- Relogin as the "USERID" used in Step 2.A

- Confirm the new ulimits are in effect by executing the commands:

# ulimit -a



- Do not restart the application until Step 3. Configure has been completed.

A. Enabling debug options will result in additional data being stored in memory buffers and written to application logs. The process file and data sizes should be increased during the data collection to ensure the data is complete.

If there are multiple processes executed by multiple user ids experiencing the issue, then all preparation steps must be repeated for each id and process.

To confirm the process environment is configured correctly, login using the USERID specified in the steps, then run the command:

# ulimit -a

The values for "file" and "data" should show as:
file(blocks) unlimited
data(kbytes) unlimited

B. As an example, to save standard error message to the file /tmp/stderr.log, use a command line syntax similar to:

# java YOUR_APP 2>/tmp/stderr.log

to save both standard out and standard error messages to the file /tmp/out.log, specify:

# java YOUR_APP > /tmp/out.log 2>&1

To confirm the messages are being redirected to the log file, view the contents of the log file.

A. Locate the failing library:

Locate the failing native library. To search, add the prefix of "lib" and suffix of ".so" to the library name in the exception :


# export LIBNAME=libLIBRARY.so


# export LIBNAME_PATH="$(find JAVA_PATH -name $LIBNAME -print)"



Confirm the library exists by making sure full path to the library along with the library name is output with:


# echo $LIBNAME_PATH 2>&1 | tee -a LIBNAME-PATH.out



If the command output :

1. is null, after making sure the library exists, continue with the rest of the instructions only if the problem persists.


2. shows mutiple entries with library in different directories, repeat Step B for each of entries. If the problem persists, continue with the rest of the instructions starting with Step C.



B. Set environment variables

Make sure path to the native library is set :

# export PATH="$(dirname $LIBNAME_PATH)":$PATH

# export LIBPATH="$(dirname $LIBNAME_PATH)":$LIBPATH




C. Add java command line options:

Enable verbose logging of the class loading and debug output:

-verbose:class,dynload,debug


Enable manual generation of AIX core file, javacores and snap traces:

-Xdump:system+java+snap:events=user



Generate AIX core file, Javacore and Snap trace files when the exception is thrown:

-Xdump:system+java+snap:events=systhrow+throw,filter=EXCEPTION,range=1..1



Enable tracing of the entry and exit of all the static initializers of the failing library:


"-Xtrace:maximal=all,methods={*.*()},resumecount=1,trigger=method{NATIVE-LIBRARY,resumethis,suspendthis},output=j9snap.trc"




To designate a directory for the generated logs, instead of the default directory, add:

-Xdump:directory=SPECIFIC_PATH



D. Restart the java application (e.g., node agent/manager) from the USERID new login session.

A. Locate the failing library:

For the failing native library "helloworld" from Step 1, search for the library as below:


Example:
# export LIBNAME=libhelloworld.so

# echo $LIBNAME
libhelloworld.so

# export LIBNAME_PATH=$(find /app/dir -name $LIBNAME -print)

# echo $LIBNAME_PATH
/app/dir/libhelloworld.so




C . Alternate ways of setting java command line options and non-default directory for generated log files:

Environment variable can be used to set Java command line options. However, options specified on java command line take precedence over environment variable settings:

# export IBM_JAVA_OPTIONS="-verbose:gc -Xverbosegclog:SPECIFIC_PATH/gc.log -Xdump:system+java+heap+snap:events=systhrow+throw,filter=java/lang/UnsatisfiedLinkError,range=1..1"



For exception in step1:

Exception in thread "main" java.lang.UnsatisfiedLinkError:helloworld
......


specify EXCEPTION as :

-Xdump:java+snap:events=systhrow+throw,filter=java/lang/UnsatisfiedLinkError,range=1..1



To enable tracing for the native library in the first stack trace in Step 1,:

......

at JavaNativeHelloWorld.(JavaNativeHelloWorld.java:17)
JavaNativeHelloWorld.
...........

specify:
"-Xtrace:maximal=all,methods={*.*()},resumecount=1,trigger=method{JavaNativeHelloWorld./clinit/,resumethis,suspendthis},output=j9snap.trc"

Replace "/" around "clinit" above with angle brackets.


To direct javacore and snap trace to specific directories instead of the default directory of START_PATH, set environment variables:

# export IBM_JAVACOREDIR=SPECIFIC_PATH



D. For application server environments such as WebSphere and WebLogic, both the node manager/agent and the application servers must be restarted after the relogin of "USERID".

A. Execute the following commands to collect the required diagnostic data.

# mkdir -p /TMP_PATH/MM-DD/UnsatisfiedLinkError/data
# cd /TMP_PATH/MM-DD/UnsatisfiedLinkError/data


Generated logs are written to the default directory of "START_PATH" or the user specified SPECIFIC_PATH.



B. Tar up the failing native library parent directory and its sub-directories to check the contents:

# tar -cvh -f ./native-library-dir.tar $(dirname $LIBNAME_PATH)



C. Check the dependents of the native library:

# ldd $LIBNAME_PATH 2>&1 | tee -a ldd.out



D. Dump the header entries of the native library.

Dump the loader section header :

for 32-bit Java library:

# dump -THov $LIBNAME_PATH > dump.out


for 64-bit Java library:

# dump -X64 -THov $LIBNAME_PATH > dump-64.out



E. Start java process with truss enabled :

# truss -aelfdo native-library-truss.out JAVA-COMPLETE-COMMAND-LINE

OR

# truss -aelfd JAVA-COMPLETE-COMMAND-LINE > native-library-truss.out




F. Collect output of AIX commands from the same AIX LPAR:

# errpt -a > errpt-a.out 2>&1
# oslevel -s > oslevel-s.out 2>&1
# prtconf > prtconf.out 2>&1
# lsps -a > lsps-a.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# ps avwwwg > ps-avwwwg.out 2>&1



G. Copy the files generated to the data directory "/TMP_PATH/MM-DD/UnsatisfiedLinkError/data" directory created in step 4.A:

# cp /START_PATH/core* ./
# cp /START_PATH/javacore*txt ./
# cp /START_PATH/Snap*trc ./
# cp /START_PATH/j9snap.trc ./

Also copy standard error, standard output, SystemOut, SystemErr, gc.log, application logs and any other logs generated to the /TMP_PATH/MM-DD/UnsatisfiedLinkError/data directory.

A. Examples of commands to be executed:

# mkdir -p /large_fs/01-31UnsatisfiedLinkError/data
# cd /large_fs/01-31/UnsatisfiedLinkError/data



If "START_PATH" can not easily be determined, execute these commands to help determine the correct path:

Example:

# ps -ef | grep -i java

rt 3211380 1 0 May 31 - 1109:46 java -Dsrse_property=/rt/pmr/test/classloader/JvmTest-dir/j_test/etc/aib.ini....


to check where the generated javacore and other logs get written, excute:

# kill -3 3211380


# procwdx 3211380
3211380: /rt/pmr/test/


Check for the generated javacore and other logs in the the above directory:

# cd /rt/pmr/test
# ls -l *javacore*

-rw-r--r-- 1 rtstaff 234048 Aug 25 10:54 javacore.20150825.105402.3211380.0010.txt


C. Check native library dependencies :

Example:


# ldd /usr/java7_64/jre/lib/ppc64/xawt/libmawt.so
libmawt.so needs:
Cannot find libawt.so
/lib/libXext.a(shr_64.o)
/lib/libX11.a(shr_64.o)
/lib/libXrender.a(shr_64.o)
/lib/libXtst.a(shr_64.o)
/lib/libXi.a(shr_64.o)
Cannot find libjava.so
/usr/java8_64/jre/bin/classic/libjvm.so
Cannot find libdbgwrapper70.so
/lib/libc.a(shr_64.o)
/lib/libpthreads.a(shr_xpg5_64.o)
/lib/libgaimisc.a(shr_64.o)
/lib/libgair4.a(shr_64.o)
/lib/libiconv.a(shr4_64.o)
/unix
/lib/libIM.a(shr_64.o)
/lib/libc_r.a(shr_64.o)
/lib/libcrypt.a(shr_64.o)
/lib/libodm.a(shr_64.o)



E. Enable truss for the failing Java command line:

Example:

# truss -aelfdo native-library-truss.out
/usr/java7/bin/java -jar
-Xdump:system+java+snap:events=systhrow+throw,filter=java/lang/Unsatisfi edLinkError,range=1..1 "-Xtrace:maximal=all,methods={*.*()},resumecount=1,trigger=method{JavaNativeHelloWorld./clinit/,resumethis,suspendthis},output=j9snap.trc" TestReportes.jar ./Letter.asper test

Replace "/" around "clinit" above with angle brackets.


G. Confirm all files and directories have been saved to the data directory:

# cp /var/myapp/*.log ./

# ls *core* *nap*

# ls *.log *.out *.txt System*

** MANDATORY **

Prior to packaging and uploading the data, confirm that the following files have been saved in the "/TMP_PATH/MM-DD/UnsatisfiedLinkError/data directory:

a. AIX core files
b. Javacore files
c. Snap traces
d. ldd and dump commands output
e. Tar file of the native library parent and sub-directories
f. Truss output
g. standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.
h. AIX commands output

After all of the files have been collected, from a command prompt, execute the following commands to package the requested files:

# cd /TMP_PATH/MM-DD/UnsatisfiedLinkError
# tar -cf - data | gzip -c > PMR.MM-DD.tgz

Examples of commands to be executed:

# cd /large_fs/01-31/UnsatisfiedLinkError
# tar -cf - data | gzip -c > 12345.678.000.01-31.tgz

Upload the packaged data to IBM secured servers using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619


If this step is reached by clicking "Step 7. Upload Data" in Step 2, at the end of the completion of the existing latest logs/data upload, click Step 2. Prepare Environment to continue with the data collection instructions.