IBM Java for AIX HowTo: Troubleshoot 'java -version' hang or timeout issue

Issue:

Java commands that use IPV6 hostname resolution hang or perform poorly.

Root Cause:

If the Domain Name System (DNS) server is not configured to handle IPv6 queries, the application may have to wait for a response from the DNS server until the IPv6 query times out.

Confirm:

Thread dumps or javacores taken during the time of the slow response or hang, will show threads with the following lines at the top of the stack:

at java.net.Inet6AddressImpl.getLocalHostName(Native Method)
at java.net.InetAddress.getLocalHost(InetAddress.java:123)
.....................

Resolution:

If your environment only uses IPv4, add the following option to the failing java command:

-Djava.net.preferIPv4Stack=true

This will disable Java application IPv6 lookup requests and only use IPv4.

Issue:

Java command experiences a hang or timeout when communicating with the DNS server.

Confirm:

To confirm the issue as DNS accessability, follow the instructions in the "Prepare", "Actions" and the "Examples" sections from:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024364

Determine if your system is using DNS or /etc/hosts, using the 'Prepare' section of the technote above. If DNS is being used, check if the connection is failing or not using the 'Actions' and 'Examples' section from the technote. If DNS is being used and host name resolution is failing, then you can confirm that there is an issue with the DNS server that your system is trying to connect to.

Resolution:

If the issue gets resolved by using /etc/hosts hostname resolution, look into DNS connection issue.

Issue:

An application does not cleanup semaphores (a.k.a., semaphore leak).

Confirm:

AIX has a maximum limit of 65535 semaphores per ID (click here for more details). There have been reported situations when applications (Java or non-Java) are not correctly deleting semaphores, resulting in a gradual semaphore leak. When the number of semaphores reaches the maximum limit for the user id, the JVM is unable to create the required semaphores on startup, resulting in the "Port Library" error message.

To confirm if the semaphore leak is occurring, from a command prompt, repeatedly execute the commands:

# ipcs -saPrX | wc -l

over a period of time. If the number of semaphores shows a continuous increase, then there is a semaphore leak.

Resolution:

a. Rebooting the system will provide temporary relief for the issue. However, over time, the issue is expected to reoccur.

b. If IBM InfoSphere or IBM DataStage is running on the same system, then follow the instructions on the web page:

http://www-01.ibm.com/support/docview.wss?uid=swg21654008

c. Open a support call to the vendor / team for the application (not the JVM) creating, but not deleting the semaphores. That vendor / team will be responsible for resolving the issues of the semaphores not being released.

Issue:

A known issue in a specific version and release of IBM Java that has been fixed in the later releases.

Resolution:

To rule out if the issue has already been fixed in one of the later releases of the Java version, please upgrade Java to the latest release of the Java version as per:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022692

Java fixes are cumulative and upgrading to the latest release will include all the fixes released so far for the Java version.

If you are already at the latest release for a Java version and the issue persists, to rule out any corrupted IBM Java Installation due to some corrupted or overwritten shared library files contributing to the issue, do the following:

1. Uninstall, then re-install that version of IBM Java for AIX
2. Retry the java command

Issue:

The RAM/physical memory is over-committed on the AIX LPAR.

Confirm:

To check if the memory is overcommitted on the system or not, monitor the output of the following 'vmstat' command, by repeatedly running it:

# vmstat -It 2 25 > vmstat.out &

Sustained paging activity is the best indication of low memory. If the "page-in" (PI) and "page-out" (PO) columns show non-zero values over "long" periods of time, then the system is short on memory. (All systems will show occasional paging, which is not a concern.)

Resolution:

To understand, optimize and tune memory, please look into the following resource and use the tunable 'vmo' with appropriate options:

https://www.ibm.com/developerworks/aix/library/au-aix7memoryoptimize1/
https://www.ibm.com/developerworks/aix/library/au-aix7memoryoptimize2/
https://www.ibm.com/developerworks/aix/library/au-aix7memoryoptimize3/

For example, if the system is configured with high number of large pages(16MB pages),which are always pinned and most of the applications on the system, are not configured to use large pages and cannot use this pinned large page memory. Hence, less memory is available to these applications. This will result in the applications constantly waiting for page-ins and this would appear as application/command hang or slowness to the user.

To check the allocated and used large pages, run the command 'svmon -G' and the output is something similar to:

# svmon -G

If you are not using those large pages on your LPAR(check the value under the inuse column, in the large pages row similar to the line highlighted above in the 'svmon -G' ouput. This 'inuse' value has to be zero in the svmon output), we suggest you to disable large pages or reduce their number. To disable large pages use the following command:

# vmo -p -d lgpg_regions

The above command will set the number of large pages to 0. Or if you want to set the number of large pages to any other lesser number they can be set using the following command:

# vmo -p -o lgpg_regions= -o lgpg_size=16777216

If, after following the above instructions, the 'java -version' continues to hang or timeout, please complete the following steps:

1. Confirm that you have completed all of the above steps.
2. Contact IBM and open a new IBM service request (new PMR).
3. Collect and upload data as per the data collection procedure for Java hang.