High CPU or hang symptoms may be observed on java.util.HashMap.findNonNullKeyEntry when a HashMap is utilized in a non-thread-safe manner. This is an unsafe coding practice that can occur at any level of development when using the HashMap API: application, application server, third party module, etc. The following will outline how to identify the problematic code and some possible methods to resolve the issue.
High CPU or hung threads with an executing stack similar to the following:
The specific stack may vary.
As a more general symptom of this issue, the executing stack will show the problematic code calling a method within the HashMap API which will in turn call the findNonNullKeyEntry() method.
Resolving the problem
In order to resolve these type of issues, the problematic code must first be identified.
In the examples above, the problematic code is com.xyz.methodABC(). This is determined by looking at the code that is making the call to the HashMap API. In both cases we see that com.xyz.methodABC() is calling the HashMap API, and therefore it is the problematic code. In the first example com.xyz.methodABC() is calling the get() method, and in the second example com.xyz.methodABC() is calling the put() method.
Once the problematic code has been identified, the developers of that code will need to review their usage of the HashMap API (Java 5, Java 6, Java 7). The discovery of high CPU or hung threads in states such as those described above is compelling evidence that the usage of the HashMap API has not been done in a thread-safe manner.
To correct the issue, the usage of HashMap objects should be done in a thread-safe (synchronized) manner as the API mandates. Alternatively, an object that is similar to a HashMap but intrinsically thread-safe could be used instead, a Hashtable or a ConcurrentHashMap for example.
HashMaps and WebSphere Application Server
As noted previously, this is not an issue that is unique to applications or third party modules. This is an issue that can occur at any level of development where the HashMap API is used, including WebSphere Application Server itself. There are a number of WebSphere Application Server APARs which address such issues. If one of these specific issues is encountered, it is recommended that the referenced Fix Pack (or greater) be applied to correct the issue.
APAR Affected Versions Released in Fix Pack
PK74719 7.0 22.214.171.124
PK97665 7.0 126.96.36.199
PM00692 6.1, 7.0 188.8.131.52, 184.108.40.206
PM02318 7.0 220.127.116.11
PM48600 7.0, 8.0 18.104.22.168, 22.214.171.124
PM44032 7.0, 8.0 126.96.36.199, 188.8.131.52
PM53640 7.0, 8.0 184.108.40.206, 220.127.116.11
A Final Note: IBM Java 6 SDK Enhancements
There have been some enhancements to the IBM Java 6 SDK to mitigate the likelihood of problems due to non-thread-safe usage of HashMap objects:
However, it is still the responsibility of the code using the HashMap objects to ensure that it is done in a thread-safe (synchronized) manner as the API requires. These enhancements cannot and will not protect against all improper usage of HashMap objects.