APAR status
Closed as program error.
Error description
Detail "java/lang/OutOfMemoryError" "Java heap space" received The heapdump is showing a large array of the following type also look to be related to the objectgrid component - 2,882,770,784 (65.47%) [40,945,032] 5,639,280 array of java/util/concurrent/RunnableScheduledFuture 0x7d07b9500 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c800 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c868 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c8d0 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c938 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c9a0 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987ca08 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987ca70 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cad8 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cb40 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cba8 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cc10 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cc78 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cce0 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cd48 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cdb0 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987ce18 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987ce80 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cee8 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cf50 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987cfb8 |- There are 5,639,260 more children Drilling down into these child objects i can see they hold the following - 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c800 |- 424 (0%) [24] 1 java/util/concurrent/Executors$RunnableAdapter 0x7f987c850 |- 400 (0%) [32] 1 com/ibm/ws/objectgrid/server/security/access/SecurityCache$Refre shRunnable 0x7fa671d90 |- 368 (0%) [16] 2 com/ibm/ws/objectgrid/server/security/access/CredentialWithBytes 0x7fa671cb8 |- 248 (0%) [16] 2 com/ibm/websphere/objectgrid/security/plugins/builtins/UserPassw ordCredential 0x7fa671d08 | |- 160 (0%) [24] 1 java/lang/String 0x7fa671cc8 | | |- 136 (0%) [136] 0 char[] 0x7fa671c38 | |- 72 (0%) [24] 1 java/lang/String 0x7fa671d18 | |- 48 (0%) [48] 0 char[] 0x7fa671ce0 |- 104 (0%) [104] 0 byte[] 0x7fa671d30
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: Users of eXtreme Scale Liberty Deployment. * **************************************************************** * PROBLEM DESCRIPTION: Client faced OOM due to slow LDAP * * response. * **************************************************************** * RECOMMENDATION: * **************************************************************** The OOM is caused by a large array of objects related to to wxs object grid component. Slow LDAP responses caused multiple objects to be created. Symptoms in the logs - Javacore shows exceptionally large number of threads (e.g. 3000+) which have security cache in their stack. Heapdump shows large arrays of - java/util/concurrent/RunnableScheduledFuture type occupying a major section of the heap. 2,882,770,784 (65.47%) [40,945,032] 5,639,280 array of java/util/concurrent/RunnableScheduledFuture 0x7d07b9500 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c800 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c868 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c8d0 |- 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c938 |- There are 5,639,260 more children These ScheduledFutureTask objects are referenced by - com/ibm/websphere/objectgrid/security/plugins/builtins/UserPassw ordCredential 504 (0%) [80] 2 java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFuture Task 0x7f987c800 |- 424 (0%) [24] 1 java/util/concurrent/Executors$RunnableAdapter 0x7f987c850 |- 400 (0%) [32] 1 com/ibm/ws/objectgrid/server/security/access/SecurityCache$Refre shRunnable 0x7fa671d90 |- 368 (0%) [16] 2 com/ibm/ws/objectgrid/server/security/access/CredentialWithBytes 0x7fa671cb8 |- 248 (0%) [16] 2 com/ibm/websphere/objectgrid/security/plugins/builtins/UserPassw ordCredential 0x7fa671d08 | |- 160 (0%) [24] 1 java/lang/String 0x7fa671cc8 | | |- 136 (0%) [136] 0 char[] 0x7fa671c38 | |- 72 (0%) [24] 1 java/lang/String 0x7fa671d18 | |- 48 (0%) [48] 0 char[] 0x7fa671ce0 |- 104 (0%) [104] 0 byte[] 0x7fa671d30 The OOM is caused by build up of both threads and objects related to the objectgrid SecurityCache.
Problem conclusion
Fixed Security cache code to prevent OOM issue even when LDAP response is very slow or in hung state.
Temporary fix
Comments
APAR Information
APAR number
PH19249
Reported component name
WS EXTREME SCAL
Reported component ID
5724X6702
Reported release
860
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2019-11-18
Closed date
2020-03-09
Last modified date
2020-03-09
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WS EXTREME SCAL
Fixed component ID
5724X6702
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSTVLU","label":"WebSphere eXtreme Scale"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"860","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
27 March 2020