Troubleshooting
Problem
java.lang.OutOfMemoryError: Java heap space - Socket Listener error shows in noapp.log -GIS 4.3
Symptom
Sporadically OutOfMemory errors on node 2 of a 2 node GIS cluster. Currently
running GIS 4.3 on a windows platform using MS SQL 2005. No other infromation
available as there are no heapdumps or javacores available.
[2010-09-17 11:05:49.174] ALL 000000000000 GLOBAL_SCOPE Exception in thread
"SocketListener1-1"
[2010-09-17 11:05:49.174] ALL 000000000000
GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17
11:12:32.69] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:32.705] ALL
000000000000 GLOBAL_SCOPE Exception in thread "Thread-1573"
[2010-09-17
11:12:32.721] ALL 000000000000 GLOBAL_SCOPE Exception in thread "Thread-23"
[2010-09-17 11:12:32.736] ALL 000000000000 GLOBAL_SCOPE Exception in thread
"Thread-26"
[2010-09-17 11:12:32.736] ALL 000000000000 GLOBAL_SCOPE
Exception in thread "RMI RenewClean-[aaa.bbb.ccc.ddd:xxxx]"
[2010-09-17
11:12:32.752] ALL 000000000000 GLOBAL_SCOPE Exception in thread "JNP Server"
[2010-09-17 11:12:32.768] ALL 000000000000 GLOBAL_SCOPE Exception in
thread "RMI
RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi.
AddrClientFactory@139746e]"
[2010-09-17 11:12:34.173] ALL 000000000000
GLOBAL_SCOPE Exception in thread "RMI
RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi.
AddrClientFactory@104e432]"
[2010-09-17 11:12:34.173] ALL 000000000000
GLOBAL_SCOPE Exception in thread "RMI
RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi.
AddrClientFactory@17d97e5]"
[2010-09-17 11:12:34.188] ALL 000000000000
GLOBAL_SCOPE Exception in thread "RMI
RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi.
AddrClientFactory@6d3947]"
[2010-09-17 11:12:34.188] ALL 000000000000
GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17
11:12:34.204] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17
11:12:34.204] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java
heap space
[2010-09-17 11:12:35.672] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:35.687] ALL 000000000000 GLOBAL_SCOPE
java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:35.687]
ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.091] ALL
000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.091] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.122] ALL 000000000000 GLOBAL_SCOPE
java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.122]
ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.138] ALL
000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.138] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.153] ALL 000000000000 GLOBAL_SCOPE
java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.169]
ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.169] ALL
000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.184] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.184] ALL 000000000000 GLOBAL_SCOPE
java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.2] ALL
000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:39.2] ALL
000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.216] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:13:10.768] ALL 000000000000 GLOBAL_SCOPE MIC value MIC Value
Length: 20
00000001 6b 68 6a ce 67 ad 13 db 19 53 fb f6 19 6b 75 54
khj?g?.?.S??.kuT
00000011 4b 76 06 ff Kv.?
[2010-09-17 11:13:10.768] ALL 000000000000 GLOBAL_SCOPE Attempting to use
signature provider Certicom
[2010-09-17 11:13:10.768] ALL 000000000000
GLOBAL_SCOPE Calling initSign...
[2010-09-17 11:13:10.768] ALL
000000000000 GLOBAL_SCOPE Adding signature...
[2010-09-17 11:13:13.578]
ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:13:13.578] ALL 000000000000 GLOBAL_SCOPE
Resolving The Problem
The main resource to analyze is verbose garbage collection logs. It was noted that when GIS got OutOfMemory errors, there were several large spikes of memory with FULL GCs running back to back. It is important to note the time that the large spikes of memory occurred to go back to all the BPs running at that time to investigate the problem.
After researching this issue, it was noted that there was a BP that was generating a really large process data. In addition that BP had around 30 or more child BPs and all of those BPs would inherit the process data from the parent. Process data is maintained in memory. Once the customer reworked the BP so that process data would not get so large, then the issue stopped occurring.
Historical Number
NFX9968
Product Synonym
[<p><b>]Escalation ID[</b><p>];00224437;[<p><b>]Severity[</b><p>];Normal;[<p><b>]Type[</b><p>];NormalFix
Was this topic helpful?
Document Information
Modified date:
14 November 2019
UID
swg21553636