IBM Support

java.lang.OutOfMemoryError: Java heap space - Socket Listener error shows in noapp.log -GIS 4.3

Troubleshooting


Problem

java.lang.OutOfMemoryError: Java heap space - Socket Listener error shows in noapp.log -GIS 4.3

Symptom

Sporadically OutOfMemory errors on node 2 of a 2 node GIS cluster. Currently running GIS 4.3 on a windows platform using MS SQL 2005. No other infromation available as there are no heapdumps or javacores available.


Error Message

[2010-09-17 11:05:49.174] ALL 000000000000 GLOBAL_SCOPE Exception in thread "SocketListener1-1"
[2010-09-17 11:05:49.174] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:32.69] ALL 000000000000 GLOBAL_SCOPE
[2010-09-17 11:12:32.705] ALL 000000000000 GLOBAL_SCOPE Exception in thread "Thread-1573"
[2010-09-17 11:12:32.721] ALL 000000000000 GLOBAL_SCOPE Exception in thread "Thread-23"
[2010-09-17 11:12:32.736] ALL 000000000000 GLOBAL_SCOPE Exception in thread "Thread-26"
[2010-09-17 11:12:32.736] ALL 000000000000 GLOBAL_SCOPE Exception in thread "RMI RenewClean-[aaa.bbb.ccc.ddd:xxxx]"
[2010-09-17 11:12:32.752] ALL 000000000000 GLOBAL_SCOPE Exception in thread "JNP Server"
[2010-09-17 11:12:32.768] ALL 000000000000 GLOBAL_SCOPE Exception in thread "RMI RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi. AddrClientFactory@139746e]"
[2010-09-17 11:12:34.173] ALL 000000000000 GLOBAL_SCOPE Exception in thread "RMI RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi. AddrClientFactory@104e432]"
[2010-09-17 11:12:34.173] ALL 000000000000 GLOBAL_SCOPE Exception in thread "RMI RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi. AddrClientFactory@17d97e5]"
[2010-09-17 11:12:34.188] ALL 000000000000 GLOBAL_SCOPE Exception in thread "RMI RenewClean-[aaa.bbb.ccc.ddd:xxxx,com.sterlingcommerce.woodstock.util.frame.jndi. AddrClientFactory@6d3947]"
[2010-09-17 11:12:34.188] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:34.204] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:34.204] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:35.672] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:35.687] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:35.687] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.091] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.091] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.122] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.122] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.138] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.138] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.153] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.169] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.169] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.184] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.184] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.2] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:12:39.2] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:12:39.216] ALL 000000000000 GLOBAL_SCOPE

[2010-09-17 11:13:10.768] ALL 000000000000 GLOBAL_SCOPE MIC value MIC Value Length: 20
00000001 6b 68 6a ce 67 ad 13 db 19 53 fb f6 19 6b 75 54 khj?g?.?.S??.kuT
00000011 4b 76 06 ff Kv.?


[2010-09-17 11:13:10.768] ALL 000000000000 GLOBAL_SCOPE Attempting to use signature provider Certicom
[2010-09-17 11:13:10.768] ALL 000000000000 GLOBAL_SCOPE Calling initSign...
[2010-09-17 11:13:10.768] ALL 000000000000 GLOBAL_SCOPE Adding signature...
[2010-09-17 11:13:13.578] ALL 000000000000 GLOBAL_SCOPE java.lang.OutOfMemoryError: Java heap space
[2010-09-17 11:13:13.578] ALL 000000000000 GLOBAL_SCOPE

Resolving The Problem

The main resource to analyze is verbose garbage collection logs. It was noted that when GIS got OutOfMemory errors, there were several large spikes of memory with FULL GCs running back to back. It is important to note the time that the large spikes of memory occurred to go back to all the BPs running at that time to investigate the problem.

After researching this issue, it was noted that there was a BP that was generating a really large process data. In addition that BP had around 30 or more child BPs and all of those BPs would inherit the process data from the parent. Process data is maintained in memory. Once the customer reworked the BP so that process data would not get so large, then the issue stopped occurring.

[{"Product":{"code":"SS3JSW","label":"IBM Sterling B2B Integrator"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Not Applicable","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}}]

Historical Number

NFX9968

Product Synonym

[<p><b>]Escalation ID[</b><p>];00224437;[<p><b>]Severity[</b><p>];Normal;[<p><b>]Type[</b><p>];NormalFix

Document Information

Modified date:
14 November 2019

UID

swg21553636