Troubleshooting
Problem
This is Part three three part document series: Diagnosing JVM Memory Problems With The Maximo Application
Resolving The Problem
Pinpointing Memory Related Code Problems Using FetchResultLogLimit
Memory leaks can have a number of causes including but not exclusively:
- Data integrity issues: perhaps a hierarchy where the parent and child are the same.
- Custom code: classes extended which may open mbosets but fails to close them
- Core code: a bug in the core application code where a condition may cause mbosets not to close
If a memory leak is identified, any data loaded without MBO processing should be reviewed, any custom code should be examined for proper processing and a PMR should be opened with IBM Support to determine if there are Fix Packs available to resolve the issue.
When the application fails or slows down due to the lack of memory and an increase in JVM garbage collection processes, how can the memory problem be identified?
Maximo provides logging tools to assist with diagnosing memory related performance problems. The most valuable of these tools are the mxe.mbocount and mxe.db.fetchResultLogLimit properties found in the Maximo System Properties application. For more information on how to set these properties, see System Properties to Monitor and Troubleshoot Performance.
When a memory leak is suspected, the fetchResultLogLimit property can be used to determine what processes are loading MBOs and mbosets. The log snippet below was generated by having fetchResultLogLimit set to 200. This generates an exception for each 200 MBOs that get loaded into an mboset. In the case shown here, 12,600 MBOs have been loaded into a single mboset. Since there are multiple mbosets, there are nearly 60,000 MBOs in memory that have not been released by the application causing memory exhaustion.
- 02 Apr 2015 15:29:20:696 [INFO] -------FetchResultLogLimit logging starts------
02 Apr 2015 15:29:20:696 [INFO] User Name : PTEST9
02 Apr 2015 15:29:20:696 [INFO] Query : select * from workorder where location = 'K10-1-L46TC1' and siteid= 'LPO'
02 Apr 2015 15:29:20:697 [INFO] MboSet Name : WORKORDER Reference = psdi.plusc.app.workorder.PlusCWOSet[RemoteStub [ref: [endpoint:[10.65.106.179:60171](local),objID:[245641]]]]
02 Apr 2015 15:29:20:697 [INFO] MboSet Size : 12600
02 Apr 2015 15:29:20:697 [INFO] Fetch count so far : 12600
02 Apr 2015 15:29:20:697 [INFO] Printing StackTrace:
java.lang.Exception
at psdi.mbo.MboSet.fetchMbos(MboSet.java(Compiled Code))
at psdi.mbo.MboSet.getMbo(MboSet.java(Compiled Code))
at psdi.plusc.app.asset.PlusCDueDate.setDateFromWO(Unknown Source)
at psdi.plusc.app.asset.PlusCDueDate.getLocationDueDate(UnknownSource)
at psdi.plusc.app.asset.PlusCDueDate.getWODueDate(Unknown Source)
at psdi.plusc.app.workorder.PlusCFldPlusCNextDate.initValue(UnknownSource)
at psdi.mbo.MboValue.initValue(MboValue.java(Compiled Code))
at psdi.mbo.MboValue.getString(MboValue.java(Compiled Code))
at psdi.mbo.MboValueData.<init>(MboValueData.java(Compiled Code))
at psdi.mbo.MboValue.getMboValueData(MboValue.java(Inlined Compiled Code))
at psdi.mbo.Mbo.getMboValueData(Mbo.java(Compiled Code))
at psdi.mbo.MboData.<init>(MboData.java(Compiled Code))
at psdi.mbo.Mbo.getMboData(Mbo.java(Compiled Code))
at psdi.webclient.system.beans.AppBean.save(AppBean.java:417)
at psdi.plusc.webclient.beans.workorder.PlusCWOAppBean.save(UnknownSource)
at psdi.webclient.system.beans.AppBean.SAVE(AppBean.java:709)
at psdi.webclient.beans.workorder.WOAppBean.SAVE(WOAppBean.java:451)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at psdi.webclient.system.tags.DefaultHandlerTag.doStartTag(DefaultHandlerTag.java(Compiled Code))
at com.ibm._jsp._control._jspx_meth_mro_defaulthandler_0(_control.java(Compiled Code))
at com.ibm._jsp._control._jspService(_control.java(Compiled Code))
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java(Compiled Code))
at javax.servlet.http.HttpServlet.service(HttpServlet.java(CompiledCode))
at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java(Inlined Compiled Code))
at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java(Compiled Code))
at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java(Compiled Code))
at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled Code))
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java(Inlined Compiled Code))
at com.ibm.ws.jsp.webcontainerext.JSPExtensionServletWrapper.handleRequest(JSPExtensionServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.include(WebAppRequestDispatcher.java(Compiled Code))
at psdi.webclient.system.controller.ControlHandler.handleEvent(ControlHandler.java(Compiled Code))
at psdi.webclient.system.controller.RequestHandler.handleEvent(RequestHandler.java(Compiled Code))
at psdi.webclient.system.controller.RequestHandler.handleRequest(RequestHandler.java(Compiled Code))
at psdi.webclient.servlet.WebClientServlet.handleRequest(WebClientServlet.java:81)
at psdi.webclient.servlet.WebClientServlet.doPost(WebClientServlet.java:36)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java(CompiledCode))
at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java(Inlined Compiled Code))
at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java(Compiled Code))
at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java(Compiled Code))
at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled Code))
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled Code))
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled Code))
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
02 Apr 2015 15:29:24:460 [INFO] -------FetchResultLogLimit logging ends------
The logging for fetchResultLogLimit prints as an exception but is not actually an application exception. It prints this way to obtain the stack trace so diagnosis can be done. The first two lines of the exception generated by the fetchResultLogLimit are overhead from the debug logging. Typically, the third line is a likely suspect for the problem code that is loading MBOs without releasing them. In the case shown here, PlusCDueDate is the likely code responsible for the memory problem.
Once a memory leak has been identified, contact support to help determine if the leak is code, data, or customization related. In some cases there may be an Interim Fix available. In others, a services engagement may be required to help determine the cause.
 | |||
| Exposing A Memory Problem Using The Maximo Debug Property MBOCount |
Related Information
[{"Product":{"code":"SSLKT6","label":"IBM Maximo Asset Management"},"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Component":"System Related","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}},{"Product":{"code":"SSLKTY","label":"Maximo Asset Management for IT"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}},{"Product":{"code":"SSWK4A","label":"Maximo Asset Management Essentials"},"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}},{"Product":{"code":"SSKTXT","label":"Tivoli Change and Configuration Management Database"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}},{"Product":{"code":"SS6HJK","label":"Tivoli Service Request Manager"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}},{"Product":{"code":"SSWT9A","label":"IBM Control Desk"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}}]
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21326804