Troubleshooting
Problem
If the deployed web applications contain many JAR files in the WEB-INF/lib directory of the application, IBM WebSphere Application Server v8.0 and later application server JVM jobs and processes might consume a high amount of CPU. This is related to the new META-INF/resources Static File Scanning feature added in WebSphere v8.0.
Symptom
- High CPU Utilization by IBM WebSphere Application Server JVM jobs/processes
- JVM WebContainer threads have the following method on their call stack:
com/ibm/ws/webcontainer/util/MetaInfResourcesFileUtils.findInMtaInfResource()
Cause
Extensive searching inside of web fragments (JAR files in a deployed application's WEB-INF/lib directory). Inside of web fragments, the META-INF/resources directory is searched to see whether any static resources exist. When the deployed web applications are bundled with many web fragments, the application server JVM might consume a high amount of CPU due to the high amount of searching the JVM's WebContainer threads perform.
Environment
IBM WebSphere Application Server v8.0 and later
Diagnosing The Problem
Gather and review multiple IBM JVM Java™ Thread Dumps (javacores) during the high CPU condition. The IBM WebSphere JVM WebContainer threads have the "com/ibm/ws/webcontainer/util/MetaInfResourcesFileUtils.findInMetaInfResource" method listed in the thread stack trace for multiple threads.
Use the following "kill -quit <PID>" command as documented to gather IBM JVM Java™ Thread Dumps (javacores) as documented here. On the IBM i OS, you can also use the WRKJVMJOB Option 12 command to generate a Java™ dump for a specific JVM job. The location of the Java™ Dump (javacore) file is listed in the JVM's job log.
Resolving The Problem
You see thread call stacks for the WebContainer threads similar to the ones listed here when you review the Java™ Dump (javacore) files gathered during the high CPU incident:
RESOLUTION:
To resolve the issue, implement the custom Web Container property "com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing=true" to prevent the application server from traversing the web fragments of each installed application.
NOTE: This property prevents static files from being loaded from the META-INF/resources directory defined within the JAR files that are located in the WEB-INF/lib directory of any deployed applications.
To specify the web container custom property:
1. In the IBM WebSphere Integrated Solutions Console, select Servers > Server Types > WebSphere application servers > server_name > Web Container Settings > Web container.
2. Under "Additional Properties", select "Custom Properties".
3. On the Custom Properties page, click "New".
4. On the settings page, enter the value "com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing" in the Name field and the value "true" in the Value field.
5. Click Apply or OK.
6. Click the Save link in the console taskbar to save your configuration changes.
7. Restart the application server for the custom property to take effect.
=======================================================================================
For additional information on the META-INF/resources static file scanning feature, review the following URLs and information provided:
https://www.ibm.com/support/knowledgecenter/SS7JFU_8.5.5/com.ibm.websphere.express.iseries.doc/ae/rweb_jsp_staticfile.html
https://www.ibm.com/support/knowledgecenter/SS7JFU_8.5.5/com.ibm.websphere.express.iseries.doc/ae/rweb_custom_props.html
META-INF/resources directories of fragments under the WEB-INF/Lib directory in the application WAR file
The web container searches pre-fragment document roots before web fragments. A web fragment comprises a JAR file in an application WEB-INF/lib directory. The JAR might include static resources in a META-INF/resources directory that are defined within the JAR file. To prevent the web container from searching META-INF/resources directories, set the com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing web container custom property to true.
The default value for the custom property is false.
com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing = true
New feature: META-INF/resources directories of fragments under the WEB-INF/Lib directory in the application WAR file is a new function in WebSphere Application Server Version 8.0.
In WebSphere 8.0 and later, Servlet specification 3.0 is supported. Here is a statement from this specification regarding the support of searching the META-INF/resources directory of a JAR file inside the web application's WEB-INF/lib directory. Ultimately, this feature is enabled by default in order to fully support the Servlet 3.0 specification in the WAS environment.
---------------
4.6 Resources
The ServletContext interface provides direct access only to the hierarchy of static content documents that are part of the Web application, including HTML, GIF, and JPEG files, via the following methods of the ServletContext interface:
¦ getResource
¦ getResourceAsStream
The getResource and getResourceAsStream methods take a String with a leading “/” as an argument that gives the path of the resource relative to the root of the context or relative to the META-INF/resources directory of a JAR file inside the web application’s WEB-INF/lib directory. These methods search the root of the web application context for the requested resource before they search any of the JAR files in the WEB-INF/lib directory. The order in which the JAR files in the WEB-INF/lib directory are scanned is undefined. This hierarchy of documents might exist in the server’s file system, in a Web application archive file, on a remote server, or at some other location.
These methods are not used to obtain dynamic content. For example, in a container supporting the JavaServer Pages™ specification1, a method call of the form getResource("/index.jsp") would return the JSP source code and not the processed output. See Chapter 9, “Dispatching Requests” for more information about accessing dynamic content.
The getResourcePaths(String path) method can be used to access the full listing of the resources in the Web application. The full details on the semantics of this method can be found in the API documentation in this specification.
-----------------------
3XMTHREADINFO "WebContainer : 106" J9VMThread:0x0000000032787900, j9thread_t:0x000000018F8D37A0, java/lang/Thread:0x000000004A4D9280, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x514, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0xDD0FB, native priority:0x5, native policy:UNKNOWN)
3XMHEAPALLOC Heap bytes allocated since last GC cycle=111336 (0x1B2E8)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/util/zip/ZipFile.open(Native Method)
4XESTACKTRACE at java/util/zip/ZipFile.<init>(ZipFile.java:150(Compiled Code))
4XESTACKTRACE at java/util/jar/JarFile.<init>(JarFile.java:150(Compiled Code))
4XESTACKTRACE at java/util/jar/JarFile.<init>(JarFile.java:114(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/util/MetaInfResourcesFileUtils.findInMetaInfResource(MetaInfResourcesFileUtils.java:144(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/util/MetaInfResourcesFileUtils.findInMetaInfResource(MetaInfResourcesFileUtils.java:84(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/util/DocumentRootUtils.handleDocumentRoots(DocumentRootUtils.java:240(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/util/DocumentRootUtils.handleDocumentRoots(DocumentRootUtils.java:195(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/util/DocumentRootUtils.handleDocumentRoots(DocumentRootUtils.java:187(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/extension/DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:889(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/extension/DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:874(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/extension/DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:435(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:79(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3751(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:304(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:962(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662(Compiled Code))
4XESTACKTRACE at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:195(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:511(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:305(Compiled Code))
4XESTACKTRACE at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:276(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/WorkQueueManager.requestComplete(WorkQueueManager.java:557(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/WorkQueueManager.attemptIO(WorkQueueManager.java:607(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/WorkQueueManager.workerRun(WorkQueueManager.java:984(Compiled Code))
4XESTACKTRACE at com/ibm/ws/tcp/channel/impl/WorkQueueManager$Worker.run(WorkQueueManager.java:1069(Compiled Code))
4XESTACKTRACE at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1690(Compiled Code))RESOLUTION:
To resolve the issue, implement the custom Web Container property "com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing=true" to prevent the application server from traversing the web fragments of each installed application.
NOTE: This property prevents static files from being loaded from the META-INF/resources directory defined within the JAR files that are located in the WEB-INF/lib directory of any deployed applications.
To specify the web container custom property:
1. In the IBM WebSphere Integrated Solutions Console, select Servers > Server Types > WebSphere application servers > server_name > Web Container Settings > Web container.
2. Under "Additional Properties", select "Custom Properties".
3. On the Custom Properties page, click "New".
4. On the settings page, enter the value "com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing" in the Name field and the value "true" in the Value field.
5. Click Apply or OK.
6. Click the Save link in the console taskbar to save your configuration changes.
7. Restart the application server for the custom property to take effect.
=======================================================================================
For additional information on the META-INF/resources static file scanning feature, review the following URLs and information provided:
https://www.ibm.com/support/knowledgecenter/SS7JFU_8.5.5/com.ibm.websphere.express.iseries.doc/ae/rweb_jsp_staticfile.html
https://www.ibm.com/support/knowledgecenter/SS7JFU_8.5.5/com.ibm.websphere.express.iseries.doc/ae/rweb_custom_props.html
META-INF/resources directories of fragments under the WEB-INF/Lib directory in the application WAR file
The web container searches pre-fragment document roots before web fragments. A web fragment comprises a JAR file in an application WEB-INF/lib directory. The JAR might include static resources in a META-INF/resources directory that are defined within the JAR file. To prevent the web container from searching META-INF/resources directories, set the com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing web container custom property to true.
The default value for the custom property is false.
com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing = true
New feature: META-INF/resources directories of fragments under the WEB-INF/Lib directory in the application WAR file is a new function in WebSphere Application Server Version 8.0.In WebSphere 8.0 and later, Servlet specification 3.0 is supported. Here is a statement from this specification regarding the support of searching the META-INF/resources directory of a JAR file inside the web application's WEB-INF/lib directory. Ultimately, this feature is enabled by default in order to fully support the Servlet 3.0 specification in the WAS environment.
---------------
4.6 Resources
The ServletContext interface provides direct access only to the hierarchy of static content documents that are part of the Web application, including HTML, GIF, and JPEG files, via the following methods of the ServletContext interface:
¦ getResource
¦ getResourceAsStream
The getResource and getResourceAsStream methods take a String with a leading “/” as an argument that gives the path of the resource relative to the root of the context or relative to the META-INF/resources directory of a JAR file inside the web application’s WEB-INF/lib directory. These methods search the root of the web application context for the requested resource before they search any of the JAR files in the WEB-INF/lib directory. The order in which the JAR files in the WEB-INF/lib directory are scanned is undefined. This hierarchy of documents might exist in the server’s file system, in a Web application archive file, on a remote server, or at some other location.
These methods are not used to obtain dynamic content. For example, in a container supporting the JavaServer Pages™ specification1, a method call of the form getResource("/index.jsp") would return the JSP source code and not the processed output. See Chapter 9, “Dispatching Requests” for more information about accessing dynamic content.
The getResourcePaths(String path) method can be used to access the full listing of the resources in the Web application. The full details on the semantics of this method can be found in the API documentation in this specification.
-----------------------
Related Information
[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.1.0"}]
Was this topic helpful?
Document Information
Modified date:
18 December 2019
UID
nas8N1020002