IBM Support

Poor response or Application timeout on zipfile operation

Flashes (Alerts)


Abstract

During application runtime, the application server may experience poor response and application requests may timeout when under load. The problem manifests on WebSphere V8.0 or WebSphere V8.5 for z/OS.

Content

The Servlet 3.0 specification is supported with WebSphere V8.0 and above. It requires searching for static resources in web fragment JAR files to find the respective resources when serving inbound requests. The web container searches for static files and JSP files in up to four different locations, depending on the application configuration:

1. The application WAR directory
2. WEB-INF directory
3. META-INF/resources directories
4. Extended document roots

If hung thread detection is enabled, or a javacore is taken during the period where there is poor response, the following java stacktrace that involves this search can be seen below:

Java Stacktrace
java/util/zip/ZipFile.open(String, int, long, boolean)
java/util/zip/ZipFile.<init>(java.io.File, int)
java/util/jar/JarFile.<init>(java.io.File, boolean, int)
java/util/jar/JarFile.<init>(java.io.File)
com/ibm/ws/webcontainer/util/MetaInfResourcesFileUtils.findInMetaInfResource(java.io.File[], String)
com/ibm/ws/webcontainer/util/MetaInfResourcesFileUtils.findInMetaInfResource(String)
com/ibm/ws/webcontainer/util/DocumentRootUtils.handleDocumentRoots(String, boolean, boolean, boolean, java.util.Map)
com/ibm/ws/webcontainer/util/DocumentRootUtils.handleDocumentRoots(String, java.util.Map)
com/ibm/ws/webcontainer/util/DocumentRootUtils.getDocumentRealPath(String, java.util.Map)
com/ibm/ws/webcontainer/webapp/WebApp.getRealPath(String)
.....

The top of the java stacktrace may have ZipFile.close instead of ZipFile.open.

If a console dump is taken, and jdmpview is used to format out the java and native stacktrace. The following native stacktrace can be used to further confirm the symptoms.

Native Stacktrace
/WebSphere/AppServer/java64/lib/s390x/compressedrefs/libj9prt26.so::semopWrapper+0x26
/WebSphere/AppServer/java64/lib/s390x/compressedrefs/libj9prt26.so::j9shsem_deprecated_wait+0x134
ExtraSymbolsModule::SH_OSCachesysv::acquireWriteLock(unsigned long)+0x2a6
ExtraSymbolsModule::SH_CompositeCacheImpl::enterWriteMutex(J9VMThread*,bool,const char*)+0x39a
ExtraSymbolsModule::SH_CacheMap::getCachedUTFString(J9VMThread*,const char*,unsigned short)+0x8d0
ExtraSymbolsModule::SH_CacheMap::notifyClasspathEntryStateChange(J9VMThread*,const char*,unsigned long)+0x588
ExtraSymbolsModule::j9shr_hookZipLoadEvent+0x94
ExtraSymbolsModule::J9HookDispatch+0xe4
/WebSphere/AppServer/java64/lib/s390x/compressedrefs/libjvm.so::JVM_ZipHook+0xee
libjvm.so::JVM_ZipHook+0x2c
/WebSphere/AppServerr/java64/lib/s390x/libzip.so::Java_java_util_zip_ZipFile_open+0x13a
<unknown>
/WebSphere/AppServer/java64/lib/s390x/compressedrefs/libj9vm26.so::javaProtectedThreadProc+0x10a
/WebSphere/AppServer/java64/lib/s390x/compressedrefs/libj9prt26.so::j9sig_protect+0x76e
/WebSphere/AppServer/java64/lib/s390x/compressedrefs/libj9vm26.so::javaThreadProc+0x72
libj9thr26.so::thread_wrapper+0x562
ExtraSymbolsModule::CELQPCMM+0xfa8

Cause
WebSphere for z/OS has the JVM share class cache enabled by default, and when under load the search through the jar files in the web application from multiple threads can surface the symptoms of IV59039.

To resolve the issue:
Move to WebSphere z/OS service level that ships the Java Service Release for IV59039 once available.

To avoid the issue choose option 1 or option 2 below:

1. Disable the JVM shareclass cache for the Control and Servant region.

Application servers > servername > Process definition > Control > Java Virtual Machine
Generic JVM arguments
-Xshareclasses:none

Application servers > servername > Process definition > Servant > Java Virtual Machine
Generic JVM arguments
-Xshareclasses:none

This may result is slighty longer startup time, and more memory usage.

2. If the applications in the server are at Servlet 2.5 specification or earlier, or if the applications are at Servlet 3.0 specification and not making use of web fragments, the SkipMetaInfResourcesProcessing can be enabled.

Application servers > servername > Web Container Settings > Web container

Under Properties, click Custom Properties.

Set:
com.ibm.ws.webcontainer.SkipMetaInfResourcesProcessing=true

[{"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Hangs\/Performance Degradation","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"8.5;8.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
25 September 2022

UID

swg21671090