Flashes (Alerts)
Abstract
If you are running IBM Rational Lifecycle Management (CLM) 4.0.4, 4.0.5, 4.0.6, 4.0.7 (without iFix), 5.0, 5.0.1, the Jazz Team Server (JTS) could be in hang stage due to a deadlock caused by Java Naming and Directory Interface (JNDI) references not being removed from weaved packages.
Content
CLM 4.0.4, 4.0.5, 4.0.6, 4.0.7 (without iFix), 5.0 and 5.0.1 can cause a hang in the Jazz Team Server on startup due to a deadlock in the ClassLoader. Depending on server configuration, the hang may not result in a user perceived impact for several hours. This problem is resolved in CLM 4.0.7 iFix (any of the 4.0.7 iFixes) and in CLM 5.0.2.
Symptoms
Some of the hang symptoms include the following:
- After the Jazz Team server is up, navigate to the active users page, try to open any user's profile. You will notice that the overview page never gets populated. The Client Access Licenses section in the page remains at loading...
- Users are unable to get licenses after login to CLM. They receive errors similar to the following:
WARN com.ibm.team.repository - CRJAZ1297I The list of floating licenses could not be updated.
- If the Jazz Team server is deployed on IBM WebSphere Application Server (WAS), there may be reports of thread hangs that are in the ClassLoader.
Defect
This issue was reported as a defect. Refer the following Jazz.net defect work item for further details:
https://jazz.net/jazz/web/projects/Jazz%20Foundation#action=com.ibm.team.workitem.viewWorkItem&id=331148
Workaround:
Restart the Application server hosting the Jazz Team Server.
Diagnosing
You can follow the steps below to verify that you are facing this defect:
Note: If the results of the following checks are both true, then you could most likely be facing this defect.
- Run diagnostics from jts/admin page, check to see if you are getting a warning similar to the following under the JVM section:
CRJAZ1459W 4 deadlocked threads found. See detailed results file for more details.
- Collect a set of javacore during the server's hang time, examine each collected javacore to see if you will find deadlock code similar to the following:
at java/lang/ClassLoader.loadClass(ClassLoader.java:665(Compiled Code))
at org/eclipse/osgi/internal/loader/SystemBundleLoader.findLocalClass(SystemBundleLoader.java:117(Compiled Code))
at org/eclipse/osgi/internal/loader/SingleSourcePackage.loadClass(SingleSourcePackage.java:33(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClassInternal(BundleLoader.java:466(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClass(BundleLoader.java:422(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClass(BundleLoader.java:410(Compiled Code))
at org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.loadClass(DefaultClassLoader.java:107(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:665(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.loadClass(BundleLoader.java:338(Compiled Code))
at org/eclipse/osgi/framework/internal/core/BundleHost.loadClass(BundleHost.java:232(Compiled Code))
at org/eclipse/osgi/framework/internal/core/AbstractBundle.loadClass(AbstractBundle.java:1197(Compiled Code))
at org/eclipse/equinox/weaving/hooks/WeavingLoaderDelegateHook.postFindClass(Bytecode PC:104(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.searchHooks(BundleLoader.java:520(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClassInternal(BundleLoader.java:486(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClass(BundleLoader.java:422(Compiled Code))
at org/eclipse/osgi/internal/loader/BundleLoader.findClass(BundleLoader.java:410(Compiled Code))
at org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.loadClass(DefaultClassLoader.java:107(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:665(Compiled Code))
at com/ibm/team/repository/jndi/internal/LDAPDirContextImpl. (LDAPDirContextImpl.java:63)
at com/ibm/team/repository/jndi/internal/JNDIServiceImpl.getLDAPDirContext(JNDIServiceImpl.java:44)
at com/ibm/team/repository/jndi/internal/JNDIServiceImpl.getLDAPDirContext(JNDIServiceImpl.java:1)
at com/ibm/team/repository/service/jts/internal/userregistry/ldap/LDAPUserRegistry.getLDAPDirContext(LDAPUserRegistry.java:227)
at com/ibm/team/repository/service/jts/internal/userregistry/ldap/LDAPUserRegistry.createDirContext(LDAPUserRegistry.java:204)
at com/ibm/team/repository/service/jts/internal/userregistry/ldap/LDAPUserRegistry.initializeContext(LDAPUserRegistry.java:189)
at com/ibm/team/repository/service/jts/internal/userregistry/ldap/LDAPUserRegistry. (LDAPUserRegistry.java:160)
at com/ibm/team/repository/service/jts/internal/userregistry/ldap/LDAPUserRegistryProvider.createRegistry(LDAPUserRegistryProvider.java:32)
at com/ibm/team/repository/service/userregistry/provider/AbstractUserRegistryProvider.getRegistry(AbstractUserRegistryProvider.java:31)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56(
Compiled Code))
at com/sun/proxy/$Proxy331.getRegistry(Bytecode PC:11)
at com/ibm/team/repository/service/jts/internal/userregistry/ExternalUserRegistryInternalService.internalGetUserRegistry(
ExternalUserRegistryInternalService.java:161)
at com/ibm/team/repository/service/jts/internal/userregistry/ExternalUserRegistryInternalService.isExternalRegistryWriteable(
ExternalUserRegistryInternalService.java:361)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56(
Compiled Code))
at com/sun/proxy/$Proxy332.isExternalRegistryWriteable(Bytecode PC:11)
at com/ibm/team/repository/service/jts/internal/userregistry/JtsExternalUserRegistryService.isExternalRegistryWriteable(
JtsExternalUserRegistryService.java:136)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56(
Compiled Code))
at com/sun/proxy/$Proxy333.isExternalRegistryWriteable(Bytecode PC:11)
at com/ibm/team/repository/service/jts/internal/userregistry/ExternalUserRegistryServiceDelegator.isExternalRegistryWriteable(
ExternalUserRegistryServiceDelegator.java:182)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56(
Compiled Code))
at com/sun/proxy/$Proxy543.isExternalRegistryWriteable(Bytecode PC:11)
at com/ibm/team/repository/service/internal/AdminRestService.getUserRegistryInfo(AdminRestService.java:704)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347(Compiled Code))
at org/eclipse/soda/sat/core/internal/record/ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56(
Compiled Code))
at com/sun/proxy/$Proxy558.getUserRegistryInfo(Bytecode PC:11)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.doModelledRestService(AbstractTeamServerServlet.java:530)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2359)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2153)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1963)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668(Compiled Code))
at org/eclipse/equinox/http/registry/internal/ServletManager$ServletWrapper.service(ServletManager.java:180(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ServletRegistration.service(ServletRegistration.java:61(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.processAlias(ProxyServlet.java:126(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.service(ProxyServlet.java:76)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668)
at org/eclipse/equinox/servletbridge/BridgeServlet.service(BridgeServlet.java:120)
at com/ibm/team/repository/server/servletbridge/JazzServlet.service(JazzServlet.java:74)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1230)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478)
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com/ibm/team/repository/server/servletbridge/BridgeFilter.processDelegate(BridgeFilter.java:165)
at com/ibm/team/repository/server/servletbridge/BridgeFilter.doFilter(BridgeFilter.java:198)
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:960)
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)
at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3837)
at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:304)
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:981)
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com/ibm/ws/ssl/channel/impl/SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code))
at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204)
at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1864)
[{"Product":{"code":"SSUC3U","label":"IBM Engineering Workflow Management"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Repository","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"}],"Version":"4.0.4;4.0.5;4.0.6;4.0.7;5.0;5.0.1","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}}]
Product Synonym
Rational Team Concert
Was this topic helpful?
Document Information
Modified date:
25 September 2022
UID
swg21693283