IBM Support

PM71282: ALLOW "THROTTLING" HUNG ALARM THREAD MESSAGES (UTLS0008), AFTER THE FIRST N COMPLETE STACK TRACES HAVE BEEN SHOWN.

Fixes are available

8.5.0.1: WebSphere Application Server V8.5 Fix Pack 1
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.5.0.2: WebSphere Application Server V8.5 Fix Pack 2
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as new function.

Error description

  • Alarms should be brief actions, with any long-running work
    taking place in another thread; hence,  alarm handlers which
    take unreasonably long will trigger "possibly hung thread"
    Trace reports (message UTLS0008) that include a stack trace of
    the alarm thread's current activity.
    
    Problem: During development, especially when a debugger is in
    use, these verbose reports may swamp other content in the trace
    file and make it unnecessarily hard to read.
    
    Solution: This APAR adds a system property,
    "com.ibm.websphere.alarmthreadmonitor.hung_alarm_throttle", to
    control this. When set to an integer greater than 0, each
    "hung" alarm handler will only report its full stack trace
    that many times. Hung-thread messages after that will show
    only the alarm handler's entry point (typically the .alarm()
    call), surrounded by "[...]" marks to indicate that
    information has been suppressed. Note that this threshhold is
    applied per handler class/method, not per total number of
    messages or per handler instance. The message text has been
    left unchanged.
    
    If the property is not set, or is set <=0, throttling is never
    applied and all hung-thread messages will output their full
    stack traces, as in previous versions.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server                                      *
    ****************************************************************
    * PROBLEM DESCRIPTION: Verbose warnings of long-running        *
    *                      alarm handlers (UTLS0008W) might make   *
    *                      system logs hard to read.               *
    ****************************************************************
    * RECOMMENDATION:  Apply the fix for this APAR and configure   *
    *                  the new                                     *
    *                  -                                           *
    *                  Dcom.ibm.websphere.alarmthreadmonitor.hung_ *
    *                  alarm_mute parameter as appropriate.        *
    ****************************************************************
    Since versions 7.0.0.25 (PM64400), 8.0, and 8.5, the
    application server monitors the activity of threads on which
    system alarms run. When a system alarm thread has been active
    longer than the time defined by the alarm thread monitor
    threshold, the application server logs the following warning
    in the system log. This message indicates the name of the
    thread that is not responding, the length of time that the
    thread has already been active, and the exception stack of the
    thread, which identifies the system component.
    (For more information, see the Information Center's description
    of "Configuring the hang detection policy", for example at
    http://pic.dhe.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.web
    sphere.nd.multiplatform.doc/info/ae/ae/ttrb_confighangdet.htm)
    An actual example of this logged warning:
    [5/9/12 7:50:32:767 EDT] 00000019 AlarmThreadMo W   UTLS0008W:
    The return of alarm thread "Non-deferrable Alarm : 0"
    (00000036) to the alarm thread pool has been delayed for 11625
    milliseconds. This may be preventing normal alarm function
    within the application server. The alarm listener stack trace
    is as follows:
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.exists(File.java:744)
    at
    sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1
    231)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:289)
    at
    java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:1036
    )
    at
    java.security.AccessController.doPrivileged(AccessController.jav
    a:314)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:429)
    at
    sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:281)
    at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:665)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:644)
    at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:657)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:644)
    at
    sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:358)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:627)
    at java.net.URL.getURLStreamHandler(URL.java:1156)
    at java.net.URL.<init>(URL.java:584)
    at java.net.URL.<init>(URL.java:476)
    at java.net.URL.<init>(URL.java:425)
    at java.net.URI.toURL(URI.java:1092)
    at
    org.apache.axis2.jaxws.description.impl.URIResolverImpl.getInput
    StreamForURI(URIResolverImpl.java:243)
    at
    org.apache.axis2.jaxws.util.CatalogURIResolver.getInputSourceFro
    mRedirectedURI(CatalogURIResolver.java:150)
    at
    org.apache.axis2.jaxws.util.CatalogURIResolver.resolveEntity(Cat
    alogURIResolver.java:127)
    at
    org.apache.ws.commons.schema.SchemaBuilder.resolveXmlSchema(Sche
    maBuilder.java:1863)
    at
    org.apache.ws.commons.schema.SchemaBuilder.resolveXmlSchema(Sche
    maBuilder.java:1921)
    at
    org.apache.ws.commons.schema.SchemaBuilder.handleImport(SchemaBu
    ilder.java:1610)
    at
    org.apache.ws.commons.schema.SchemaBuilder.handleXmlSchemaElemen
    t(SchemaBuilder.java:224)
    at
    org.apache.ws.commons.schema.XmlSchemaCollection.read(XmlSchemaC
    ollection.java:481)
    at
    org.apache.axis2.description.WSDLToAxisServiceBuilder$1.run(WSDL
    ToAxisServiceBuilder.java:148)
    at
    org.apache.axis2.description.WSDLToAxisServiceBuilder$1.run(WSDL
    ToAxisServiceBuilder.java:146)
    at
    java.security.AccessController.doPrivileged(AccessController.jav
    a:229)
    at
    org.apache.axis2.java.security.AccessController.doPrivileged(Acc
    essController.java:78)
    at
    org.apache.axis2.description.WSDLToAxisServiceBuilder.getXMLSche
    ma(WSDLToAxisServiceBuilder.java:145)
    at
    org.apache.axis2.description.WSDL11ToAxisServiceBuilder.copyExte
    nsibleElements(WSDL11ToAxisServiceBuilder.java:2473)
    at
    org.apache.axis2.description.WSDL11ToAxisServiceBuilder.processT
    ypes(WSDL11ToAxisServiceBuilder.java:420)
    at
    org.apache.axis2.description.WSDL11ToAxisServiceBuilder.processT
    ypes(WSDL11ToAxisServiceBuilder.java:437)
    at
    org.apache.axis2.description.WSDL11ToAxisServiceBuilder.processT
    ypes(WSDL11ToAxisServiceBuilder.java:437)
    at
    org.apache.axis2.description.WSDL11ToAxisServiceBuilder.processT
    ypes(WSDL11ToAxisServiceBuilder.java:409)
    at
    org.apache.axis2.description.WSDL11ToAxisServiceBuilder.populate
    Service(WSDL11ToAxisServiceBuilder.java:349)
    at
    org.apache.axis2.jaxws.description.impl.EndpointDescriptionImpl.
    buildAxisServiceFromWSDL(EndpointDescriptionImpl.java:977)
    at
    org.apache.axis2.jaxws.description.impl.EndpointDescriptionImpl.
    setupAxisServiceFromDBL(EndpointDescriptionImpl.java:873)
    at
    org.apache.axis2.jaxws.description.impl.EndpointDescriptionImpl.
    <init>(EndpointDescriptionImpl.java:405)
    at
    org.apache.axis2.jaxws.description.impl.ServiceDescriptionImpl.<
    init>(ServiceDescriptionImpl.java:401)
    at
    org.apache.axis2.jaxws.description.impl.ServiceDescriptionImpl.<
    init>(ServiceDescriptionImpl.java:297)
    at
    org.apache.axis2.jaxws.description.impl.DescriptionFactoryImpl.c
    reateServiceDescriptionFromDBCMap(DescriptionFactoryImpl.java:27
    3)
    at
    org.apache.axis2.jaxws.description.DescriptionFactory.createServ
    iceDescriptionFromDBCMap(DescriptionFactory.java:524)
    at
    com.ibm.ws.websvcs.desc.WSModuleDescriptorImpl.buildJAXWSService
    s(WSModuleDescriptorImpl.java:1307)
    at
    com.ibm.ws.websvcs.desc.WSModuleDescriptorImpl._getJAXWSServices
    (WSModuleDescriptorImpl.java:695)
    at
    com.ibm.ws.websvcs.desc.WSModuleDescriptorImpl.getJAXWSServices(
    WSModuleDescriptorImpl.java:679)
    at
    com.ibm.ws.websvcs.deploy.URLPatternHelper.getJAXWSWebServicesUr
    lPatterns(URLPatternHelper.java:103)
    at
    com.ibm.ws.websvcs.deploy.URLPatternHelper.getWebServicesUrlPatt
    erns(URLPatternHelper.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
    Impl.java:60)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at
    com.ibm.ws.odc.cell.TreeBuilder.getWebServicesUrlPatterns(TreeBu
    ilder.java:2794)
    at
    com.ibm.ws.odc.cell.TreeBuilder.processWebModule(TreeBuilder.jav
    a:2377)
    at
    com.ibm.ws.odc.cell.TreeBuilder.readApplication(TreeBuilder.java
    :2006)
    at
    com.ibm.ws.odc.cell.TreeBuilder.readApplication(TreeBuilder.java
    :1857)
    at
    com.ibm.ws.odc.cell.TreeBuilder.doConfigChangePostProcessing(Tre
    eBuilder.java:3245)
    at
    com.ibm.ws.odc.cell.TreeBuilder.handleConfigChange(TreeBuilder.j
    ava:3095)
    at
    com.ibm.ws.odc.cell.TreeBuilder.configChanged(TreeBuilder.java:3
    039)
    at
    com.ibm.ws.odc.cell.TreeBuilder$LocalEventListener.configChanged
    (TreeBuilder.java:5154)
    at
    com.ibm.example.LongRunningAlarmHandler$LongRunningAlarmEventDis
    patcher.run(LongRunningAlarmHandler.java:365)
    at
    com.ibm.example.LongRunningAlarmHandler.alarm(LongRunningAlarmHa
    ndler.java:167)
    at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1783).
    The stack dumps are very useful for finding and resolving the
    causes of these delays. But when a delay is frequent or
    persistant, the size of these dumps may make other information
    in the log files hard to find and use.
    To address this, we have introduced another monitor tuning
    property:
    Name: -Dcom.ibm.websphere.alarmthreadmonitor.hung_alarm_mute
    Value: The maximum number of times any single alarm will
    report its full call stack. After that, only the line
    representing the alarm handler itself will be reported.
    The value 0 means to report all stack traces in full, as was
    done before this APAR.
    Default: 0 (do not mute)
    For example, when muted the above warning would be shortened
    to:
    [5/9/12 7:50:32:767 EDT] 00000019 AlarmThreadMo W   UTLS0008W:
    The return of alarm thread "Non-deferrable Alarm : 0"
    (00000036) to the alarm thread pool has been delayed for 11625
    milliseconds. This may be preventing normal alarm function
    within the application server. The alarm listener stack trace
    is as follows:
    [...] at
    com.ibm.example.LongRunningAlarmHandler.alarm(LongRunningAlarmHa
    ndler.java:167) [...]
    To add this argument to the server settings, complete the
    following actions:
    1) Under Server Infrastructure on the server settings page in
    the administrative console, click Java and process
    management > Process definition .
    2) [z/OS only] Select Control .
    3) Select Java virtual machine .
    4) Add the argument to the JVM generic arguments section, eg
    -Dcom.ibm.websphere.alarmthreadmonitor.hung_alarm_mute=3
    

Problem conclusion

  • Apply the fix for this APAR and configure
    -Dcom.ibm.websphere.alarmthreadmonitor.hung_alarm_mute as
    appropriate.
    
    The fix for this APAR is currently targeted for inclusion in
    fix packs 7.0.0.27, 8.0.0.5 and 8.5.0.1.  Please refer to the
    Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM71282

  • Reported component name

    WEBSPHERE APP S

  • Reported component ID

    5724J0800

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-08-21

  • Closed date

    2012-09-05

  • Last modified date

    2012-09-05

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE APP S

  • Fixed component ID

    5724J0800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP

  • R850 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 850

Reference #: PM71282

Modified date: 05 September 2012