IBM Support

Using PsList to troubleshoot high CPU usage in Windows

Troubleshooting


Problem

While troubleshooting high CPU utilization in IBM® WebSphere® Application Server on Windows operating systems, the perfmon tool may not provide enough information. The PsList tool can be used to collect CPU usage about individual threads in the Java process. PsList is a light-weight tool bundled in the PsTools utilities provided by Microsoft.

Symptom

High CPU utilization in IBM® WebSphere® Application Server on Windows operating systems

Resolving The Problem

NOTE: This document is designed to be used in conjunction with the MustGather: Performance, hang, or high CPU issues on Windows technote.


Setup

Before you restart the server

  1. Download PsTools.zip from the Microsoft Technet.
  2. Unzip the contents of PsTools.zip to a local directory say c:\pstools. This will extract a couple of .exe files, one of which will be PsList.exe.
  3. Copy the attached startpslist.bat file to the same directory as the output of the extracted PsTools.zip. This simple script is provided to automatically run PsList and collect its data. By default, it runs PsList.exe every 5 seconds. This interval can be changed by editing the TIME_SLEEP value in the script.

Collecting the Data
  1. Start the server and wait for high CPU utilization to occur. Note: If the you cannot wait until 100% CPU utilization to occur, then start the script anytime after starting the server. However, the longer the script runs, the larger the pslistlog*.txt file(s) will grow.
  2. At the time when the server Java™ process is consuming high CPU :
    • From the Windows Task manager determine the process ID(s) of the Java processes using high CPU.
    • Open a command prompt for each identified Java process and go to c:\pstools (or wherever you have extracted PsList.exe) on each of them.
    • On each prompt, run the command: startpslist.bat [Java Process ID]
  3. Collecting the javacores and other server logs requested in the MustGather: Performance, hang, or high CPU issues on Windows.
  4. After collecting the MustGather data in step 3, terminate the startpslist.bat by issuing the command: CTRL+BRK or CTRL+C.
  5. Collect all pslistlog_*.txt files created in c:\pstools. These files use the naming convention: pslistlog.<PID>.<Date when started>-<TIME when started>.txt

Determining which thread(s) are using high CPU from PsList and javacores
  1. Determine the time when the CPU was high or close to 100% and find this time period in the PsList output.
  2. High values in the User Time and Kernel Time columns indicate high CPU usage of that particular thread. The time given in the output is from the time thread was started. Therefore, it is recommended to look in consecutive PsList outputs and identify the thread(s) that spiked.

    In the sample output below, the thread in bold is consuming the highest amount of CPU:

    java 3592:
    Tid Pri Cswtch State User Time Kernel Time Elapsed Time
    3724 8 30410 Ready 0:00:31.421 0:00:18.593 0:35:24.451
    1372 9 2764 Wait:UserReq 0:00:00.140 0:00:00.171 0:34:07.716
    3168 15 1043 Wait:UserReq 0:00:00.125 0:00:00.062 0:30:00.355
    512 8 4683 Ready 0:00:06.875 0:00:03.078 0:23:16.102
    2464 15 2069 Wait:UserReq 0:00:00.046 0:00:00.000 0:23:15.727
    3424 15 477 Wait:UserReq 0:00:00.296 0:00:00.156 0:15:38.365
    344 15 131 Wait:UserReq 0:00:00.031 0:00:00.062 0:14:13.552
    2844 15 137 Wait:UserReq 0:00:00.093 0:00:00.015 0:14:13.333
    744 15 137 Wait:UserReq 0:00:00.078 0:00:00.062 0:14:13.317
    1596 15 115 Wait:UserReq 0:00:00.093 0:00:00.031 0:14:13.302
    2924 15 3107 Wait:UserReq 0:00:03.343 0:00:02.046 0:13:42.442
    3000 15 43 Wait:UserReq 0:00:00.000 0:00:00.000 0:12:45.473

  3. Note the thread ID (Tid) of this thread. Convert the decimal thread ID into the hexadecimal number.

    3724     in decimal converts to E8C in hexadecimal

  4. In the javacores taken during this same time, search for the identified thread ID in hex. Below is the search for E8C in the javacore:

    "WebContainer : 2" (TID:0x109A3AD8, sys_thread_t:0x639215F0, state:R, native ID:0xE8C) prio=5
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.exists(File.java(Compiled Code))

    In this example, "WebContainer : 2" is the thread utilizing high CPU. The stack trace can then be used to determine which code is causing the high CPU usage for this Java process.

    Note: The startpslistYYYYMMDD.bat file is for Windows machines that use the YYYYMMDD date format.

startpslist.batstartpslistYYYYMMDD.batstartpslistYYYYMMDD.bat

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"100% CPU Usage","Platform":[{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21304776

Page Feedback