IBM Support

IBM Java for AIX HowTo: Resolve core dumps by processes running on Power 8

Question & Answer


Question

IBM Java for AIX HowTo: Resolve core dumps by processes running on Power 8

Answer

This document provides step-by-step instructions on how to resolve the issue of Java processes generating core dumps on POWER8 during LPM or when executing tprof in parallel to running applications that use IBM Java for AIX.

If all of these steps have been completed and the problem persists, then it is recommended that you open a new IBM support call with the IBM AIX Java Support team for a more in-depth analysis into the issue. Prior to any additional analysis, the IBM AIX Java Support team will confirm that all of these steps have been completed since this is a well known and documented issue.
The instructions in this document make references to generic terms in Italics that will need to be replaced with information specific to the support call and the environment. It is very important that consistent and accurate values be used in place of the Italicized generic terms when collecting the data to ensure the prompt and correct delivery of the data when uploaded.
Generic Term Replace with
PMR The full IBM PMR number (e.g. , PMR12345.b678.c000).
Overview
Step-by-Step Instructions

Problem:

Applications using IBM Java for AIX using Power 8 hardware with Power 8 mode enabled, may generate process core dumps due to a known issue with Event-Based Branching facility (EBB) and the enhanced Performance Monitoring Unit (PMU).

IBM Java version 8.0 for AIX, uses the Power 8 Runtime Instrumenation (RI) feature facilitated by EBB facility and the PMU and therefore, Java 8 is susceptible to experiencing these application crashes. The most commonly reported scenarios are:

- When attempting to use Live Partition Mobility (LPM) with running Java processes
- When executing tprof (as a command or through a performance data collection script) in parallel to running Java processes

The active Java processes will terminate and generate process core dumps during these common scenarios.

More advanced users with ability to analyze the dumps, can check for references to "MMCR0" register as cause of the issue from the core files.

Recommendations are listed below to resolve this issue.

Workarounds:

Below is a list of workarounds that may resolve the issue as an alternative to applying an ifix:

AIX Workarounds:

1) Run in Power 7 Compatibility Mode as a temporary fix.
2) If migrating using LPM, stop the processes before the migration and restart them after the migration, to avoid processes from crashing.

Recommendations for process core dumps during LPM:

Option 1:

Upgrade IBM Java 8.0 to SR5 or above to prevent the core dumps issue.

To upgrade stand alone IBM Java 8, please follow instructions as per:
http://www-01.ibm.com/support/docview.wss?uid=isg3T1022693

If you need assistance with upgrading WebSphere packaged Java, please open a new IBM support call with the IBM WebSphere Support team.

If the existing IBM Java 8.0 installs cannot be upgraded immediately, please make sure IBM Java 8.0 applications, are restarted after applying one of the options listed below to avoid crashes.

There are no expected side effects or impact to Java applications or agents after the upgrade. This option requires a restart of the process.

Option 2:

For IBM Java 8.0 releases before SR5, the issue can be avoided by adding the following JVM command line option to the Java command of the crashing process(es):

-XX:-RuntimeInstrumentation

No side effects or impact is expected for the Java applications or agents after applying the command line option. This option requires a restart of the process.

Option 3:
The issue can be avoided by running the system in Power 7 Compatibility Mode.
To determine the processor mode, at the command prompt, execute the following command:

# prtconf | grep "Processor Implementation Mode"

CPU Compatibility Mopde can be changed to POWER 7 from the HMC through the LPAR properties, following the steps below:
i. Select the LPAR and deactivate it.
ii. Go to Configuration -> Manage Profiles.
iii. Select the profile that needs to be activated.
iv. Go to Processors, change the Processor compatibility mode to Power 7 and boot it using the same profile.

This option requires a restart of the system.

Option 4:
If migrating using LPM, stop the processes before the migration and restart them after the migration, to avoid processes from crashing.
For example, from a command line prompt, run the following commands to identify the Java applications using Java 7.1 or Java 8:

# ps -ef | grep java

For each command listed, work with the appropriate system's or application's administrator to temporarily shutdown these processes using IBM Java 8.0, before using LPM and restart once LPM has completed.

Workarounds:

Below is a list of workarounds that may resolve the issue as an alternative to applying a fix:

AIX Workarounds:

Recommendations for process core dumps when executing tprof:
When experiencing process core dumps while executing tprof in parallel to IBM Java 8.0 applications, apply the APAR below that is applicable to your current AIX level, to fix the issue:

IV97097: JAVA 1.8 PROCESS COREDUMP DUE TO EBB ILL INST WHEN TPROF IS RUN (AIX 7200-02)
IV96941: JAVA 1.8 PROCESS COREDUMP DUE TO EBB ILL INST WHEN TPROF IS RUN (AIX 7200-01)
IV97096: JAVA 1.8 PROCESS COREDUMP DUE TO EBB ILL INST WHEN TPROF IS RUN (AIX 7200-00)

IV96353: JAVA 1.8 PROCESS COREDUMP DUE TO EBB ILL INST WHEN TPROF IS RUN (AIX 7100-04)
IV97095: JAVA 1.8 PROCESS COREDUMP DUE TO EBB ILL INST WHEN TPROF IS RUN (AIX 7100-05)

Troubleshoot:

If, after following the above instructions, the application continues to generate core dumps, please complete the following steps:

1. Confirm that you have completed all of the above steps.
2. Contact IBM and open a new IBM service request (new PMR).
3. Collect and upload data as per the data collection procedure for automatically generated process core dumps.

Step 7:

ACTION

Step 8:

ACTION

Step 9:

ACTION

Step 10:

ACTION

Step 11:

ACTION

Step 12:

ACTION

Step 13:

ACTION

Step 14:

ACTION

Step 15:

ACTION

Step 16:

ACTION

Step 17:

ACTION

Step 18:

ACTION

Step 19:

ACTION

Step 20:

ACTION

Step 21:

ACTION

Document Type: Instruction
Content Type: Howto
Hardware: Power 8
Operating System: AIX 7.1 and 7.2
IBM Java: Java 8.0
Author(s): Vidya Makineedi
Reviewer(s): Roger Leuckie

[{"Product":{"code":"SG9NGS","label":"IBM Java"},"Business Unit":{"code":null,"label":null},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
17 June 2018

UID

isg3T1024484