IBM Support

IBM Java for AIX HowTo: Exempting Java Applications from SED

Question & Answer


Question

IBM Java for AIX HowTo: Exempting Java Applications from SED

Answer

This document provides step-by-step instructions for disabling Buffer Overflow Protection (BOP) using Stack/heap Execution Disable (SED) on AIX Java.
Overview

What is Stack Execution Disable Protection (SED) ?
AIX implements buffer overflow protection using Stack/heap Execution Disable (SED) to prevent exploits of buffer overflows. "Buffer overflow attacks occur when an internal program buffer is overwritten because data was not properly validated (such as command line, environmental variable, disk or terminal I/O). Attack code is inserted into a running process through the buffer overflow, changing the execution path of the running process. The return address is overwritten and redirected to the inserted-code location." SED prevents buffer overflow attacks by not executing code in data areas of memory.

Just-In-Time (JIT) compilers and SED
AIX Java has Just-In-Time (JIT) compilers that dynamically generate and run native object code while executing java applications. To function correctly, Java applications must be exempt from SED's buffer overflow protection.

Exempting applications from SED
SED sets flags in the header of executable files to control the level of stack execution. By default, all Java launchers have the appropriate bit set to indicate this file does stack/heap based execution. You can verify this with:

sedmgr -d ExecutableName

For example:
sedmgr -d /usr/java7_64/bin/java
/usr/java7_64/bin/java : exempt

Applications that use their own Java launchers and create JVM instances using JNI may not be exempt.

For example,
sedmgr -d ./SampleProgram
./SampleProgram: system

These applications must be explicitly patched to exempt them from SED.

To turn off the SED request bit:
sedmgr -c exempt ExecutableName

Configuring sedmgr for the entire system
The stack execution disable (SED) mechanism in AIX is implemented through system wide mode flags, as well as individual executable file-based header flags. You can choose to turn off SED system wide.

Use the command: sedmgr -m value to change the system wide settings where value is:
off - The SED mechanism is turned off and no process is marked for SED protection.
select - Only a select set of files are enabled and monitored for SED protection.
all - All executable programs loaded on the system are SED protected except for the files requesting an exemption from SED mode.

To turn off SED use:

sedmgr -m off

and reboot the system

Section 5

Section 6

Section 7

Section 8

Section 9

Section 10

Section 11

Section 12

Section 13

Section 14

Section 15

Section 16

Section 17

Section 18

Section 19

Section 20


Contact IBM Support


If, after reading and following the above instructions, further assistance is required, please complete the following steps:

1. Confirm that you have review and completed all of the above steps.

2. Contact IBM and open a new IBM service request (i.e., a new IBM PMR).

3. Collect and upload data as per the data collection procedures noted in the above sections or package and upload the current data and details by following the instructions on this web page:

IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM

Document Type: Technical Document
Content Type: General
Hardware: all Power
Operating System: all AIX Versions
IBM Java: all Java Versions
Author(s): John Carver
Reviewer(s): John Carver

[{"Product":{"code":"SG9NGS","label":"IBM Java"},"Business Unit":{"code":null,"label":null},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
17 June 2018

UID

isg3T1025240

Page Feedback