Domino JVMs create extra AIX shared memory segments causing Domino server crash

On 32-bit Lotus Domino 8.5.2 and higher running on 64-bit AIX, whenever a Domino task (HTTP, AMGR, etc.) loads a Java Virtual Machine (JVM), a new shared memory segment is allocated. The segment appears to be marked for deletion, but the task that started the JVM remains attached.
When Domino memory manager has allocated 8 segments, all of the Domino tasks will be mapped into those 8 shared segments, but the next task to load a JVM will be mapped into a 9th segment that will result in a crash.

The below example is from an AMGR crash:

m 18874373 0xffffffff D-rw------- notes notes notes notes 1 67108864 3735802 3735802 13:39:04 no-entry 13:39:04

The root cause has been narrowed down to an update SR7 made to JVM version 1.6 that is installed with Domino 8.5.2. The specific change that resulted in this issue is, on AIX systems, the Java heap is now allocated with 64K pages by default, instead of 4K pages. When the 64K allocation is used and the AIX system is configured to support "Large Page size", the JVM will use shmat() to allocate the heap from shared memory segments.

If the AIX system is not configured to support "Large Page size", the JVM will revert back to using 4k pages and shmat() is not used.

This issue has been reported to Quality Engineering as SPR# JWAE8NAPW7 .

As a workaround:
To prevent the issue, set the following NOTES.INI parameter which will revert all the Domino JVMs back to using 4K page:

JavaDebugString=-Xlp4K

For more information about the change made to JVM version 1.6, see the document: IBM 32-bit SDK for AIX platforms, Java Technology Edition - SDK Guide -> "Change in default page size increases memory usage" section