When any 32-bit Lotus Domino task starts a JVM on 64-bit AIX, an extra shared memory segment is created, which can eventually lead to a server crash.
On 32-bit Lotus Domino 8.5.2 and higher running on 64-bit AIX, whenever a Domino task (HTTP, AMGR, etc.) loads a Java Virtual Machine (JVM), a new shared memory segment is allocated. The segment appears to be marked for deletion, but the task that started the JVM remains attached.
When Domino memory manager has allocated 8 segments, all of the Domino tasks will be mapped into those 8 shared segments, but the next task to load a JVM will be mapped into a 9th segment that will result in a crash.
The below example is from an AMGR crash:
0xffffffff D-rw------- notes notes notes notes
1 67108864 3735802 3735802 13:39:04 no-entry 13:39:04
The root cause has been narrowed down to an update SR7 made to JVM version 1.6 that is installed with Domino 8.5.2. The specific change that resulted in this issue is, on AIX systems, the Java heap is now allocated with 64K pages by default, instead of 4K pages. When the 64K allocation is used and the AIX system is configured to support "Large Page size", the JVM will use shmat() to allocate the heap from shared memory segments.
If the AIX system is not configured to support "Large Page size", the JVM will revert back to using 4k pages and shmat() is not used.
This issue has been reported to Quality Engineering as SPR# JWAE8NAPW7 .
As a workaround:
To prevent the issue, set the following NOTES.INI parameter which will revert all the Domino JVMs back to using 4K page: