Lotus Domino support for the Windows /3GB switch
On the Windows® 32-bit platform, heavily loaded Domino® servers, or servers with complex application designs may experience crashes due to high memory usage. More specifically, a specific process may reach the limit of its user address space. By default on Windows 32-bit, the users' address space is 2 gigabyte (GB) in size. This limit can be quickly reached on heavily loaded Domino servers.
The 4 GB address space limit is inherent to any 32-bit application. Installing more RAM, CPU or disk will not overcome this limit. The portion of the 4 GB address space given to the user mode (referred to as the user address space) is platform dependent. On W32, by default, the user address space is 2 GB, with the kernel reserving a separate 2 GB, resulting in a total of 4 GB of address space.
With ever increasing memory demands by complex Enterprise applications such as Domino, the 2 GB user address space can quickly become saturated, resulting in errors or crashes. This can become especially pronounced in cases where multiple enterprise components operate under one process, such as Domino, Oracle, and Java™. In these cases, it is possible to increase the user's address space size to 3 GB using the /3GB boot.ini switch for certain releases of Windows.
Does IBM® Lotus® recommends this course of action?
Before we provide the IBM Lotus recommendation for using the /3GB switch, it is important to provide the proper context for this discussion.
|What does the /3GB switch do?|
On certain W32 platforms, it is possible to change the default percentages of the virtual address space by using the boot.ini parameter /3GB. Using the /3GB switch changes the split of the total virtual address space so that the user address space is increased to 3 GB, and the kernel address space is decreased to 1 GB. This setting in no way changes the overall virtual address space size (the process still only had 4 GB total), and it does not have an impact on how much virtual memory is managed by the system. The /3GB switch simply changes the split between user and kernel space for each process.
Below is a table that provides a list of W32 releases with which the /3GB switch is compatible. This information was taken from the Microsoft Article “Memory Support and Windows Operating Systems”:
|W32 Versions that Support /3GB|
|Windows NT 4.0 Enterprise Edition|
|Windows 2000 Advanced Server|
|Windows 2000 Datacenter Server|
|Windows 2003 Server|
|Windows 2003 Enterprise Server|
|Windows 2003 Datacenter Server|
|Windows XP Professional|
There are various implications when using this switch that must be considered, both application level (Domino) and kernel level (Operating System) factors:
Application (Domino) Level - The /3GB switch has its uses; however, even with this kernel-level setting enabled, a 32-bit process will not be able to use more than 2 GB user space unless it is compiled with the /LARGEADDRESSAWARE switch. This parameter must be specified when compiling the 32-bit executable.
If the kernel is configured with the /3GB setting, but an exe is not compiled with the Large Address Aware option, then the process’ user address space will still be restricted to 2 GB. Contrary to logic, the additional 1 GB is not given back to the kernel, but is instead flagged as a reserved block within the user address space. The result is that 1 GB of the virtual address space is lost. Hence, the /3GB switch should only be used if the process in question is compiled with /LARGEADDRESSAWARE. Misuse of the /3GB setting can cause more problems than it solves.
Beginning in Domino 7.0.1, the Domino server is compiled to be Large Address Aware, so that it can make use of the additional address space up to the full 3 GB. From an application-level perspective, this is all that is required to use the additional 1 GB of address space.
Kernel Level - The application level is not the only consideration. There is a far more important set of implications regarding kernel level performance. When the /3GB switch is used on W32, in actuality, one GB of the kernel's address space is taken, and given back to the user address space. This means the kernel's address space is cut in half.
While certain complex applications may need the extra room in their process address space, the drawback to using this switch is that the kernel must compress its memory usage into the remaining one GB. This will affect all processes system-wide, not only those that use the full 3 GB of user space, since it is the system that is booted into /3GB mode, and the kernel address space is shared across all processes. This can and will have substantial performance implications, particularly for certain kernel components such as the paged-pool, non-paged pool, and the System Page Table Entries (PTE).
Specifically, the use of /3GB will reduce the paged pool maximum size for Windows 2003 from 470 MB to below 200 MB. As default 80% page pool policy, the average paged pool would be around 160 MB, which is far below what is normally expected or desired, as Domino consumes a single 1 KB paged pool for each 1 MB database opening. In the worst scenario, Windows OS might log Event ID - 2020, or Domino might intermittently report the error "Hardware/OS error (Insufficient system resources exist to complete the requested service)."
In addition, when enabling the /3GB switch on Windows 32-bit, Lotus Domino performance testing indicates between a 10-15% increase in CPU usage (due to additional overhead from the constrained kernel). Domino testing indicates that this feature scales better under Windows 2003 than it does under Windows 2000.
Microsoft generally recommends that you keep the range of the /userva switch to between 2900 MB and 3030 MB. However, Domino performance testing indicates that a value of /userva=2800 works well. Again, this is only an option for Domino 7 and higher. For more information regarding the use of the /userva switch, see the following Microsoft article: http://support.microsoft.com/kb/316739
|RAM vs. Address Space Size|
The fact that a process' user address space is increased to 3 GB does not mean that it will use 3 GB of RAM. It simply means that the process can allocate and access up to 3 GB of virtual memory. In fact, under the hood, the kernel makes the decision as to what data is paged out to disk, and what stays in RAM. The kernel is very conservative about what stays in RAM. So, in reality, very few pages in a process' user address space reside in RAM at any one time.
Another way to put it: the process cannot demand that it be given a certain amount of RAM. It is ultimately up to the kernel as to what stays in RAM and what gets paged out. In most instances, a process is not technically aware of which parts of its data are paged in and out of the page file. In short, the application knows no difference between RAM and page file. Therefore, we cannot directly discuss how much RAM a specific application requires, but in theory, only how much virtual memory it may use.
You may be thinking: "OK, if I use the /3GB switch, and I have 8 GB of RAM, that should be 3 GB of RAM for Domino Partition #1, 3 GB of RAM for Domino Partition #2, leaving 2 GB of RAM for the kernel, right?"
It is important to refrain from drawing a one to one ratio between RAM and address space. While these concepts are related, they are not synonymous. Again, the majority of the user address space for a single process usually points to pages on disk at any given moment, with only a small portion of its address space pointing to RAM. The amount of RAM usage by a single process is not calculable in advance; it depends on process and system activity, and is ultimately decided by the kernel at runtime. The only way to get a rough figure of RAM usage within a particular environment is through a simulation of user load (as realistic as possible).
You might also be thinking: "OK, if I have less than 2 GB or RAM, how much RAM does the kernel get vs Domino? Does Domino get 3/4, and the kernel get 1/4, assuming I am using the /3GB switch?"
Again, you cannot draw a direct relationship between RAM and user address space. Even if you have less than 2 GB of RAM, the address space for the user is unchanged at 3 GB (assuming you are using the /3GB switch). The decreased amount of RAM just means that more if its address space points to the page file at any given moment, as controlled by the kernel.
The main point to take away from the above paragraphs is that the /3GB switch is used to adjust a process' address space, not its RAM usage, since low memory conditions reflect a limitation in address space rather than RAM usage.
|The /PAE Switch|
The use of the /PAE setting does NOT affect the size of the virtual address space for each 32-bit application, but only affects the total amount of RAM that the kernel/CPU can manage system-wide.
• /3GB affects the virtual user address space of each process
• /PAE affects the amount of RAM that can be managed by the OS
The /PAE switch affects the amount of RAM that the CPU can address. The use of page file is not affected by this switch, since the page file is stored sequentially on disk. A system may commit far more than 4 GB of virtual memory (across all processes) without the /PAE switch enabled, where most of this memory usage is in the form of the page file, and where only a portion of RAM is in use. By its nature, the kernel can manage the full amount of RAM, and can allow the total virtual memory usage across all 32-bit applications to exceed far greater than 4 GB. In other words, you don’t need to have the /PAE switch enabled to commit more than 4 GB of total virtual memory (the sum of RAM and page file), but only if you want to make use of more than 4 GB of RAM across all processes.
Below is a table that provides RAM limits (using /PAE) for various versions of Windows Operating Systems. This information was taken from MSDN articles Q283037 and Q292934:
|Windows 32-bit Version||Maximum RAM Support with /PAE|
|Windows XP Professional||4 GB|
|Windows 2000 Server||4 GB|
|Windows 2003 Server||4 GB|
|Windows 2000 Advanced Server||8 GB|
|Windows 2003 Enterprise Server||32 GB|
|Windows 2000 Datacenter Server||64 GB|
|Windows 2003 Datacenter Server||64 GB|
|Windows 2000/2003 Datacenter Server (with /3GB)||16 GB|
|/3GB vs. /PAE|
…A program that requests 3 GB of memory is more likely to be
able to have more of its memory remain in physical memory
rather than be paged out, which increases the performance of
programs that are capable of using the /3GB switch. The
exception is when the /3GB switch is used in conjunction with
the /PAE switch, the operating system does not use any memory
in excess of 16 GB. This behavior is caused by kernel virtual
memory space considerations. Thus, if the system restarts with
the /3GB entry in the Boot.ini file, and the system has more than
16 GB of physical memory, the additional physical random
access memory (RAM) is not used by the operating system…
|IBM Lotus Domino recommendations|
In cases where a server is crashing due to the 2 GB limit, there are various options available:
- Use the /3GB Switch: As discussed above, this is the least desirable option, and is generally not recommended. If you use this option, consider using the /userva=2800 syntax. Due to its potentially constrained nature, it it strongly recommended that you test this configuration prior to implementing it in production.
- Memory Tuning: Tune down memory usage, including reducing shared memory size, reducing thread counts, and redesigning applications and agents. In a number of cases, a defect or misconfiguration may be the cause for higher than expected memory usage, and action can be taken to remedy this situation. You should call into IBM Lotus Support for assistance on troubleshooting high memory usage.
- Domino Partitioned Servers: In cases where no Domino defect is found or no further tuning or configuration changes can be made, another option is to spread the user load across multiple Domino partitions. Since each Domino partition has its own set of address spaces, dividing the application load across multiple partitions allows for significantly more virtual space inside which to maneuver (assuming that the Domino server is not originally hardware constrained). The same end can be accomplished through the use of Domino partitions on VMWare.
- Windows 2003 64-bit: Another option is to run your 32-bit Domino Server on the Windows 2003 64-bit Operating System. In doing so, this moves the kernel's address space out of the 32-bit address space and into a separate 64-bit address space. This frees up the Domino 7 user address space to occupy nearly the entire 4 GB (32-bit) address space (since it is compiled as Large Address Aware). Since Domino 6 is not Large Address Aware by default, it still has a 2 GB user address space. See related information for additional caveats on W64.
- Domino Native 64-bit: Once available, the Domino 8 Native 64-bit version will remove the 32-bit limitation on address space altogether, providing a much larger address space. However, this will not remove all Operating System limits or Domino limits, since these limits are enforced to maintain a balanced server. Don't forget, within the native 64-bit environment you may once again encounter your hardware limits (i.e. it will once again become theoretically possible for a single 64-bit process to exhaust system-wide RAM). For details on Domino 8 64-bit availability, see:
FAQ; 64-bit version of Domino (Technote #1296452)
|Related Microsoft Information|
Below is a table which has been taken and modified from Microsoft article #294418. For more information, please refer to the Microsoft article titled "Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003" (see link below)
|32-bit Total Address Space (per process)||
|32-bit User Address Space (per process)||
4 GB (LAA)
2 GB (non-LAA)
3 GB ( LAA with /3GB)
2 GB (default)
|64-bit Total Address Space (per process)||
|64-bit User Address Space (per process)||
|Paging file size||
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
Large memory support is available in Windows Server 2003 and in Windows 2000
How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
Memory Support and Windows Operating Systems
Windows 2000 Datacenter Server Does Not Locate Memory Greater Than 16 GB
Use of the /3GB switch in Exchange Server 2003 on a Windows Server 2003-based system
Server is unable to allocate memory from the system paged pool
More support for:
Software version: 6.0, 6.5, 7.0, 8.0, 8.5
Operating system(s): Windows
Software edition: All Editions
Reference #: 1257005
Modified date: 09 June 2010