IBM Support

Lotus Domino support for the Windows /3GB switch

Technote (FAQ)


Question

On the Windows® 32-bit platform, heavily loaded Domino® servers, or servers with complex application designs may experience crashes due to high memory usage. More specifically, a specific process may reach the limit of its user address space. By default on Windows 32-bit, the users' address space is 2 gigabyte (GB) in size. This limit can be quickly reached on heavily loaded Domino servers.

Cause

The 4 GB address space limit is inherent to any 32-bit application. Installing more RAM, CPU or disk will not overcome this limit. The portion of the 4 GB address space given to the user mode (referred to as the user address space) is platform dependent. On W32, by default, the user address space is 2 GB, with the kernel reserving a separate 2 GB, resulting in a total of 4 GB of address space.

With ever increasing memory demands by complex Enterprise applications such as Domino, the 2 GB user address space can quickly become saturated, resulting in errors or crashes. This can become especially pronounced in cases where multiple enterprise components operate under one process, such as Domino, Oracle, and Java™. In these cases, it is possible to increase the user's address space size to 3 GB using the /3GB boot.ini switch for certain releases of Windows.

Does IBM® Lotus® recommends this course of action?


Answer

Before we provide the IBM Lotus recommendation for using the /3GB switch, it is important to provide the proper context for this discussion.

What does the /3GB switch do?
As mentioned above, on W32, the default size for the user address space is 2 GB, with the other 2 GB used as the kernel address space, for a total of 4 GB. In essence, you can think of the kernel address space as being shared across all processes (when they execute in kernel mode), with the same 2 GB being used by all. The user address space is mostly private to each process, unless the process chooses to allocate share-able memory.

On certain W32 platforms, it is possible to change the default percentages of the virtual address space by using the boot.ini parameter /3GB. Using the /3GB switch changes the split of the total virtual address space so that the user address space is increased to 3 GB, and the kernel address space is decreased to 1 GB. This setting in no way changes the overall virtual address space size (the process still only had 4 GB total), and it does not have an impact on how much virtual memory is managed by the system. The /3GB switch simply changes the split between user and kernel space for each process.

Below is a table that provides a list of W32 releases with which the /3GB switch is compatible. This information was taken from the Microsoft Article “Memory Support and Windows Operating Systems”:

W32 Versions that Support /3GB
Windows NT 4.0 Enterprise Edition
Windows 2000 Advanced Server
Windows 2000 Datacenter Server
Windows 2003 Server
Windows 2003 Enterprise Server
Windows 2003 Datacenter Server
Windows XP Professional

There are various implications when using this switch that must be considered, both application level (Domino) and kernel level (Operating System) factors:

Application (Domino) Level - The /3GB switch has its uses; however, even with this kernel-level setting enabled, a 32-bit process will not be able to use more than 2 GB user space unless it is compiled with the /LARGEADDRESSAWARE switch. This parameter must be specified when compiling the 32-bit executable.

If the kernel is configured with the /3GB setting, but an exe is not compiled with the Large Address Aware option, then the process’ user address space will still be restricted to 2 GB. Contrary to logic, the additional 1 GB is not given back to the kernel, but is instead flagged as a reserved block within the user address space. The result is that 1 GB of the virtual address space is lost. Hence, the /3GB switch should only be used if the process in question is compiled with /LARGEADDRESSAWARE. Misuse of the /3GB setting can cause more problems than it solves.

Beginning in Domino 7.0.1, the Domino server is compiled to be Large Address Aware, so that it can make use of the additional address space up to the full 3 GB. From an application-level perspective, this is all that is required to use the additional 1 GB of address space.

Kernel Level - The application level is not the only consideration. There is a far more important set of implications regarding kernel level performance. When the /3GB switch is used on W32, in actuality, one GB of the kernel's address space is taken, and given back to the user address space. This means the kernel's address space is cut in half.

While certain complex applications may need the extra room in their process address space, the drawback to using this switch is that the kernel must compress its memory usage into the remaining one GB. This will affect all processes system-wide, not only those that use the full 3 GB of user space, since it is the system that is booted into /3GB mode, and the kernel address space is shared across all processes. This can and will have substantial performance implications, particularly for certain kernel components such as the paged-pool, non-paged pool, and the System Page Table Entries (PTE).

Specifically, the use of /3GB will reduce the paged pool maximum size for Windows 2003 from 470 MB to below 200 MB. As default 80% page pool policy, the average paged pool would be around 160 MB, which is far below what is normally expected or desired, as Domino consumes a single 1 KB paged pool for each 1 MB database opening. In the worst scenario, Windows OS might log Event ID - 2020, or Domino might intermittently report the error "Hardware/OS error (Insufficient system resources exist to complete the requested service)."

In addition, when enabling the /3GB switch on Windows 32-bit, Lotus Domino performance testing indicates between a 10-15% increase in CPU usage (due to additional overhead from the constrained kernel). Domino testing indicates that this feature scales better under Windows 2003 than it does under Windows 2000.


/USERVA Switch
On Windows 2003, an additional switch is available to use in conjunction with the /3GB switch, known as the /userva switch. You can use /userva to tune the amount of the address space (in MB) that is given to user mode, using a value between 2048 MB (2 GB) and 3072 MB (3 GB). Since a given application may not require the full 3 GB space, but may perform well with 2.8 GB, this switch can be used and allow the remainder of the 3 GB to be returned to the kernel. This additional space for the kernel allows for an increase in the size of the PTE. For instance, you may specify a split of 2800 MB (2.73 GB) and 1296 MB (1.26 GB) with the following syntax:


/3GB /Userva=2800

Microsoft generally recommends that you keep the range of the /userva switch to between 2900 MB and 3030 MB. However, Domino performance testing indicates that a value of /userva=2800 works well. Again, this is only an option for Domino 7 and higher. For more information regarding the use of the /userva switch, see the following Microsoft article: http://support.microsoft.com/kb/316739


RAM vs. Address Space Size
Let's take a moment to stress one important point. The modification of the user and kernel address space sizes does NOT change the amount of RAM installed on the system. Furthermore, the relationship between RAM and the address space is not one-to-one. Pages accessed in the address space by a process can either exist in RAM, or the page file; the sum of these two media is referred to as virtual memory usage.

The fact that a process' user address space is increased to 3 GB does not mean that it will use 3 GB of RAM. It simply means that the process can allocate and access up to 3 GB of virtual memory. In fact, under the hood, the kernel makes the decision as to what data is paged out to disk, and what stays in RAM. The kernel is very conservative about what stays in RAM. So, in reality, very few pages in a process' user address space reside in RAM at any one time.

Another way to put it: the process cannot demand that it be given a certain amount of RAM. It is ultimately up to the kernel as to what stays in RAM and what gets paged out. In most instances, a process is not technically aware of which parts of its data are paged in and out of the page file. In short, the application knows no difference between RAM and page file. Therefore, we cannot directly discuss how much RAM a specific application requires, but in theory, only how much virtual memory it may use.

Misconceptions
You may be thinking: "OK, if I use the /3GB switch, and I have 8 GB of RAM, that should be 3 GB of RAM for Domino Partition #1, 3 GB of RAM for Domino Partition #2, leaving 2 GB of RAM for the kernel, right?"

Wrong.

It is important to refrain from drawing a one to one ratio between RAM and address space. While these concepts are related, they are not synonymous. Again, the majority of the user address space for a single process usually points to pages on disk at any given moment, with only a small portion of its address space pointing to RAM. The amount of RAM usage by a single process is not calculable in advance; it depends on process and system activity, and is ultimately decided by the kernel at runtime. The only way to get a rough figure of RAM usage within a particular environment is through a simulation of user load (as realistic as possible).

You might also be thinking: "OK, if I have less than 2 GB or RAM, how much RAM does the kernel get vs Domino? Does Domino get 3/4, and the kernel get 1/4, assuming I am using the /3GB switch?"

Again, you cannot draw a direct relationship between RAM and user address space. Even if you have less than 2 GB of RAM, the address space for the user is unchanged at 3 GB (assuming you are using the /3GB switch). The decreased amount of RAM just means that more if its address space points to the page file at any given moment, as controlled by the kernel.

The main point to take away from the above paragraphs is that the /3GB switch is used to adjust a process' address space, not its RAM usage, since low memory conditions reflect a limitation in address space rather than RAM usage.


The /PAE Switch
On Windows, by default the kernel can manage up to 4 GB of RAM (this does not include the page file). However, under certain versions of Windows, namely the Advanced Editions of 2000 and 2003, an administrator can increase this usage by using the setting /PAE within the boot.ini file. The /PAE (Physical Address Extension) allows the kernel to recognize and manage all of the installed RAM, up to 64 GB. The upper range depends on the exact version of Windows, but is no greater than 64 GB. Of course, in order to make use of the /PAE, it is necessary to have the correct memory hardware that contains the 36-address pin configuration.

The use of the /PAE setting does NOT affect the size of the virtual address space for each 32-bit application, but only affects the total amount of RAM that the kernel/CPU can manage system-wide.

To summarize:
/3GB affects the virtual user address space of each process
/PAE affects the amount of RAM that can be managed by the OS

The /PAE switch affects the amount of RAM that the CPU can address. The use of page file is not affected by this switch, since the page file is stored sequentially on disk. A system may commit far more than 4 GB of virtual memory (across all processes) without the /PAE switch enabled, where most of this memory usage is in the form of the page file, and where only a portion of RAM is in use. By its nature, the kernel can manage the full amount of RAM, and can allow the total virtual memory usage across all 32-bit applications to exceed far greater than 4 GB. In other words, you don’t need to have the /PAE switch enabled to commit more than 4 GB of total virtual memory (the sum of RAM and page file), but only if you want to make use of more than 4 GB of RAM across all processes.

Below is a table that provides RAM limits (using /PAE) for various versions of Windows Operating Systems. This information was taken from MSDN articles Q283037 and Q292934:

Windows 32-bit Version Maximum RAM Support with /PAE
Windows XP Professional 4 GB
Windows 2000 Server 4 GB
Windows 2003 Server 4 GB
Windows 2000 Advanced Server 8 GB
Windows 2003 Enterprise Server 32 GB
Windows 2000 Datacenter Server 64 GB
Windows 2003 Datacenter Server 64 GB
Windows 2000/2003 Datacenter Server (with /3GB) 16 GB


/3GB vs. /PAE
The simultaneous use of /PAE and /3GB are to some degree at odds with each other. In order to support additional RAM, the /PAE switch requires more kernel space. However, when used in conjunction with the /3GB switch, the kernel actually has less address space to use. The net result is that a system running both /3GB and /PAE cannot address any more than 16 GB of memory at best, due to space considerations with the kernel. This limitation is described in MSDN article Q283037 titled “Large Memory Support is Available in Windows 2000” - it reads:

…A program that requests 3 GB of memory is more likely to be
able to have more of its memory remain in physical memory
rather than be paged out, which increases the performance of
programs that are capable of using the /3GB switch. The
exception is when the /3GB switch is used in conjunction with
the /PAE switch, the operating system does not use any memory
in excess of 16 GB. This behavior is caused by kernel virtual
memory space considerations. Thus, if the system restarts with
the /3GB entry in the Boot.ini file, and the system has more than
16 GB of physical memory, the additional physical random
access memory (RAM) is not used by the operating system…


IBM Lotus Domino recommendations
While the Domino 7 server is Large Address Aware, and while it may seem attractive to increase the address space of a Domino 7 server, Lotus r ecommends against the use of the /3GB switch, and recommends that you use this option only when there is no other alternative. Decreasing the kernel's address space can and will have a negative impact and essentially steals the kernel's resources to increase the user's resources. Since Domino 6 is not Large Address Aware, using the /3GB switch on Domino 6 has no affect, and only takes away the kernel's address space.

In cases where a server is crashing due to the 2 GB limit, there are various options available:

  1. Use the /3GB Switch: As discussed above, this is the least desirable option, and is generally not recommended. If you use this option, consider using the /userva=2800 syntax. Due to its potentially constrained nature, it it strongly recommended that you test this configuration prior to implementing it in production.

  2. Memory Tuning: Tune down memory usage, including reducing shared memory size, reducing thread counts, and redesigning applications and agents. In a number of cases, a defect or misconfiguration may be the cause for higher than expected memory usage, and action can be taken to remedy this situation. You should call into IBM Lotus Support for assistance on troubleshooting high memory usage.

  3. Domino Partitioned Servers: In cases where no Domino defect is found or no further tuning or configuration changes can be made, another option is to spread the user load across multiple Domino partitions. Since each Domino partition has its own set of address spaces, dividing the application load across multiple partitions allows for significantly more virtual space inside which to maneuver (assuming that the Domino server is not originally hardware constrained). The same end can be accomplished through the use of Domino partitions on VMWare.

  4. Windows 2003 64-bit: Another option is to run your 32-bit Domino Server on the Windows 2003 64-bit Operating System. In doing so, this moves the kernel's address space out of the 32-bit address space and into a separate 64-bit address space. This frees up the Domino 7 user address space to occupy nearly the entire 4 GB (32-bit) address space (since it is compiled as Large Address Aware). Since Domino 6 is not Large Address Aware by default, it still has a 2 GB user address space. See related information for additional caveats on W64.

  5. Domino Native 64-bit: Once available, the Domino 8 Native 64-bit version will remove the 32-bit limitation on address space altogether, providing a much larger address space. However, this will not remove all Operating System limits or Domino limits, since these limits are enforced to maintain a balanced server. Don't forget, within the native 64-bit environment you may once again encounter your hardware limits (i.e. it will once again become theoretically possible for a single 64-bit process to exhaust system-wide RAM). For details on Domino 8 64-bit availability, see:



FAQ; 64-bit version of Domino (Technote #1296452)


Related Microsoft Information
Under Windows 64-bit, it is no longer necessary to use either the /3GB switch or the /PAE switch, because, by default, the 64-bit Kernel can fully manage large amounts of RAM. In addition, as mentioned, each 32-bit process is able to use nearly all 4 GB in user mode (provided it is compiled as Large Address Aware).

Below is a table which has been taken and modified from Microsoft article #294418. For more information, please refer to the Microsoft article titled "Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003" (see link below)

Windows Component
64-bit Windows
32-bit Windows
32-bit Total Address Space (per process)
4 GB
4 GB
32-bit User Address Space (per process)
4 GB (LAA)
2 GB (non-LAA)
3 GB ( LAA with /3GB)
2 GB (default)
64-bit Total Address Space (per process)
16 terabytes
N/A
64-bit User Address Space (per process)
8 terabytes
N/A
Paging file size
256 terabytes
16 terabytes
Hyperspace
8 GB
4 MB
Paged pool
128 GB
470 MB
Non-paged pool
128 GB
256 MB
System cache
1 terabyte
1 GB
System PTEs
128 GB
660 MB

Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
http://support.microsoft.com/kb/294418

Large memory support is available in Windows Server 2003 and in Windows 2000
http://support.microsoft.com/kb/Q283037

How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
http://support.microsoft.com/kb/316739

Memory Support and Windows Operating Systems
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

Windows 2000 Datacenter Server Does Not Locate Memory Greater Than 16 GB
http://support.microsoft.com/kb/Q292934

Use of the /3GB switch in Exchange Server 2003 on a Windows Server 2003-based system
http://support.microsoft.com/default.aspx?scid=kb;en-us;823440

Server is unable to allocate memory from the system paged pool
http://support.microsoft.com/kb/312362

Related information

FAQ: 64-bit version of Domino

Document information

More support for: IBM Domino

Software version: 6.0, 6.5, 7.0, 8.0, 8.5

Operating system(s): Windows

Software edition: All Editions

Reference #: 1257005

Modified date: 09 June 2010