IBM Support

Recommendations for setting NSF_BUFFER_POOL_SIZE_MB

Technote (FAQ)


Question

On computer systems with more than 2 GB of RAM, specifically on Unix platforms (Linux®, AIX®, Solaris), if the NSF_BUFFER_POOL_SIZE_MB parameter is not set, then the Lotus® Domino® 6.x and 7.x server automatically calculates the maximum value for the Unified Buffer Manager (UBM) as 3/8th's of 4 GB Physical RAM (assuming 4 GB of RAM is installed on the machine). This calculation can possibly result in an unacceptably high UBM size of 1.5 GB. How can this potential risk be mitigated?

Cause

By default, Domino 6 and 7 use the amount of RAM installed on the machine (up to 4 GB) to calculate the maximum size for the Unified Buffer Manager (UBM). On UNIX, if no parameters are used to mitigate the UBM maximum size, it may grow to 1.5 GB in size. When added to other Domino shared memory components, total shared memory usage may exceed per-process limits, which can cause problems on certain UNIX platforms such as AIX.

Answer

This document provides the most up-to-date recommendation for setting the NSF_BUFFER_POOL_SIZE_MB parameter to ensure maximum stability of the Lotus Domino 6, Domino 7, and Domino 8 servers. This document supersedes previous recommendations for the various INI parameters that affect the UBM size.


Topics in this article
Recommendations for Domino 6 & 7
Recommendations for Domino 8
About partitioned Domino servers
Recommended UBM sizes
Details about the UBM
Details about various memory INI parameters
Comparing the memory INI parameters
Default values for memory INI parameters



Recommendations for Domino 6 & 7 [return to top]
When installing a new Domino 6 or 7 server, particularly on UNIX, z/OS®, or i5/OS® platforms, it is now the best practice to directly set the size of the UBM with the NSF_BUFFER_POOL_SIZE_MB parameter to a value of 512 MB or lower. In an upgrade situation, if the NSF_BUFFER_POOL_SIZE_MB (or similar parameter) is already set, there is no need to change this value. Generally speaking, Domino 6 or Domino 7 on Windows32 can be run with the default UBM size.

Several other notes.ini file parameters can also be used to affect the size of the UBM:

  • PercentAvailSysResources (Domino R5 and later)
  • ConstrainedSHM, ConstrainedSHMSizeMB (Domino 6 and later)
  • MEMAddressableMem, MEMAddressableMemSizeMB (Domino 7 and later)
  • MEM_EnablePreAlloc, MEM_EnableSubAlloc (Domino 7 and later)

While these other parameters exist, it is now recommended that the primary mechanism for modifying the UBM (Unified Buffer Manager) size is to use the NSF_BUFFER_POOL_MB parameter . Also recommended is that this parameter should be set for any system with more than 2 GB of RAM, with a value not to exceed 750 MB. Values higher than this result in increased memory usage without appreciable performance improvements. Refer to the section on Recommended UBM sizes for specific value recommendations.

It is recommended that when configuring new Domino servers, administrators migrate away from the use of the other parameters listed above in order to affect the size of the UBM (NSF Buffer Pool).

Note: The use of ConstrainedSHM is not considered the optimal method of setting the maximum UBM size, because it affects the UBM size only indirectly. However, once the UBM maximum is set using NSF_BUFFER_POOL_SIZE, ConstrainedSHM can be further used to control the overall amount of shared memory usage as needed (because the UBM is just one component of shared memory usage). See the section below on Memory INI details for specific behavior.



Recommendations for Domino 8 [return to top]
Domino 8 now enforces a maximum UBM size of 512 MB by default on Windows32, Linux, AIX, Solaris and z/OS platforms, and it enforces a maximum UBM size of 400 MB on i5/OS. Therefore, there is no need to take action to properly size the UBM in Domino 8. For additional details on this new behavior, refer to the document " New default limit on UBM size in Domino 8" (#1268988)



Partitioned Domino servers [return to top]
In Domino Release 5 and earlier, it was necessary to configure partitioned servers so that memory resources were balanced between partitions (using the parameter PercentAvailSysResources). This was particularly the case on systems with less than 2 GB of RAM. However, with more modern systems, the hardware easily outstrips a single 32-bit process' ability to use RAM. Modern systems are better positioned to handle the RAM demands of multiple Domino servers. In addition, Domino versions 6 and later include internal checks designed to enforce the native 32-bit limits, further reducing the need to balance memory usage between partitions.

As a consequence, the set of UBM sizing recommendations listed here apply equally well to partitioned Domino servers as they do to single Domino server installs. Given that the 32-bit process address space is the current limitation on Domino memory usage, on systems with greater than 4 GB of RAM, there is little distinction between sizing the UBM on a partitioned server and sizing the UBM on a single server install.

When setting the max UBM size on newly installed partitioned servers on systems containing more than 2 GB of RAM, it is recommended that you adjust the UBM size in the same way as you would on a single install (by setting the size to 512 MB or lower). On systems with less than 2 GB of RAM, it is not recommended for newer versions of Domino to run partitioned servers.



Recommended UBM sizes [return to top]
As clarification, the various INI parameters listed here, including NSF_BUFFER_POOL_SIZE_MB, either directly or indirectly affect the maximum size of the UBM. These parameters specify the largest size to which the UBM can grow, and does not necessarily specify how large the UBM will be at any given moment, based on demand. However, on most heavily loaded servers, it is likely that the UBM will hit the maximum size specified.

Below is the recommended range of values for the maximum UBM size, depending on platform:

Operating system Recommended UBM Maximum (Domino 32-bit)
Windows32 512 MB - 750 MB
Linux 512 MB - 750 MB
AIX 512 MB - 750 MB
Solaris 512 MB - 750 MB
i5/OS 512 MB - 750 MB
z/OS & zLinux 200 MB - 400 MB

The lower values for i5/OS and z/OS are a result of more efficient file caching on those platforms, which mitigates the need for a larger UBM maximum. While the range for the max UBM size is relatively broad, testing by IBM and results in the field have shown that 512 MB is sufficiently large for Domino on Windows32, Linux, AIX, Solaris and i5/OS, while a value of 400 MB is sufficiently large for z/OS & zLinux.

Considerations for Domino on AIX
On AIX, 3/8th's of 4 GB (or approximately 1500 MB) is 3/4 of total shared memory (2 GB) that can be allocated by Domino when using the Large Memory Model. Most large environments require more than 500 MB of shared memory for the additional activities outside the buffer pool; hence the increased buffer pool size can result in outages (since total shared memory activities can exceed the 2 GB limit on AIX).

While the use of ConstrainedSHMSizeMB=2048 (or lower) is valid, beginning with 7.0.2 FP1, the best practice is to use AIX_LIMIT_SHM_SEGMENTS=8 (or lower) instead.

Starting with 7.0.2 FP1, the best practice to control the number of shared segments is the notes.ini parameter AIX_LIMIT_SHM_SEGMENTS. If you want to maintain the same amount of shared memory as previous versions of Domino, then set AIX_LIMIT_SHM_SEGMENTS=8 (8*256=2048)

If you were using ConstrainedSHMSizeMB below 2048, use the following calculation to change to AIX_LIMIT_SHM_SEGMENTS: Integer of ( ConstrainedSHMSizeMB / 256 )
    Example of calculating AIX_LIMIT_SHM_SEGMENTS when ConstrainedSHMSizeMB = 1800:
    (1800 / 256) = 7.03125 = 7

Customers wanting more shared memory starting 7.0.2 FP1 must set AIX_LIMIT_SHM_SEGMENTS to 9 or 10 and additionally set AIX_VERY_LARGE_MM=1 in the notes.ini. If the NSF_BUFFER_POOL_SIZE_MB setting is not in place, more memory may be directed to the buffer pool, possibly resulting in unacceptably high memory usage.

Learn more about this new parameter AIX_LIMIT_SHM_SEGMENTS by reading document #1253505 titled " AIX_LIMIT_SHM_SEGMENTS used to control shared memory on Domino/AIX ."



Details about the UBM [return to top]
The Unified Buffer Manager (UBM) constitutes the single largest contribution to shared memory usage on the Domino server. The UBM (referred to as the NSF Buffer Pool in older releases of the Notes server) is responsible for caching disk I/O for all the databases on a server, improving the performance of the server. By default, Domino sets the maximum size of the UBM based on the amount of RAM installed on the system, taking roughly 3/8th of installed RAM. Furthermore, Domino 6 and Domino 7 cap the amount of RAM used in this calculation at 4 GB, because a single 32-bit process can use no more than 4 GB of RAM.

The optimal size for the UBM varies based on platform and load. However, generally, the accepted range for the UBM is 500 MB - 750 MB for Windows and UNIX, and 200 MB - 400 MB on i5/OS and z/OS. The UBM size can be further tuned by examining the statistic:

Database.Database.BufferPool.PercentReadsInBuffer

This value should remain at 90% or above for optimal performance (as long as you do not exceed a UBM size of 750 MB). A decrease in UBM size may require a moderate increase to the DBCache.MaxEntries parameter. You want to minimize Overcrowding rejections by increasing the MaxEntries value to 1.5x of the highwater mark. You can monitor the following statistics to tune this cache appropriately to compensate for a smaller UBM.

Database.DbCache.CurrentEntries
Database.DbCache.MaxEntries
DbCache.OvercrowdingRejection
Database.DbCache.HighWaterMark

On Unix, if no INI parameters are used, Domino sets the upper size of the UBM at 3/8th of 4 GB, resulting in a UBM up to 1.5 GB in size. This maximum size for the UBM can be problematic due to increased memory usage in cases where servers are heavily loaded. As a result, setting a more moderate upper size for the UBM is one of the most important aspects of tuning Domino memory usage.

Although IBM does provide a range of Recommended UBM Sizes, a one-size-fits-all approach is not appropriate for every environment; some manual adjusting of the UBM size may be necessary for some environments. For instance, in a mail environment, little data is shared between users and/or threads, so it does not make sense to have a large UBM. However, in cases of custom workflow applications, more data may be shared, so an increase in the UBM size may be necessary.

Note : It is important to comment that the maximum UBM size refers to the maximum amount of virtual memory used on the system rather than the amount of RAM used. With the exception of Domino on i5/OS, Domino does not have direct control over how much of the UBM is actually contained in RAM vs the page file. This means that, depending on Operating System and activity, the amount of RAM used by the UBM will only be a portion of the total UBM usage. For more explanation on this point, see the RAM vs. Address Space discussion in the section on considerations for Memory Usage in the Domino Performance Best Practices (document #7008849).



Details about various memory INI parameters [return to top]
In Domino 6, Domino 7, and Domino 8, there are eight separate INI parameters that can be used to control the size of the UBM, and hence indirectly, overall Domino Shared memory usage:
  • PercentAvailSysResources - Introduced in R5, this parameter was used to reduce memory usage on systems with large amounts of installed RAM, as well as shared resources between partitioned servers. In short, this parameter affects only the size of the Unified Buffer Manager (UBM), which makes up only one part of memory shared memory usage. In addition, this parameter does not take into account the inherent limits of a 32-bit process address space. With the introduction of additional memory usage parameters in Domino 6, it is strongly recommended to discontinue the use of this INI parameter.
  • ConstrainedSHM , ConstrainedSHMSizeMB - Introduced in Domino 6, this parameter actually enforces an overall limit to the amount of shared memory a Domino server will use. Unlike PercentAvailSysResources, this parameter not only affects the size of the UBM, but also enforces a hard limit on the total amount of shared memory that a single Domino server will allocate. Additionally, the default values for this parameter take into account the limits of a 32-bit process address space for each platform, scaling memory usage more effectively than the use of PercentAvailSysResources. Consult the table below for the default value of the parameters on each platform.
  • Mem_AddressableMem, Mem_AddressableMemSizeMB - Introduced in Domino 7, these parameters behave in a similar fashion to PercentAvailSysResources, in that they affect the size of the UBM, but do not actually enforce an overall limit to shared memory usage. Mem_AddressableMem differs from PercentAvailSysResources in that it takes into account the user address space size on each platform for a 32-bit process. Consult the table below for the default value of the parameters on each platform.
  • MEM_EnablePreAlloc, MEM_EnableSubAlloc - Available starting in Domino 7, these parameters further extended control over Domino memory management. MEM_EnablePreAlloc will cause Domino to allocate all shared memory up to its limit up-front when the server starts. For example, on Windows32, Domino would allocate 1.5 GB of shared memory up-front, whereas on Solaris, Domino would allocated 3.4 GB of shared memory. MEM_SubAlloc forces Domino to allocate large pools of shared memory, for example 200 MB, out of which traditional DPools are allocated, which range from 4 MB to 8 MB. When these two parameters are turned on, they automatically invoke both ConstrainedSHM and MEM_AddressableMem, which are used to enforce the maximum shared memory limits, as well as indirectly the UBM maximum size. Consult the table below for the default value of the parameters on each platform.
  • NSF_Buffer_Pool_Size_MB - Introduced in R4, this parameter directly sets the maximum UBM size rather than relying on an indirect calculation as is the case with the above parameters. This INI only affects the UBM and does not serve to limit overall shared memory of a Domino server. The advantage of this parameter is that you eliminate the guess work involved in setting the UBM's maximum size. This INI can be used in conjunction with ConstrainedSHM, as it takes precedence when sizing the UBM, where ConstrainedSHM can still limit overall shared memory usage.

Note: Domino 7.0.2 FP1 on AIX has introduced a new parameter than is the preferred method of constraining shared memory, one that is more intelligent regarding the AIX segmented architecture. Please refer to document #1253505 for more information about the parameter AIX_LIMIT_SHM_SEGMENTS.



Comparing the memory INI parameters [return to top]
It is important to realize that nearly all the above INI parameters have been designed to impact the amount of RAM that Domino recognizes on the system, which in turn impacts (indirectly) the maximum size of the UBM, because it is based on perceived RAM. In Domino 6 and Domino 7, out of the box, Domino scales the amount of RAM that it "perceives" on the system down to 4 GB (assuming more than 4 GB of RAM is installed); this is done because a 32-bit application can never manage more than 4 GB of RAM. However, even with this reduction down to 4 GB, the UBM may still exceed a desirable size as outlined above. Under such circumstances, the various INI's listed above can be used to further reduce the amount of perceived RAM that Domino uses so that a more reasonable UBM maximum size is achieved.

As originally envisioned, reduction in the amount of perceived RAM could potentially affect other memory pools and caches. However, in actuality, as it exists in Domino 6, 7 and 8, only the UBM is affected by adjusting the amount of perceived RAM on the system. This means that once the UBM is sized, most of the above listed INI parameters have no additional affect. As a consequence of the new recommendation to use NSF_BUFFER_POOL_SIZE_MB directly, most of these additional parameters are no longer needed, with a few exceptions.

Below is a table to compares the behavior of these various parameters to give you a better idea of each parameter's use:

INI parameter Affects calculated RAM Affects Max UBM Size Limits Total Shared Memory Usage Accounts for Address Space Size
PercentAvailSysResources Yes Yes No No
ConstrainedSHM
ConstrainedSHMSizeMB
Yes Yes Yes Yes
MEM_AddressableMem
MEM_AddressableMemSizeMB
Yes Yes No Yes
MEM_EnablePreAlloc (*)
MEM_EnableSubAlloc (*)
Yes Yes Yes Yes
NSF_BUFFER_POOL_SIZE_MB No Yes No No

Note (*): MEM_EnablePreAlloc and MEM_EnableSubAlloc are considered to limit shared memory due to the fact that they both enforce the ConstrainedSHM parameter by default when enabled.

The main point to realize from this table is that, while nearly all of these parameters affect the amount of RAM that Domino recognizes on the system and while they affect UBM size one way or another, few of them have additional benefit beyond establishing the UBM maximum.

The only parameters that are further useful are ConstrainedSHM, MEM_EnablePreAlloc and MEM_EnableSubAlloc, which have the further role of limiting overall shared memory usage (the UBM is just one component of shared memory usage). Additionally, MEM_EnablePreAlloc and MEM_EnableSubAlloc are only used under the very specific case where the administrator wishes to allocate all shared memory at server startup. This is done so that other components do not prevent Domino memory usage. This is usually not needed nor recommended unless you have a specific explicit need to use this feature based on past experience, or as directed by IBM Support.

Once the UBM is set directly, ConstrainedSHM is most likely the only additional parameter that one will need to consider using, and even then, only if one needs to ensure that Domino leaves enough room in the address space for other components.

WARNING: Administrators should only override the value for ConstrainedSHM after discussing an appropriate value with IBM Support. Setting this value too low can cause the Domino server to crash if it is unable to allocated shared memory for a critical operation. ConstrainedSHMSizeMB should not generally be set lower than 1500 MB.



Default values for memory INI parameters [return to top]
Below are two tables that list the default values for each of the memory usage INI parameters discussed above:

Operating System ConstrainedSHM default in Domino 6 ConstrainedSHM default in Domino 7
Windows32
2 GB
1.5 GB
Linux
3 GB
2.5 GB
AIX
2.25 GB
2 GB
Solaris
3 GB
3.4 GB
z/OS
2 GB
1.5 GB
i5/OS
2 GB
2 GB

Operating System MEM_AddressableMem default in Domino 7 MEM_EnableSuballoc default in Domino 7
Windows32
2 GB
200 MB
Linux
3 GB
256 MB
AIX
3 GB
256 MB
Solaris
3.9 GB
256 MB
z/OS
2 GB
256 MB
i5/OS
3 GB
256 MB



Each of these default values can be adjusted using its counterpart listed below:

Enforce default using Change default using
ConstrainedSHM ConstrainedSHMSizeMB
MEM_AddressableMem MEM_AddressableMemSizeMB
MEM_EnableSuballoc MEM_EnableSuballocSize

Note: As previously stated, you should only change the default value upon direction from IBM Support. In most cases, these values will be lowered (not raised) in order to limit memory usage.

Related information

New UBM limit in Domino 8
Notes/Domino Best Practices: Performance
AIX_LIMIT_SHM_SEGMENTS used to control shared memory...
Shared Memory can be pre-allocated at startup in Domino
FAQ: 64-bit version of Domino

Document information

More support for: IBM Domino

Software version: 6.5, 7.0, 8.0, 8.5

Operating system(s): AIX, IBM i, Linux, Solaris, Windows, z/OS

Software edition: All Editions

Reference #: 1286171

Modified date: 05 November 2007