Open Mic Webcast Replay: Deep Dive into Domino Memory - 20 March 2012
STEwebcastDocument; lste_webcast; Tech Exchange; STEwebcast; openmic call; open microphone call; discussion; conference call; customer call; submit; ask the experts; discuss; open mic; questions; answers; questions and answers; q&a; replay; recording; webcast; FAQ; best practice; transcript; domino memory
IBM hosted an Open Mic webcast with Lotus Development and Support Engineers on 20 March 2012 at 11:00 AM EDT, on the topic "Deep Dive into IBM Lotus Domino Memory."
|Presentation and associated files|
Just an FYI: There are no shared memory managers in the OS other than very simple allocations, which is part of why we have our own for shared
Q. For AS/400 the Domino memory management is the same, for example, HTTP XPages?
A. Yes, Notes memory managers are consistent across all platforms until we get to native APIs, like malloc/mmap/shmat/MapViewOfFile. Basically, private + shared = total used memory, which has to be = the addressable memory for process. For 32 bit on 32 bit, this is 2GB because the kernel uses 2GB, while for 32 bit on 64 bit, this is usually 4GB since the 64-bit OS has 16 Exabytes of addressable space, except for AIX, which still has a segmented memory architecture.
Also, for the default nsf buffer pool size, it's 1GB for 64-bit Domino, and 512MB for 32-bit Domino.
Q. How much work has been done toward changing the default parameters within Domino to take advantage of 64-bit Domino on Microsoft Windows platforms?
A. Is there an issue you see on 64 bit? If you are running out of memory somewhere, we need a PMR/SPR on it. We did change some things for 64 bit; for example, I tested allocating 120GB of shared memory, but we didn't change everything. To be able to get to that 120GB, I had to make some code changes
Q. Since most servers are now running on 64 bit, and the price of memory is very low, it is difficult to figure out exactly what to change to take advantage of 64 bit. For example, I tell all our Windows customers just to get to 32-bit Domino on 64-bit Windows, and expect that fixes all of their memory issues.
A. Are you running 32- or 64-bit Domino? For most workloads 32-bit Domino on a 64-bit OS provides that additional 2GB of addressable memory and is sufficient. For 64-bit Domino we've not had reports of any memory lacks, but if we do get them, we will work on them. If you see a pool filling up, just PMR/SPR it and we'll evaluate it. I have done some changes in 64 bit but will only change other things if customers see a deficiency/issue.
Q. I have noticed that some settings are starting to change to take advantage of 64 bit, but is it best to just keep adding memory and then have Domino adjust settings automatically, or should I change all the memory settings? If I have unlimited memory, why not increase all settings to the maximum on 64 bit ?
A. Because we've had issues with that; for example, when we increased the default stack size to 4MB for 64 bit, we had problems with not having enough physical memory to handle the number of threads required for Notes Traveller. Plus, if it's not "broke", we don't fix it. There are so very many pools in Domino, and we can only work on those with which we see issues. Also, keep in mind that a Domino partition is ~20 processes, so if the partition is using 2GB shared, and each process uses, say, 512MB local memory, then that amounts to 12GB memory.
Q. So did you revert back on the 4MB setting ?
A. We shrunk it to 1MB for 64 bit, and 512KB for 32 bit.
Q. Where can I direct questions/issues regarding font display settings and calendar views?
A. That won't be covered during this call, but you can direct your questions to our Notes/Domino forum http://www.lotus.com/ldd/nd85forum.nsf/Dateallthreadedweb?OpenView.
FYI: The Namelookup cache default size is 16MB for Domino versions prior to 8.5.2. Starting in Domino 8.5.2, the default is 64MB. For most customers the default settings for all these pools are fine. I would highly recommend not changing the defaults without a engagement with Lotus Support, who will work with you to determine if it's appropriate.
Q. What is considered to be a very large Inbox?
A. I like to keep Inbox's at 90 days, but I also leave Unread Documents in the Inbox.
Q. 90 days seems like a lot of mail; for some of our users, it could mean a considerable number of messages.
A. It's really up to the Administrator. Certainly could do 60 or 30 in the Mail Inbox Maintenance configuration for each server, under the Server Tasks Tab --- Administration Process Tab. Also, it's important to train your users ahead of time to mark things Unread that they want to retain that are older than the 30, 60 or 90 days.
Q. We will look at that to see if we can add some management in that area; we're thinking in terms of numbers of emails instead of days.
A. Understood, and I've heard that request before. Currently it's only controlled by days, but I'll pass it along to the Admin Development Team again.
Q. What about mail file size versus Inbox size?
A. Mail file size is no longer a direct indicator of memory usage, as the result of the Mail Inbox maintenance tool. Before this tool you could infer the size of the Inbox from the size of the mail file. The key consideration for memory impact of the mail file is on the COUNT of entries in the Inbox, not the size of the mail file. Generally, a count below 300 is good and even below 1000 is OK, but anything above that is a concern that should be addressed.
Q. The default memory settings for HTTPJVMMaxHeapSize to support XPages seems to have been increased in later Domino releases---I think the last time was 8.5.2---but servers upgraded in our environment did not get this new setting.
A. Yes, the JavaMaxHeapSize default has changed for both 32 bit and 64 bit, to help provide more memory for XPages. For 32-bit Domino, the default HTTPJVMMaxHeapSize was 256MB in 8.5.0 and 8.5.1; whereas in 8.5.2 and later, the default is 64MB. In 64-bit Domino the default HTTPJVMMaxHeapSize is 1 GB.
Q. But it was only the default that was changed, so I had to change it manually on all old servers being upgraded.
A. Setting HTTPJVMMaxHeapSizeSet=1 in the Notes.ini will keep the server from resetting back to 64MB. The Domino Configuration Tuner would have flagged that.
Q. When you change default settings and servers were running the default settings, can you please have the value updated during the upgrade? In other words, if we have changed some settings, please don't change them, but any settings that were at default should be updated to the new default settings. Presumably most servers would be running default settings, but not the same defaults, if they were initially installed on Domino 8.5, maybe the Tuner could find all settings that are not on the new default settings?
A. I'll talk to the developer who belongs to this area of code. Of course, that's an HTTP-specific setting and does not affect other Notes processes. I think historically when we change defaults we always honor Notes.ini settings first, as is true with the NSF buffer pool size variable. Your Notes.ini settings always take precedence, but the installer doesn't know this, so it cannot update accordingly.
Q. Can you address Domino 8.5.x 64-bit for Linux on System z (OS390) 64-bit; specifically, what is the size of the process address space? 4GB, 1TB??
A. 64-bit Domino for Linux on zSeries has 64 bit address space, so it's whatever the OS let's us have; 16 Exabytes is 64 bit address space. So if the OS does the same 50/50 as was the case in the past, it means we have 8 Exabytes of address space.
Q. Do you restart just Domino or the server, to clear out memory?
A. Restarting Domino will clear out Domino memory (shared and private).
Q. We are using a Domino business partner's Defrag tool. Does defragging assist with improving memory usage?
A. It doesn't since they do not know about Domino internal memory structures/layout, at least not that I know of. I'd have to research whatever the tool is.
Q. It's defrag.nsf by Preemptive Consulting in Australia. Works like a charm.
A. Sounds like it works on NSF files, not our memory. The defragging is for file fragmentation and not memory. Just an FYI to all: We do not ship memstats.
Note that it's a good idea to defrag your drives from time to time with Domino Down. Also, rebuilding views and rebuilding full text Indexes from time to time defrags each of those. Copy-style compacting of databases defragments storage in the .NSF as well. So your tool may be doing any of those things. Would not be surprised if Domino already has out-of-the-box tools available that would do what this third-party tool is doing, but they may make it easier.
Q. How does the "logical disk usage exceeds configured thresholds" error message relate to memory?
A. The "logical disk usage" message refers to disk queue lengths exceeding the suggested thresholds, which means that there are too many requests waiting on the disk. It is not a memory-related message.
General FYI: Local memory that goes to the operating system is still managed in part by Domino as we are handle based.
Q. Is "EVENT_CORRELATION_POOL_SIZE" also a memory-related Notes.ini entry?
A. Yes, it controls the size of the event cache pool. But, again, I would not change it unless instructed to by Lotus Support after analyzing any issues you have.
Q. When this pool is full (per the log.nsf), can I ignore it or should I care?
A. Yes, you should care. You might need to work with Support to determine why you are at FULL. This may be related to Domino Domain Monitoring (DDM) and might be something configured that needs some tuning. You should also care because it means you are no longer capturing events, and it could mean there is something occurring too often---something that you don't want to occur, like an agent going wild.
If you are seeing the EVENT_CORRELATION_POOL_SIZE message, it often requires the pool to be increased. The default size is 70MB, with a Max size of 100MB. If, after increasing the pool, you still see this message and would like further review, please open a ticket with Support for review.
Q. What is the statistic to check the dpool response?
A. If you are asking what stat can be reviewed to evaluate the performance of memory pool searches, then SH MEM DUMP will generate a file named "memory_juliandate.dmp" in the IBM_TECHNICAL_SUPPORT directory. Look in this file for a section like this:
209 system shared memory pools
1132 MB total pools size
1036 MB total pools used
91.55% pool utilization
14.78 pools visited per allocation
13.75 pools skipped
1.03 pools searched
28.34 free blocks searched per allocation
25.48 free blocks searched per free
Note the POOLS SEARCHED value; if this is below 1.8, then memory allocation searches should not be affecting performance.
Q. What would be the most important stats to monitoring Mem performance?
A. I can't suggest a single best stat to monitor for memory usage; however, the overall allocated memory (mem.allocated) is a good indicator of overall health.
Q. Our monitoring system (Orion) has a problem with monitoring memory in that it adds up to InUse and Standby, because the Standby will be the rest of the free memory. Free memory is actually 1 or 0%. Any experience on monitoring memory? By the way we are running Domino 8.5.2 FP1 on Windows 2008 R2 64 Bit, Auf VMWare ESX Servers.
A. Domino Domain Monitoring provides Alerts based on actual Domino allocated memory, and Warnings are raised when memory usage on a system is high.
Q. How can you increase the size of the Name Lookup Cache in Domino?
A. You can control the Name Lookup Cache via the Notes.ini parameter NLCache_Size=# in bytes:
Database.NAMELookupCacheMaxSize = The maximum size, in bytes, allowed for the namelookup cache on this server .
Database.NAMELookupCachePool.Peak = The maximum size, in bytes, used for the namelookup cache on this server since startup.
Database.NAMELookupCachePool.Used = The current size in bytes used for the namelookup cache on this server.
NOTE: Link to the forum entry for this session. If we could not get to your question, please post it here:
Q. We have an application server using a Web service to service users' application requests. We have multiple applications on the server and have very large memory usage being allocated for HTTP requests. Is it possible to determine which of the applications of the those hosted thru HTTP is the driver of that high memory usage?
A. Yes, but you'll need some help. IBM has tools to help you debug and determine where that memory usage is in your application---whether it's native in Domino or Java. Go to the discussion forum, post this question there, and we will follow up.
Q. Regarding the last slide in the PRZ, in which you discourage overallocating effective virtual mem: Can you make concrete suggestions, for example, what page sizes to use for a 4 GB system, and for an 8 GB, and so on?
A. Yes, Generally, the performance criteria I use is: Is there any hope of delivering a response time in 2 sec or less? And what I've found is that, if you're paging more than 25% of avail memory, then you're likely not going to be able to achieve that goal. Mem allocations for virtual memory is really a reflection of performance, which is subjective. So the goal around the sub-2-sec response time is really where I draw the line for memory allocations not to exceed 25% more than available installed memory, and this is applicable to Windows and AIX platforms.
Q. How can you increase the size of the Name Lookup Cache in Domino? (also answered in the Web Chat transcript)
A. You can control the Name Lookup Cache via the Notes.ini parameter NLCache_Size=# in bytes. Note, however, that increasing the NLCache will tie up memory that, if you don't really need it for NLCache, then would be better used elsewhere. So, one thing you can look at when issuing a "show stat" is whether you've hit the existing NLCache size limit in bytes and how much you're currently using.
Q. Regarding partitioned servers: What is the addressable space for a Windows 32-bit platform?
A. We're limited in the application space to a 2GB address space for a 32-bit Domino on 32-bit Windows.
Q. For each partitioned server?
A. Yes, but the OS is limited too, so for each shared mem process you have a 2G limit, and for each process in that Domino server, their private process space, which might include parts of the shared memory process space, has a 2GB limit.
Q. What can we learn from the Statrep database with respect to memory usage and our environment?
A. We do report very general information regarding memory usage in the Domino server, for example, the net memory usage number is reported, but there won't be any specific detail about why there was a memory increase from that info. The Statrep database is very useful, for example, for setting general warning thresholds when reaching a certain amount of memory allocated. But if you want to understand if, for example, your NLCache is constrained, Statrep doesn't give you that level of detail. It doesn't give you pool-structure-level memory usage info, so it can be useful at a high level, but it lacks the detail of pool structure usage that you'd need in order to understand where memory is being used in the environment.
Q. We have 64-bit Domino running on 64-bit AIX, and I wonder if the strategy for memory management changes, in that it seems you almost have to rein it in rather than trying to keep it from bumping the threshold that the 32-bit systems had. We've run 32-bit before, and never had paging space, whereas now we do.
A. So you're concerned that Domino memory usage has been greater than planned or expected. I would ask: Is the memory usage "valid"; that is, is it driven by end-user activity, or are we having some type of workload management issue that 's driving higher-than-expected memory usage? And to understand that, the first thing would be to determine how much memory the servers are actually using compared to how many users you have in your environment, and the type of workload you're running, to see if it's in the "ballpark" of normal.
As I said before, workload changes---mailfile size changes, number of docs in Inbox, application db size changes---and all these factors can have massive, manifold increases on Domino's memory-management requirements. Users might not be aware they're putting a burden on the environment because, whereas in the 32-bit environment you would have run into system constraints due to 32-bit limitations, now at 64 bit we're just allocating new memory.
So when you say you're using page space, my first question would be how much memory is Domino actually using? If you ran a "show mem dump", the summary section would show you how much memory Domino is using and how many users you are trying to support. Generally speaking, for a mail server, you should be around between 1.5 and 2 MB of memory per user, so if you've got 3,000 users on your server and your memory usage is much higher than ~ 3GB, that would suggest there's some other workload issues here that are driving up your memory usage. Could be a problem in the server or a problem in workload, which would need to be determined, but I'd start with "Are we "normal?" as a baseline.
FYI: Regarding the NameLookupCache parameter we discussed earlier, if you issue a "show stat" and look at Database.NAMELookupCacheMaxSize, that is the maximum size your server's currently configured to support. And Database.NAMELookupCachePool.Peak tells you how much of the NameLookupCache you've actually used. So, if your peak is nowhere near your maximum, you're not even using the amount of memory that's already allocated to NL pool.
Original publication date