Open Mic Webcast: Preventative Actions and Troubleshooting Common Domino Memory Issues - 30 October 2012 [slides and chat transcript attached]
IBM hosted an Open Mic webcast with IBM Development and Support Engineers on Tuesday, October 30, 2012. The topic was " Preventative Actions and Troubleshooting Common Domino Memory Issues."
This Open Mic will elaborate and answer questions about the information in this wiki article: Preventative Actions and Troubleshooting Common Domino Memory Issues
|Web Chat transcript|
Q. Will there be any Open Mics on Domino/Notes and Citrix?
A. We are just starting to draft our 2013 schedule and will consider Citrix as a possible choice. Open Mics about Citrix seem to be very popular.
FYI from IBM: The slides are available as a PDF in the IBM Support Webcast document #7036159, "Open Mic Webcast: Preventative Actions and Troubleshooting Common Domino Memory Issues - 30 October 2012 [slides attached]."
Q. From where can we get the Lotus Notes Diagnostic Utility (LND)?
A. You can get the LND from the IBM Support Downloads document #4019151, " Lotus Notes Diagnostic utility."
Q. Is it possible to run IBM Sametime on a 64-bit Domino server?
A. Sametime does not yet have a 64-bit product and therefore won't run on a 64-bit Domino server. For more details, refer to the IBM Support Technote #1420335, " 64-bit support for Lotus Sametime 8.5."
FYI from IBM: In Domino 8.5.3, the "Maximum number ... exceeded" message has been modified to inform you that the message is referring to out of handles. Also in 8.5.3, we have increased the max # of handles on all platforms, except 32-bit AIX/Solaris and OS/400, and the LSXBE backend has been changed to provide more available memory.
The increased handle limits exist in 8.5.3 for both 32-bit and 64-bit Domino (just to clarify since the slide could be read as only 64-bit). And actually 64-bit has even more handles, but they did all get handle increases except for 32-bit AIX/Solaris and OS/400 (OS/400 may get the update in a 8.5.3 Fix Pack).
Q. 64-bit is not supported by BlackBerry either, is it?
A. Once RIM has a 64-bit product, it will run on a 64-bit server. Note that there is, in fact, a 64-bit Notes Traveler product.
Q. We have absolutely no live applications running, we just configured and left it, but we have a memory issue on all servers with 4GB of RAM. When we increased the RAM on one server to 16GB, the issue was resolved; however, we still have the issue on all servers with 4GB RAM, and we cannot afford to increase RAM on all servers. We are on Microsoft Windows/longhorn/64 6.0.
We've had PMR #10795,999,866 open with IBM for months but no one is able to fix it. We did almost everything mentioned in the presentation, but no real answer from IBM.
A. When you say that you have memory issues, do you see higher RAM consumption than you expect? Also, do you mean Windows 2008? If so, you should upgrade to 2008 R2 as there were issues with Windows memory management
Q. Yes, we have seen higher RAM consumption than expected. I think HTTP is using more, though nothing is running---no live applications running---and it is on Windows 2008 R2.
A. Thank you for providing your PMR number. We will ensure your PMR gets looked at by the appropriate experts.
Domino 64-bit will scale up many things in memory that were lower on 32-bit, since we were previously heavily constrained by address space, and we can get significant improvements by caching and accessing more data in memory than we would, if we needed to involve the disk for those same operations.
Similarly, the 64-bit OS will consume more memory as it will scale many areas of kernel memory to utilize the increased addressability. The most noticeable increase in OS memory would be the System Cache, which will allow the OS to keep references to many more databases for quicker access when Domino requires it.
As an example, HTTP will have a default of 1 GB for the JVM, whereas on 32-bit it would have been much lower depending on your version (64 MB -- 256 MB). Another example is that threads for the Domino processes will consume more memory on 64-bit Domino than they did on 32-bit. so those can add to additional consumption.
Q. So what would be the fix?
A. It could be multiple things; depending on your requirements, you could constrain memory more in certain areas. For example, lowering the HTTP JVM if you aren't using it since will set aside 1 GB on startup. I haven't seen any data from your server so I can't say definitively, but 4 GB of RAM on a Windows 2008 server running 64-bit Domino is likely going to be very low for a production environment due to the factors I mentioned previously, as well as others.
Again, going with the HTTP process since you used that as an example earlier: The UBM (our shared I/O cache) will have a 1 GB cap on 64-bit Domino, and any additional shared memory will add to the HTTP address space. Then add the 1 GB JVM for HTTP and several hundred MB for the HTTP threads, and even a couple hundred MB additionally for other private memory usage. When you add those things alone and---excluding other possible consumption---you have HTTP over 2.5 GB in address space.
We cannot really work individual PMRs during this live event, but will be sure to look at your PMR when the event concludes.
Q. How large are the increased handle limits in Domino 8.5.3?
A. I don't think we published them but, for example, the private DHANDLE went from 6399 to 131071 and shared DHANDLE went from 64K to 1.6 million. These numbers have a "fudge factor" on 32-bit---they are somewhat less---- but that gives you an idea. There is a factor operation that takes the maximum by 3/4 for 32-bit and 4/4 for 64 -bit. But the factor was there previously for 32-bit with the previous limits, so the ratio change is still 6399 --> 131071.
Note that the limits are different depending on multiple factors, for example, whether we're talking about 32-bit vs. 64-bit Domino, or shared vs. private; or handle vs memhandle. The one that was typically most limiting was the private handle limit that was mentioned (private DHANDLE going from 6399 to 131071).
Q. The "show LDAP pool" command returns a message that the command is not recognized on my 8.5.3 Domino server.
A. The "show ldap pool" command will be available with a special debug hotfix provided by Support when troubleshooting.
Q. Are there any plans to release a 64-bit version of the Linux on SystemX Domino?
A. Yes, our current plan is to ship 64-bit Domino for Linux on x86_64 in our next major release.
Q. We seem to get Semaphore errors more than any other. Can you tell us about them?
A. Prior to 8.5.3 Semaphore Debugging (Debug_Capture_timeout=1 & Debug_Show_Timeout=1) was not enabled by default. As of 8.5.3 it is on by default now.
Having semaphore timeouts/Locks in the console log doesn't always mean there's an issue; it is normal to have some of these during the day. Unless you notice a real performance/hang condition, these should be considered as warnings only. The parameters will, however, create useful debug files if Support is involved.
Semaphore issues mean contention on a resource; you must identify the semaphore and the resource in order to troubleshoot it. Also, in 8.5.3 we turned on the sem timeout message by default; previously you had to turn them on, so that might be it.
LND can help you with semdebug.txt files: Collect an NSD and run ''Automatic process" on the semdebug data.
Q. Is the size of an NSF database a probable issue with memory problems?
A. The size of an NSF could lead to increased memory consumption on your server, but it is really dependent on what is contained in the database and the purpose/use of it. Most commonly we see larger issues with performance accessing large databases, and that will have a greater impact than the additional memory consumption.
For example, larger views take longer to rebuild, meaning that it takes longer for users to get access to them, but they typically don't cause such a great increase in memory that you see issues in that area.
Q. What kind of DDM probes and stats would you recommend to monitor on a regular basis for busy servers?
A. Your choice of Monitoring probes will depend on where the server can be busy, for example, applications (AMgr/Java), mails, replication, or indexing. The IBM Redbooks publication, " Lotus Domino Domain Monitoring," might be of help.
There are also a lot more lectures from previous Lotusphere events in the " Look into using Domino Domain Monitoring" Forum topic.
FYI: The Support Downloads document #4205506, "Download IBM Support Assistant (ISA) Lite for Lotus Notes/Domino" states that ISA Lite be used back to v6.x.
For more information on ISA (including Lite) refer to the IBM Support Assistant web site.
Q. Why do we face memory issues with Domino 8.5.3 servers on Windows 2008 (not R2). Do we have any hotfix for this?
A. I would recommend that you upgrade to R2. There were cache problems in previous versions of Windows 2008. See the Microsoft Developer Network articles, " SetSystemFileCacheSize function" and "CreateFile function" for more details. Also, IBM Technote #1391477, " Domino 32-bit runs poorly on Windows 64-bit, " provides advice on this issue.
Other relevant Technote:
- "Windows 2008 64-bit can cause a significant CPU increase and I/O degradation when Domino opens many databases."