IBM Support

Tips for Enhancing and Troubleshooting Notes 6 Client Performance

Education


Abstract


The purpose of this document is to emphasize performance enhancements included in the Notes 6 client, to provide you with tips to enhance Notes 6 client performance, to provide you with a list of known issues, and, finally, to provide you with useful tips for troubleshooting issues that may arise with the Notes 6 client. Note: While this document was written specifically for Notes 6.x, the tips for enhancing and troubleshooting client performance apply also to Notes 7.0.

Content


Tips for Enhancing and Troubleshooting Notes 6 Client Performance

by

Mike Cregger, Ken Green and Jonathan Sir Hendrey; Software Engineers on the World Wide Support Engineering Team



Welcome
Thank you for your interest in how to maximize the performance of the Notes 6 client! Notes 6 is the premier messaging and collaboration rich client for the Domino 6 server. There are several enhancements in the Notes 6 client to increase the power and performance over the already powerful Notes 5 client. The purpose of this document is to emphasize performance enhancements included in the Notes 6 client, to provide you with tips to maximize Notes 6 client performance, to provide you with a list of known issues, and, finally, to provide you with useful tips for troubleshooting issues that may arise with the Notes 6 client.

Note: While this document was written specifically for Notes 6.x, the tips for enhancing and troubleshooting client performance apply also to Notes 7.0.

What's New in Notes 6?
Building on the success of the Notes 5 client, there were many enhancements and new features added to Notes 6; all of which focused on achieving the following goals:
  • Improve ease of use
  • Remain the premier mail, calendar and scheduling, and task management client
  • Strengthen replication and mobility features
  • Maintain focus on collaboration
  • Enhance performance
  • Streamline deployment and administration

To read more about how Notes 6 reaches each of these goals, refer to the article (available on the IBM developerWorks Web site) titled “Lotus Notes 6 Technical Overview”. Here is an excerpt from that article that pertains to enhanced Notes 6 performance:

Performance
The most noticeable performance improvement for Notes 6 is that many features are now "multitasking," which means you can do something else while the Notes client is busy performing an operation in the background. Activities such as polling for new mail, detaching files, replicating, printing, monitoring alarms, subscription processing, running background agents, replacing designs, and copying databases take advantage of the new Notes 6 multitasking abilities.

Other Notes 6 client performance improvements include:

· Reduced use of both memory and graphics objects
· An improved compression algorithm of which Notes network data (NRPC) can now take advantage
· More end-user control concerning selective replication and management of file attachments
· Increased file detachment speed

For each Notes 6 user to realize the improvements that increase Notes 6 client performance, there is a price. The changes made to the Notes 6 client will make the operating system, PC hardware, network and Domino 6 server work harder to keep pace with it. Remember, as you continue reading, that there are many pieces that must work together for each Notes user to have a successful Notes 6 client experience.


What to Expect?
From the perspective of performance, users will find a better and more consistent experience with the Notes 6 client. Fewer interruptions will be experienced from the standpoint of the Notes users' interface. In a heterogeneous environment, there are faster response times from the Domino 6 server and fewer transactions with the Domino 6 server. For information on the performance improvement available in a complete Notes/Domino 6 environment, refer to the IBM developerWorks article titled “Domino 6 performance features”.

In the real world, we know that every user does not have leading edge technology on their desktop and that users are connected to dissimilar networks. Notes 6 does, however, require additional resources to take advantage of the new performance features. More memory and CPU power is strongly recommended in order to better handle the multiple threads running simultaneously as well as the many other new features of the Notes 6 client. In addition, network devices must be able to handle the workload of several background activities communicating concurrently with the Domino 6 server.

For Notes/Domino 6 benchmark information, refer to the IBM developerWorks article titled "Domino 6 performs!".


Installation and Configuration Tips
When installing or upgrading to Notes 6, there are a number of guidelines that can be used to optimize both the installation and overall performance of the client.
  • Whenever possible, use the latest version of the Notes client. IBM Lotus continuously incorporates code improvements into the latest release. For related information, refer to the following Web sites:

    Notes/Domino Fix List
    Downloads: Lotus Notes and Domino

  • Upgraded mail templates will increase in size by approximately 10MB.
  • After installing Notes 6, you should verify that there are no constrained resources on the workstation. In general, 15 - 20% of the hard disk on a workstation should be available as free space to maintain reasonable performance.
  • The minimum amount of memory required for Notes 6 to function is 256MB; however, in order to run Notes 6 along with other applications on the workstation effectively 512MB is recommended.
  • Due to increased multitasking functionality in Notes 6, administrators may find that network and local resource usage increases after upgrading. The exact nature of that increase will be dependent on the typical operations Notes is performing. For example, replication and messaging are common areas where multitasking is used more frequently.
  • Notes 6 introduces "policies", which is a mechanism that simplifies the centralized administration of Notes 6 clients. Using policies, many client settings such as security, bookmarks and encoding, can be set from the Domino Directory. This feature enhances and can replace earlier mechanisms such as the user setup profile, though they are retained in Notes/Domino 6 for backwards compatibility.
  • It is not necessary to uninstall earlier versions of Notes prior to installing Notes 6. In specific circumstances, there was a possibility of old files being retained by the installer during an upgrade of earlier versions of Notes; with the Windows Installer used by Notes 6, that possibility is eliminated. Note that, however, you should continue to create a backup of custom template designs and user-specific databases prior to upgrading.
  • Upgrading mailfiles is often a significant part of the Notes upgrade process. A new feature in Notes 6, Seamless Mail Upgrade, provides the ability to administrators to upgrade mail file templates automatically. When this feature is enabled, the design of users' mailfiles will be upgraded the first time the Notes 6 client is started.
  • IBM Lotus recommends that you upgrade Domino servers before Notes clients.
  • The On-Disk Structure (ODS) for databases has changed with Domino 6. The latest ODS version, 43, is backwards compatible with Notes 4 and 5 clients accessing databases on a Domino 6 server; however, Notes 4 and 5 clients cannot access local ODS 43 databases.


Maintenance Tips
  • Notes is highly dependent on the good working condition of the operating system it is hosted on as well as interactions with other installed programs. You should confirm that the operating system is patched to the latest level recommended by the vendor. If any anti-virus programs are installed on the workstation, we also recommend checking for any updates for that application as well.
  • The replication interval of local mailfiles should be 15 minutes or higher. More frequent replication intervals may cause excess replication traffic on the network, which can prevent other operations from completing quickly.
  • Local mail databases require periodic compaction depending on their level of activity. Large amounts of unused space in local databases can lead to low workstation disk space and reduced responsiveness in messaging operations.
  • In some cases, a large client cache can result in a general degradation in performance. In Notes 6, the cache file has been renamed from CACHE.DSK to CACHE.NDK. To clear the cache, close the Notes client and rename the CACHE.NDK. When you restart the client, a new CACHE.NDK will be created.


Troubleshooting
Given the complexity of the Notes 6 client and its interaction with the Domino 6 server, identifying the root cause of performance issues can be a daunting task.

Start here!
Look through this table of known issues to see if one matches the issue that you have encountered in your Notes/Domino environment. If you do not find a match, or, if you find a match however the provided solution does not resolve the issue, proceed to the "How to Diagnose/Debug Client Performance Issues" section for further direction.



Problem

Symptoms

SPR #

Technote #

Scrolling Through Folders Is Slow Using Notes 6.x Client

Users notice delay when attempting drag-and-drop operations onto nested folders.

KTOT62FHFT

1208921

Switching Between Views in Notes 6.x Client Is Very Slow vs. in Notes 5.x Client

Slow response time when switching between views that large text strings in columns.

JEIN5TXNHR

1162303

Severe Performance slow down on IOCP on Endp_AcquireToken when connection is closed

Daily performance degradation, slowdowns, and hangs on Domino 6.x servers running on Unix.

RGET5Q5TJL

1176238

Severe Performance Issues with Long Held Lock Manager Locks on clubusy.nsf

Various performance issues arise on clustered servers with a large number of users.

RGET67JUVB


Error: "...Maximum Allowable Documents Exceeded for Temporary Index"

Full text error messages appear in Notes Log.

n/a

1093466

What Does Error "This is extremely inefficient" Mean?

Error appears when attempting to perform a full-text search on databases that are not full-text indexed.

n/a


Notes 6.x Intermittently Hangs when Users Attempt to Open Messages and receive error: "Remote System No Longer Responding"

Notes 6.x client hangs when users attempt to open messages in server-based mailfile.

n/a

n/a

Notes 6.x Client Pages Out After 60 Seconds of Inactivity

Memory usage of NLNOTES.EXE drops considerably after brief period of client inactivity.

IDEA5VSS27

n/a

Replication Error on Server Console: "ATTEMPT TO ACCESS DATABASE %d by%u Was Denied"

Server slowdown during replication when user does not have access to protected directory links.

MKRN628M5V

1179115

File Listing on Files Tab of Administrator Client Takes a Long Time to Display

Slow response when accessing the Files tab of the Domino Administrator client; unnecessary subdirectory types are retrieved.

TGUZ5YAQP4

1180442

How Can the Nagle Algorithm Be Disabled?

Parameter to disable automatic concatenation of small buffer messages

n/a

1192365

Chronos: Error Full Text Indexing <mailfile.nsf>: Message Queue Is Full

"Chronos: Error Full Text Indexing..." appears repeatedly on the server console and in the Notes Log.

BSPR65FJ2R


How to Diagnose/Debug Notes Client Performance Issues
When you encounter a Notes 6 client performance issue, it is important to distinguish whether the cause of the performance issue actually exists on the client, on the server, or elsewhere within the infrastructure. It is also very important to gather a clear and concise description of the issue (e.g., where, when, and to whom the issue occurs, etc.). Having a thorough understanding of the issue will make it easier to determine the type of troubleshooting that needs to be done, which can include application-level debugging, network-level debugging, or another approach.

Here are several questions that can help you gather the information necessary to narrow down specifics of the problem:
  • What are the users doing when they notice the problem?
  • What are the keystrokes the user makes directly leading up to the occurrence of the issue?
  • Does the issue occur primarily when the user accesses a particular application/database?
  • What is the reproducibility of the issue? Can you reproduce the issue on a clean workstation?
  • Is the issue load related? Does the problem usually occur during peak times?
  • Is this a non-returning hang, delay, or loop?
  • If this is a delay, how long is the delay (number of minutes/seconds).
  • Does an error or status message display in a dialog box, the client status bar, or the local or server NOTES.LOG?
  • What is the result of pressing the Ctrl + Break keys during the problem? Can they continue working or does the problem persist?
  • Are there any intermediary servers, routers, or firewalls between the affected client and server? If so, does the issue happen when the client is directly connected to the server?
  • Does it matter where the client is located;such as, on a particular network segment, LAN, or WAN?
  • When was the first known occurrence of the issue?
  • Was the issue first reported after an upgrade?
  • What is the scope of the issue?
  • Does the problem follow the user to another machine?
  • If this happens on only one or a few machines, you should review and troubleshoot each workstation and user habits. For example, does the issue occur when the user performs particular tasks such as switching between views or after leaving the client idle?


Tools for Troubleshooting
Here is a list of some tools often used for troubleshooting Notes client issues. Some of the tools listed can be used on both the client and server. Following this list, there is a description of how some of the tools can be used to actually troubleshoot Notes client performance issues.

Notes System Diagnostic (NSD)
One of the primary data collection tools for Notes/Domino is the NSD utility. NSD runs on both the Notes client and the Domino server automatically; however, sometimes troubleshooting may require that you run the NSD utility manually. This can help determine certain problems during Notes client hang or performance issues. It can also be used to capture snapshots of the process stacks to help identify the activity during the occurrence.

Notes/Domino Logging
Logging is used for monitoring and troubleshooting problems on both the client and the server. As there are too many logging options to list them all, we have mentioned just a few that are commonly used for troubleshooting certain types of performance issues.
  • Network level debug:

    LOG_SESSIONS=1
    LOG_CONNECTIONS=1
    DEBUG_TCP_ALL

  • Performance problems when performing a particular task or function:

    CLIENT_CLOCK=1

  • Agent/Application logging and debugging:

    LOG_AGENTMANAGER=1
    DEBUG_AMGR=*

  • View/Indexing logging:

    LOG_UPDATE=1
    LOG_VIEW_EVENTS=1

Domino Semaphore Debugging
Notes/Domino uses semaphores and a lock manager (software flags/locks) to ensure that certain tasks complete before others can begin. Many times performance issues occur because a process has a semaphore locked, causing other processes to back up while waiting for access to something. Enabling semaphore debugging is very common in the early stages of working performance issues. NOTE: Semaphore timeouts do not always indicate a performance problem. It is not uncommon for semaphore timeouts to occur on a busy server. For related information, refer to Optimizing server performance: Semaphores (Part 1).

Domino Statistics
Domino can generate server and server-platform statistics. For related information, refer to the Domino Administrator Help topic "Statistics and the Domino System".

Operating-System Level Performance Monitoring Tools
  • Windows NT Performance Monitor (Perfmon)
    This tool can be used to monitor CPU, disk, memory, and network performance. For additional information on this tool, refer to the following:
    Improving Web site performance
    www.microsoft.com
  • Process Monitor
    Process Monitor shows real-time file system, Registry and process/thread activity. Available on versions of Windows starting with Windows 2000 SP4, Windows XP SP2, Windows Server 2003 SP1, and Windows Vista.
  • iostat
    The iostat command reports Central Processing Unit (CPU) statistics and input/output statistics for the entire system, adapters, tty devices, disks, and CD-ROMs.
  • vmstat
    The vmstat command reports virtual memory statistics.
  • truss
    A Unix tool, the truss command is used to trace the system/library calls (not user calls) and signals made/received by a new or existing process. Use strace on some linuxes.
  • tprof
    A Unix tool, the tprof command reports CPU usage for individual programs and the system as a whole. This tool can help determine which sections of a program are most heavily using the CPU.
  • Intel VTune
    This tool can be used for profiling system/library calls on Windows. For more information, refer to Intel® VTune™.
  • Network Sniffers
    Network sniffers are diagnostic software applications that provide protocol-level analysis of data flowing through a network, packet-by-packet. This information can help administrators monitor and identify performance problems.
Troubleshooting scenarios:
Depending on the type of problem, there are different approaches that can be used to trouble shoot an issue and to collect related data. Below are examples of troubleshooting that may be applicable if users report slow client performance.
  • One of the first things usually done for performance issues, is to enable semaphore logging and collect some initial data. Normal sets of data to collect at this point are the console log, semdebug.txt files, and NSDs. We typically request a series of NSDs (or stack dumps) to be taken during the performance problem. Taking sequential NSDs (or stack dumps on Windows) will provide a picture of what the processes are doing. If semaphore timeouts are occurring, this data is correlated to see the particular processes/threads that are locking the semaphores.
  • Many times, third-party software/hardware is involved. If you encounter a performance issue, you should consider disabling or removing third-party software/hardware and then re-testing the issue. This is a good practice just to ensure that some other non-Domino/Notes application is not involved. This includes software such as Notes add-ins, anti-virus software, and backup/archival software. If you re-test without these and the problem does NOT occur, IBM Lotus can help work with the third-party vendor to correct the problem.
  • If it appears that the slowdown may be related to database/application indexing and accessing views, we would typically start by enabling the following logging:
    LOG_UPDATE=1
    LOG_VIEW_EVENTS=1
  • The Update task cycles through the databases on the server every 15 minutes updating views. If you find that view updates take longer than only a few seconds to complete, that would indicate something to investigate further. Additionally, we should see any highly used database being indexed every 15 minutes during the day. However, if it takes longer to cycle around to a particular database, this can indicate that the indexer was most likely still busy with the previous cycle of indexing databases.
  • Client clock debug is requested many times to see what the NRPC communication looks like between the client and the server. This is enabled in the client NOTES.INI and the output is written to the CONSOLE.LOG for the client.
    CLIENT_CLOCK=1
    CONSOLE_LOG_ENABLED=1
    This is mostly used to help narrow down what NRPC functions are taking a longer time, although some inherently take longer than others. This debug is not used to define the problem, as there are many factors involved. Instead, client clock data may help direct the troubleshooting effort towards another area; such as, authentication taking long, opening views taking long, updating a note taking long, an error occurs at a particular point of the conversation, etc. Based on this, additional logging or debugging will probably be required. Again, you should not start troubleshooting by just collecting client clock data, and say that it indicates a problem; it is used to help guide the troubleshooting effort. Client_clock is best used when there is a defined task the user does which exhibits the performance problem and the data can be captured for that particular task. Then, if there are other users or times where the problem does not occur, capture the data when the user performs the same task and compare. Client clock data is rather verbose, so try to catch the problem with the minimal amount of steps.
  • If you are receiving “Server not responding” or “Network operation did not complete” type messages, then you may want to look towards enabling network level debug/logging. If timing out, Client clock can be used to see at what point in the process it occurs. Connection logging and session logging can be used to confirm that the connection has not been closed on one side (server or client) but not the other. TCP debug in Domino can sometimes identify TCP level errors that are occurring. Errors will occasionally occur, so you should look for errors that occur repeatedly.
    LOG_SESSIONS=1
    LOG_CONNECTIONS=1
    TCP_DEBUG_ALL=1
  • Perfmon data can be used to look at the TCP objects, and see if connections are being dropped repeatedly. Sniffer data can also be used to get a closer look at the TCP level communications. Tools like Ethereal provide some built-in analysis. This is very low level debug that can be useful to determine where and when connections are being dropped.
  • Another simple network level test is to look for TCP level packet loss or fragmentation. This is discussed in the document titled "How to Ping by Packet Size to Establish MTU" (#1086718). A simple test can be done to repeatedly ping the server from the client for a period of time to monitor how often packet loss occurs.
  • Then, there is the whole area of application-level performance tuning. One of the early questions above asked was: What are the users doing when they notice the problem? This is particularly important if the problem usually occurs when using a particular application. In this area, you would get more into looking at the particular databases, agents, and views that are being used when the performance is worst. Logging and additional debug can be enabled for views and agents. Custom debugging can be added to the application to see what is taking longest. For related information, refer to the following articles:

Final Remarks
This document has tried to make performance problem analysis a more exacting science. There are many different pieces that all must come together and perform to provide each Notes 6 user with the experience they expect. In many ways the Notes 6 client is like a fine watch. There are many moving parts, but when one goes out of sync with the other parts it can lead to poor performance. You have been given guidelines to keep all the parts of the Notes 6 client synchronized. There is a list of known issues to help solve some problems you may encounter; and there are steps which describe the information IBM Lotus Support needs to help resolve your problem as quickly as possible. IBM wants every customer to be 100% satisfied with the Notes 6 client and beyond.

Original publication date

2005/6/23

Document information

More support for: IBM Notes

Software version: 6.5, 7.0

Operating system(s): Windows, iOS

Reference #: 7006235

Modified date: 21 October 2009


Translate this page: