Update on SPR CSCT836HFL - LotusScript and Java functions can return "The collection has become invalid"

Certain LotusScript and Java view functions like GetDocumentByKey and GetAllDocumentsByKey now return "The collection has become invalid" when accessing frequently updated views. This is a change from prior releases where your code would wait until the view was up to date and then return the most current information. Note: The change was introduced in May of 2009 (7.0.4), July of 2009 (8.0.2 Fix Pack 2, 8.5 Fix Pack 1), and October of 2009 (8.5.1).

What's the current status of this issue?

A fix is available in the following releases: 8.5.2, 8.5.1 Fix Pack 3, 8.0.2 Fix Pack 6, and 7.0.4 Fix Pack 2.

In this technote, originally published on March 12th, 2010, customers were alerted to the availability of a hotfix for this issue (SPR CSCT836HFL/APAR LO49368). The issue manifests in custom Notes applications with agents that access view collections and can be diagnosed by the existence of the following errors in the Domino server log:

-- "Error 4000: %a's Certification Log" (in 7.0.4, 8.0.2 FP2 and 8.5 FP1)
-- "Error 4678: The collection has become invalid" (in 8.5.1 and 8.0.2 FP3)

What's changed?

The hotfixes developed in March were recalled in April. A fix is available in 8.5.1 Fix Pack 3 and will be available in additional Fix Packs shortly (see below for detailed schedule). Updated hotfixes are available for customers unable to wait for a Fix Pack.


No action on your part is required if:

You are not currently impacted by this issue and you have not previously applied a hotfix for SPR CSCT836HFL.


Action on your part is recommended if:

You are encountering one of the errors described in this technote for SPR CSCT836HFL.

Recommended Actions:

• If possible, IBM recommends applying a Fix Pack with the fix. This could mean waiting for the availability of a fix in a Fix Pack or Maintenance Release (see below).
• If you need the fix sooner, and a Fix Pack is not available for your release, request the new hotfix.

Which Maintenance Release and Fix Packs contain the fix es ?

• 8.5.2 Maintenance Release
• 8.5.1 Fix Pack 3
• 8.0.2 Fix Pack 6
• 7.0.4 Fix Pack 2

How can you determine if you are encountering this issue and need the fixes?

In releases prior to those cited below, your server would hang. In the releases cited below, your log will display one of the following errors:

-- "Error 4000: %a's Certification Log" (in 7.0.4, 8.0.2 FP2 and 8.5 FP1)
-- "Error 4678: The collection has become invalid" (in 8.5.1 and 8.0.2 FP3)


What factors increase the likelihood of encountering this issue?

In general, any activity that modifies a view collection during programmatic access to that view. Here are more details on what can modify the collection in a fashion that could increase the chance of encountering the problem:

• Views which allow entry insertions anywhere instead of at the end
• Views which allow entry deletions while query applications are running.
• Views which allow editing of summary value fields included in a View Selection Formula (may cause entries to disappear from the View temporarily while the view is updated).
• Applications which self-modify the view they are reading with insertions, deletions, and modification of summary value fields.
• Views experiencing heavy insertion and deletion traffic at application run times.
• Applications that run View.getDocumentByKey/getAllDocumentsByKey operations while updates are occurring.
• Any installation which sets the server time ahead for any reason, even temporarily.

What factors decrease the likelihood of encountering this issue?

• Perform query applications on Views sorted by date ascending, so all insertions go in at the end.
• Run queries during low/no update times.
• Mark entries for removal and do housecleaning during low use times.

How the new fixes work:

For local applications (those that are found on the same server where the agent is located), the fix behavior is as follows: The GetDocumentByKey and GetAllDocumentsByKey calls check first if the desired view is current. If the view is not current, the view is brought up to date and the desired content is retrieved from the view. If additional view updates are made during the LScript call, these changes are queued up until the necessary data has been retrieved by the application.

For remote applications (those that are found on a different server than where the agent is running), the previous looping behavior remains, with the change that the calls will try 500 times before returning an error (as opposed to 10 times as it was before).

In the rare event that the error returns after looping 500 times, a notes.ini tuning parameter allows the looping number to be modified.

MAX_VIEW_GETBYKEY_RETRY=#

where # is a value between 1 and 2,000,000,000.

Customers who experience the "Collection has become invalid" error after applying the fix should try increasing the notes.ini parameter in increments of 1000. It is not possible to disable all looping.

It is the intent to create the same behavior with the remote case as with the local case in the future. This requires more work than IBM can release in a hotfix or Fix Pack. This work is scheduled for a Maintenance Release post 8.5.2 availability.


History

When LotusScript function calls that allowed accessing a view to retrieve information were introduced into Notes/Domino, these calls were designed in a way to ensure the latest information was being retrieved from the view. After the data was retrieved, if the view collection had been updated, these function calls automatically discarded the results and repeated the operation.

There were two flaws with the original approach:

1. It was not optimal for performance.
2. It did not allow for exiting if a view that was never brought current (most likely in highly active views or views with future time/date values) and therefore could cause a hang condition.

To address this "hang" issue, code modifications were made in 7.0.4, 8.0.2 FP2, 8.5 FP1, and 8.5.1. The code change put a limit to the number of times these functions were allowed to discard and re-retrieve current view information. This fix, however, was flawed in that if the threshold was exceeded, instead of returning the current view information, an error was returned. To compound matters, the earliest fix for this issue returned the wrong error message: "Error 4000: %a's Certification Log". This error was changed in 8.0.2 FP3 and 8.5.1 to: "The collection has become invalid". Changing the error did not address the root issue of some applications failing to return the expected data.

The new fix uses a few approaches to both avoid a hang condition and reduce significantly the chance that a customer's application will return an error instead of the desired results.