IBM Support

How many documents will cause a database to show the "Unable To Extend an ID Table - Insufficient Memory" error when replicating or compacting?

Technote (FAQ)


Question

What causes the ""Unable To Extend an ID Table - Insufficient Memory" error to appear? How many documents can an ID Table hold before encountering this error when Replicating or Compacting?

Answer

The limit on documents that an ID table can accommodate is indeterminate. The ID table data structure compresses consecutive runs of NoteIDs (RRVs) into a single entry in fixed-size table. As such, the total number of notes that can be stored in an ID table is dependent on how fragmented the NoteIDs are in the database. Here's where it starts to get confusing, so let me provide an example to illustrate.
Let's say we have a database with the five following NoteIDs:

0x0004
0x0008
0x0010
0x0014
0x001C

NoteIDs are always allocated with an offset of four from the last allocated NoteID. Based on that fact, we know that these are the total possible slots for notes in this database:

0x0004
0x0008 ---> 0x0004 + 4
0x000C ---> 0x0008 + 4
0x0010 ---> 0x000C + 4
0x0014 ---> 0x0010 + 4
0x0018 ---> 0x0014 + 4
0x001C ---> 0x0018 + 4

The unused slots in the database have been greyed out to highlight the concept of "gaps" in the map of allocated RRVs. This is of critical importance for ID tables. To restate the earlier point, an ID table compresses consecutive runs of NoteIDs into a single entry in a fixed-size table.

If we build an ID table from the example notes, the following will occur:

<Example 1: ID table with non-consecutive notes>
Offset, Repeat
0x0004, 1
0x0010, 1
0x001C, 0

Because NoteIDs are always allocated with an offset of four, we can represent consecutive NoteIDs (where each NoteID an offset of four from the previous NoteID) with only two pieces of information:

1. The beginning offset
2. The number of repeats (this means how many times we can add four to the last NoteID and land on another NoteID)

If you look at our Example 1 ID table, we start at offset 0x0004. The next ID is 0x0008. Because this is four greater than 0x0004, we can use the ID table entry for 0x0004 to represent this ID by incrementing the repeat value. The next ID is 0x0010 (remember that 0x000C isn't a note in the database). This is eight greater than 0x0008, so we cannot use a repeat value to represent this document. Therefore, it gets a new entry in the ID table. The next note is 0x0014, which is four greater than 0x0010. Thus we can use a repeat to represent it. The last ID, 0x001C, is eight greater than 0x0014, so it needs a new entry in the table.

Now let's look at the same table if the two unused slots were instead actual documents. The total slots now look like this:

0x0004
0x0008 ---> 0x0004 + 4
0x000C ---> 0x0008 + 4
0x0010 ---> 0x000C + 4
0x0014 ---> 0x0010 + 4
0x0018 ---> 0x0014 + 4
0x001C ---> 0x0018 + 4

Every NoteID is four greater than the last. If we build an ID table from this example, it looks like this:

<Example 2: ID table with consecutive notes>
Offset, Repeat
0x0004, 6

In example 2, every note in the database can be represented by a single entry in the ID table because they are all consecutive. As a result, this table only needs one entry, where the previous non-consecutive example had an ID table that had three entries in it.

The key point of understanding is that databases with a higher concentration of consecutive NoteIDs will have much more efficient ID tables in which single entries represent many IDs. If the database has a highly fragmented map of allocated NoteIDs, then the ID table will be very inefficient, and single entries will represent few IDs. In the worst case, a single entry represents just a single NoteID, and you will exhaust the ID table's memory limitation with relatively few IDs.

Because each database is different, we cannot know how fragmented the customer's map of allocated NoteIDs is, so we have no way of even estimating how many documents they can have before encountering the problem. All we can do are speak to generalities like these:

1. The higher the document count, the greater the chance to encounter problems.
2. Volatile databases (high rate of creates and deletes) are most susceptible to the problem, especially if deletes are random with respect to time.
3. Creating a new database helps by partially reordering the slots for NoteIDs. This can be done by creating a new replica or creating a new copy.

Document information

More support for: IBM Domino
Replication

Software version: 7.0, 8.0, 8.5, 9.0

Operating system(s): Windows

Reference #: 1413921

Modified date: 12 May 2015


Translate this page: