IBM Support

CDS Server Preferencing

Technote (troubleshooting)


CDS Server Preferencing

Resolving the problem

CDS Server Preferencing

A CDS server preferencing feature has been added to DCE for AIX and Solaris. This feature was added in a PTF for AIX DCE 2.2; it appears in the base DCE 2.0 for Solaris product and in the base DCE 3.1 products for both AIX and Solaris.

This preferencing feature involves a change to the CDS client software's algorithm for selecting a CDS server when multiple CDS servers exist in a cell. It's a purely local change on the client, so there are no interoperability problems if some machines in a cell have this patch and others don't.

The README that was shipped with the patch that introduced this feature is as follows:

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

Defect 20308

SYNOPSIS: Need CDS server preferences

This delta introduces a CDS preferencing feature.

The CDS clearinghouse used to service a given request will be chosen based on
a rank which is either calculated by an IP-address-proximity algorithm (as an
approximation of network topology) or read from an optional preference file

An example of the preference file format is:

# Comments are ignored, as are blank lines

# Clearinghouse_name Rank
# Smaller numbers are preferred over larger.
/.../mycell/foo_ch 200 # comments here OK too
/.:/bar_ch 200 # note /.:/ is OK
baz_ch 300 # as are local names
/.../othercell/fnord_ch 10 # and cross-cell names
/.../othercell/zoink_ch # illegal line: no rank; next is OK
/.../othercell/zoink_ch 30
/.:/bar_gda 200 # gdad's are clearinghouses too

There is one line for each clearinghouse whose rank you want to override.
Ranks are 16-bit unsigned integers, in decimal (0-65535) or hex (0x0-0xffff).
Lower numbers are preferred over higher numbers. A rank of 65535 (0xffff)
means that the clearinghouse will not ever be used by this client. Blank
lines and comments ("#" to end-of-line) are permitted. Whitespace is
ignored, with the exception that the rank must appear on the same line as the
associated clearinghouse. The preference file is read once, when cdsadv

The default ranks are:

0 for clearinghouses marked "onLAN" (i.e., for those
clearinghouses for which we receive advertisements or that
answer solicitations; typically these are clearinghouses
that are on the same ethernet segment as the client)
5000 for clearinghouses on the same host as the client
20000 for clearinghouses on the same subnet as the client
30000 for clearinghouses on the same network as the client
40000 for all other clearinghouses

You can use a command like "dcecp -c cdscache show -clearinghouse /.:/foo_ch" to see a clearinghouse's rank.

Specifying a preference value in the prefs file causes the "onLAN" flag to be

Write operations into CDS will still always go to the master clearinghouse
for the relevant directory, regardless of any preferences (even if the
master clearinghouse has a rank of 65535/0xffff). Reads will go to the
most-preferred clearinghouse that stores a copy of the data to be read. If
the preferred clearinghouse goes down, then reads will go to the
next-most-preferred clearinghouse, and so on. The CDS client will
periodically ping the down clearinghouse and will revert back to it when it
comes back up.

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

Some additional points:

  • Prior to this patch, the behavior was that a client would pick a CDS server essentially at random, and would tend to stick with that server until something forced it to go elsewhere. With this patch installed, the default behavior will be that CDS reads will go to the closest available CDS server. You can override that default via the /opt/dcelocal/etc/cds_serv_pref file.
  • Preferencing is done on a per-client basis. Preference values may or may not be different on each client machine. You can choose to override the default on some clients but not on others if you want to.
  • If you install this patch (or any subsequent patch), then you're using the preferencing feature. If you don't create an /opt/dcelocal/etc/cds_serv_pref file on a particular machine, then default server preferences (based on network topology) will be used. You need not create the file if you just want clients to use the closest available server.
  • If two servers are equally preferred, then the client will choose one of them in an unpredictable way and will tend to stick with that one. (The client will not attempt to distribute its reads equally among the equally-preferred servers.)
  • The preferencing feature affects only CDS reads. Writes go to the master, as always. The feature affects only CDS, not security or any other component.
  • To be completely correct I should way "CDS clearinghouse" rather than "CDS server" in this note, since the preferencing feature is implemented in terms of clearinghouses rather than server machines. It's all the same as long as you have one clearinghouse per CDS server, as is normal and recommended. If however you choose to create multiple clearinghouses on a single server machine, then the clearinghouses are treated separately.
  • You can use cdscp show server on CDS server machines or dcecp's cdscache show command on client machines to see read and write counts that will give you a rough idea of which servers are being used. These commands are not part of the patch (they've always been there), but the additional "Rank" field in the output of dcecp cdscache show is new.
  • Suppose server1_ch holds the master copy of /.: and that server2_ch is a replica. Suppose that a client prefers server2_ch over server1_ch (perhaps because the two servers are on different subnets, and the client is on the same subnet as server2). The following can occur:

    % dcecp
    dcecp> dir create /.:/x
    dcecp> dir create /.:/x/y
    Error: Requested entry does not exist

    This is because the second dir create began by looking in /.: for information about /.:/x and didn't find it. It turns out that CDS masters wait up to 60 seconds before transmitting changed data to replicas; so during the 60 seconds after a change is made the replica can legally be out of date. The second dir create above went to server2_ch to learn about /.:/x, and it failed because server2_ch didn't yet know that /.:/x exists.

    There are several possible cures:

          1. Wait 60 seconds between the two create operations

          2. Skulk the parent directory between the two create operations, like this:

            dcecp> dir create /.:/x
            dcecp> dir synch /.:
            dcecp> dir create /.:/x/y
          3. Create the second directory via a dcecp command that explicitly states which clearinghouse to use, like this:

            dcecp> dir create /.:/x
            dcecp> dir create /.:/x/y -clearinghouse /.:/server1_ch
          4. Use the dcecp convenience variable set _s(cds) to force all directory-related commands in this dcecp session to server1_ch, like this:

            dcecp> set _s(cds) /.:/server1_ch
            dcecp> dir create /.:/x
            dcecp> dir create /.:/x/y

            This is probably the easiest thing to do if you're creating a directory tree via dcecp.

          5. Create an /opt/dcelocal/etc/cds_serv_pref file that specifies that the master (server1_ch) is to be preferred. This is probably your best bet, at least temporarily, if you're going to be creating a directory tree via some script or other software that can't be modified. You'll have to stop DCE, setup the file, then restart DCE. You can stop DCE, remove the preferences file, and restart DCE again after you're done creating directories.

        This behavior could also occur prior to implementation of this preferencing feature; it's an expected consequence of CDS replication semantics. It tended not to happen much before, though: Since the first create nudged the client to the master, the reads in the second create also went to the master. In the past, an overwhelming percentage of read requests were directed to the master CDS server at many sites. This is much less likely to happen with the new CDS preferencing feature -- reads should be satisfied from replica servers more often now. However, as in the example above, reads that go to a replica can legally return out-of-date information for a short time (generally around a minute or so after a modification is made).

        If you want to retain something similar to the old behavior, where most read requests were satisfied from the master CDS server and out-of-date replicas were not a problem, then you can set up preference files on your client machines such that the master server is always preferred (assuming you know that your master CDS server can handle the load, of course). If you do this, then read requests will be directed to a replica only if the master CDS server is unavailable. Not to imply that this is necessarily good idea, just mentioning the possibility...

  • Document information

    More support for: Distributed Computing Environment

    Software version: 3.1, 3.2

    Operating system(s): AIX, Solaris

    Reference #: 1112377

    Modified date: 14 July 2009