Mass Configuration Change for Windows/Linux/Unix agents

Answer

This document shows one way to change Windows/Linux/Unix ITM Agent configurations in large numbers with minimal effort. One example is altering the primary/secondary remote TEMS the agent reports to. This document explains how that can be done using Always True situations and action commands. First we discuss environment variables. Next we discuss the Windows environment and in a third section we will discuss the Linux/Unix environment.

Product supported command to make configuration changes.

In the last few years a number of commands have been added which can make some configuration changes. The following is a list of new tacmd functions. If those commands work for you, please use them instead of the ad hoc techniques which follow.

The numerous "tacmd" commands that can be used to update settings remotely from the TEMS:

tacmd configureSystem
http://publib.boulder.ibm.com/infocenter/tivihelp/v61r1/index.jsp?topic=%2Fcom.ibm.itm.doc_6.3%2Fcmdref%2Fconfiguresystem.htm

tacmd setAgentConnection
http://publib.boulder.ibm.com/infocenter/tivihelp/v61r1/index.jsp?topic=%2Fcom.ibm.itm.doc_6.3%2Fcmdref%2Fsetagentconnection.htm

tacmd setOverride
http://publib.boulder.ibm.com/infocenter/tivihelp/v61r1/index.jsp?topic=%2Fcom.ibm.itm.doc_6.3%2Fcmdref%2Fsetoverride.htm

tacmd settrace
http://publib.boulder.ibm.com/infocenter/tivihelp/v61r1/index.jsp?topic=%2Fcom.ibm.itm.doc_6.3%2Fcmdref%2Fsettrace.htm

Configuration is controlled Environment Variables

Agent configuration is controlled by environment variables. Some are set by the configuration process while others are set using information in ITM product documentation or on instructions from IBM support or other information sources. When planning for a mass configuration change, the first step is to determine what the environment variable is, both its form and value.

In all cases, you need to determine the product code for the agent. Since the agent has already been installed, you can view the Portal Client Managed System Status view, looking for the PRODUCT column for the information. For planning, here is a technote with many product codes in case you are uncertain.

Comprehensive List of IBM Tivoli Monitoring: ITM 6.X Product Codes
http://www.ibm.com/support/docview.wss?uid=swg21265222

For Windows, the runtime environment variables are stored in the Windows Registry. The top level registry level is HKEY_LOCAL_MACHINE\SOFTWARE\Candle

The next level down is the product code and then a fixed structure. Here is an example for Windows OS Agent

HKEY_LOCAL_MACHINE\SOFTWARE\Candle\KNT\Ver610\Primary\Environment

and the registry view looks like this



For Linux/Unix agents, the data is defined in several places. In <installdir>/config there is a file xx.ini - where xx is the product code. During agent startup, that xx.ini file is combined with symbolic values resolved using the <installdir>/config/.ConfigData/kxxenv file to create a xx.config file which is included in startup process. For example, there is a line in the Linux OS Agent file lz.ini

CT_CMSLIST=$NETWORKPROTOCOL$:$HOSTNAME$;$HSNETWORKPROTOCOL$$MIRROR$

In lz.config you see the translated results.

nmp180:/opt/IBM/ITM/config # grep -i cmslist lz.config
export CT_CMSLIST='ip.pipe:host1;ip.pipe:host2'

This last shows the form that is needed for the mass change.

Running Commands on multiple servers using action commands

During this process, we will need to run commands on many agents with a single action.
Here is a technote about how to do that.

Running Commands Using Always True ITM situations
http://www-01.ibm.com/support/docview.wss?uid=swg21390488

Another aspect is restarting agents. That can be done en masse - see this technote

Recycling agents using Situation Action commands Including OS Agents
http://www-01.ibm.com/support/docview.wss?uid=swg21440392

Windows ITM Environment Variables

If you are uncertain of the name and/or form of the environment variable, then work on a test system and view the Windows Registry.

Some variables are updated by the configuration process. From MTEMS working on CT_CMSLIST

1) right-click on agent task line
2) Select Reconfigure...
3) In the second dialog box enter the primary and secondary remote TEMS for the agent.
4) OK out and the agent will restart.

For other environment variables the supported way to update is via Edit Variables.

1) right-click on task line
2) Select Advanced
3) Select Edit Variables…
4) click on listbox arrow and Edit or Add the variable.

After the change, review the Windows Registry and record the name and form of the data.

Incidently, the <installdir>\tmaitm6\kxxcma.ini file will also be changed and the following added to the end.

[Override Local Settings]
variable=value

This is only for reference because the windows registry contains the changed data which the agent actually uses. It only affects later manual reconfigurations and can be ignored for the mass change scenario. It should be remembered if you do a manual reconfiguration on a server later on.


Mass Configuration Change on Windows Platform

Experiment with a single server having the agent installed. Do the MTEMS update as needed. Run the regedit program and locate the registry entry involved. When you have found it, do a File/Export... and export that Environment path to a .reg file. Edit the .reg file using Wordpad to a minimal size only including the variable(s) of interest. [Wordpad is needed because the file is in UTF-8 format.] In this test case the file was ntenv.reg and it looked like this after removing most of the settings.

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Candle\KNT\Ver610\Primary\Environment]
"CT_CMSTEST"="ip:pipe:host1;ip.pipe:host2"

The ntenv.reg file needs to be created in all the target environment. A good place to create it in Windows is %CANDLE_HOME%\logs. Here is the command to run in this particular case

cd %CANDLE_HOME%\logs && (echo Windows Registry Editor Version 5.00 && echo [HKEY_LOCAL_MACHINE\SOFTWARE\Candle\KNT\Ver610\Primary\Environment] && echo "CT_CMSTEST"="ip:pipe:host1;ip.pipe:host2") >ntenv.reg && reg import ntenv.reg

The command is 230 characters long. The limit is 506 characters, although the target platform may have other limits. Here is a rundown on how it works

1) Set the current directory:
cd %CANDLE_HOME%\logs &&

2) Put the required contents into the file:
(echo Windows Registry Editor Version 5.00 && echo [HKEY_LOCAL_MACHINE\SOFTWARE\Candle\KNT\Ver610\Primary\Environment] && echo "CT_CMSTEST"="ip:pipe:host1;ip.pipe:host2") >itmenv.reg &&


3) Put the new value into the registry:
reg import itmenv.reg
.

As always, this must be practiced on a test server. Afterwards you can change the distribution to a Managed System List and perform this to a large number at the same time.

Note: On a 64 bit Windows system the registry folder for product settings was reported as

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Candle

However as always you should examine the registry before and after an install to determine what is needed in your environment.


Linux/Unix ITM Environment Variables

If you are uncertain of the name and/or form of the environment variable, then work on a test server and view the config file. Some variables are updated during a configure (itmcmd config -A xx) or via the gui (itmcmd manage &). After the configuration, start and stop the agent and view the xx.config file update. For the test case of CT_CMSLIST on the Linux OS Agent the results are

export CT_CMSLIST='ip.pipe:host1;ip.pipe:host2'

For non-configuration variables,they are usually added to the xx.ini file having the form

variable=value

After the start process recreates the xx.config file, the value will usually be

variable='value'

This is because the xx.config is included into a Korn shell process and that is what is expected. The main reason for this method is to skip the java processing which creates the xx.config file. The override file is included directly with no processing. Sometimes an export is needed as prefix. That will either be given to you by documentation/instructions or you may have to experiment. That is needed if the environment variable is needed by a separate Linux/Unix process which is spawned from the initial process.


Mass Configuration Change on Linux/Unix Platform

The following technique will work after agent maintenance is performed.

The general idea is to create a file on $CANDLEHOME/config with the override environment variable(s). Then an update is made to the ini file to cause the override to be logically included from the config file. All the following commands are issued via the Always True situations. This example shows a single override file. If possible, it could be used for all agents installed on the server. If that is impossible, you could have a separate file per override, say xxenv.ovr. The first command required for this example is:

echo export CT_CMSLIST='ip.pipe:host1;ip.pipe:host2' > $CANDLEHOME/config/override

Where you are setting the primary and secondary remote TEMS's. Incidentally you can have more than two targets if needed. The second command is

echo . $CANDLEHOME/config/override >>lz.ini

These commands are processed by the Linux/Unix command system() function and some values have special meaning. For example double quote will need to be prefixed with a escape character, like this \". Be sure to do lots of testing on a test server. Another example is

echo export KDC_FAMILIES='ip.pipe port:1918 SKIP:15 COUNT:1 use:y ip use:n sna use:n' > $CANDLEHOME/config/override

Which would be used to force a particular agent listening port usage. Long term, you might have more than one setting in the override file like this:

(echo export KDC_FAMILIES='ip.pipe port:1918 SKIP:15 COUNT:1 use:y ip use:n sna use:n' && echo CT_CMSLIST='ip.pipe:host1;ip.pipe:host2')> $CANDLEHOME/config/override

After thoroughly testing the result you can change the Always True distribution to run on a full Managed System list of agents.

The override file must have file permissions and ownership that allow it be read, During initial testing, determine what the file permission and ownership of the ini file is. If this change is needed, the creation command would look like this:

echo export ... > $CANDLEHOME/config/override ; chmod nnn $CANDLEHOME/config/override ; chown owner:group $CANDLEHOME/config/override

The semicolons allow you to run commands in sequence. Again, serious testing is required on a test server of each type before initiating a mass configuration.

Large Environment considerations - checking on which agents have been updated

In a large dynamic environment, you may not be able to update all agents at the same time. Because of that, a way is needed to determine what agents have not been changed. In the above commands you run them using an Always True situation.

For Linux/Unix, have one situation to detect if the override file is missing

( Path == '/opt/IBM/ITM/config' AND MISSING(File) == ('override'))

An action command could collect the originnodes in a list at the TEMS - if there are many of them. The current size in bytes is not available, so for cases where it already exists, determine a checksum by first doing this on a known server:

( Path == '/opt/IBM/ITM/config' AND File == 'override' AND Checksum Algorithm == CRC32 AND Checksum != 0)

The alert will tell you the actual checksum. In my test case it was 2863812960. Next the following situation can be used across all servers to determine the overrides which are not at the correct level:

( Path == '/opt/IBM/ITM/config' AND File == 'override' AND Checksum Algorithm == CRC32 AND Checksum != 2863812960)

For Windows, you could use the xxenv.reg file as the current state. Using the NT File Trend attribute group the missing is logically the same:

( Watch Directory == 'c:ibm\itm\logs' AND MISSING(Watch File) == ( 'ntenv.reg' ))

And then use:

( Watch Directory == 'c:ibm\itm\logs' AND Watch File == ntenv.reg)

To determine the correct value for Current Size, do something like this:

( Watch Directory == 'c:ibm\itm\logs' AND Watch File == ntenv.reg AND Current Size (Superseded) != 193)

Which will alert when the file is not the correct size. With a combination of the missing file and the checksum or size check, you can determine which agents have not yet been updated.