IBM Tivoli Netcool/OMNIbus, Version 7.4

Configuring controlled shutdown of an ObjectServer

You can configure a controlled shutdown of any ObjectServer such that pending changes are forwarded to IDUC clients before the ObjectServer shuts down. This minimizes the possibility of data loss on shutdown.

About this task

To enable controlled shutdown, the ObjectServer schema must be updated with a set of triggers and procedures that are provided in an SQL import file control_shutdown.sql, which is stored in the $NCHOME/omnibus/extensions/control_shutdown directory. The ObjectServer must also be set up to run under process control because the nco_pa_stop utility needs to be called from an external procedure to shut down the ObjectServer.

The triggers and procedures that are provided in the control_shutdown.sql file will orchestrate a controlled shutdown. The ObjectServer is first brought to a restricted state. Connections identified for non-IDUC clients (such as nco_sql and nco_config) are dropped, and an IDUC FLUSH command is initiated to send pending changes to all identified IDUC clients (such as gateways and event lists). Any new connection requests to the ObjectServer are blocked. If store and forward is enabled for probes, any new alerts are stored in a store-and-forward file until the probe can successfully reconnect to an ObjectServer. When the data retrieval is completed for the IDUC clients, the nco_pa_stop utility is used to shut down the ObjectServer process that is running under process control.

Sections of the control_shutdown.sql file must be edited to specify information that is required for setting up the configuration.

To configure and perform a controlled shutdown for an ObjectServer:

Procedure

  1. Go to the $NCHOME/omnibus/extensions/control_shutdown directory, and copy the control_shutdown.sql file to the $NCHOME/omnibus/etc directory, or to another preferred location.
  2. Remove the default read-only permissions from your copy of the control_shutdown.sql file and review the file to familiarize yourself with its contents. Then edit the file as follows:
    • Locate the following section of code for the ext_shutdown procedure:
      -----------------------------------------------------------------------
      -- External procedure to shutdown OS using nco_pa_stop
      ...
      -----------------------------------------------------------------------
      
      create or replace procedure ext_shutdown (in process_name Char(255), 
      in username Char(255), in pass Char(255), in paserver Char(255)) executable '$OMNIHOME/bin/nco_pa_stop' host 'nchost1' user user1 group grp1 arguments ' -process ' + process_name + ' -user ' + username + ' -password ' + pass
      + ' -server ' + paserver go

      Example values are formatted in bold text. Replace the nchost1 placeholder with the name of the host on which you want to run the nco_pa_stop utility to shut down the ObjectServer. Replace user1 with the appropriate user ID and replace grp1 with the group ID under which to run nco_pa_stop. The ext_shutdown procedure is called from the control_shutdown procedure and the final shutdown trigger in the control_shutdown.sql file, so these sections of code also need to be amended.

    • Locate the following section of code for the control_shutdown procedure:
      ------------------------------------------------------------------------
      -- Procedure to drop and flush connections for controlled shutdown
      ------------------------------------------------------------------------
      
      create or replace procedure control_shutdown()
      declare
        iduc_clients int;
      ...
      ...
      ...
       if ( iduc_clients = 0 )
        then
      	-- do nothing , simply shutdown
      	-- call external procedure to shutdown OS using nco_pa_stop
      	-- Replace 'MasterObjectServer' with ObjectServer process name in PA conf file
      	-- Replace 'user1' with username to execute nco_pa_stop.
      	-- Replace 'pass1' with password to execute nco_pa_stop.
      	-- Replace 'AGG_PA' with PA server name to connect.
      	 execute procedure ext_shutdown ( 'MasterObjectServer', 'user1', 'pass1', 'AGG_PA' );
        else
      	-- Enable trigger to check if GET IDUC is finished for all the clients.
      	 execute procedure enable_control_shutdown;
       end if;

      Example values are formatted in bold text. On the execute procedure ext_shutdown line, replace the MasterObjectServer, user1, pass1, and AGG_PA placeholders with the ObjectServer process name defined in the process agent configuration file, the user credentials for running nco_pa_stop, and the name of the process agent that the ObjectServer uses to run the external automation.

    • Locate the following section of code for the final_shutdown trigger:
      ------------------------------------------------------------------------------
      -- Trigger to reset Pending flag on  GET IDUC from all clients
      -----------------------------------------------------------------------------
      
      create or replace trigger final_shutdown
      group control_shutdown_triggers
      ...
      ...
      ...
       if( pending_cnt = 0 ) then
      	-- disable this trigger group and shutdown ObjectServer.
      	 execute procedure disable_control_shutdown;
      	-- call external procedure to shutdown OS using nco_pa_stop
      	-- Replace 'MasterObjectServer' with ObjectServer process name in PA conf file 
      	-- Replace 'user1' with username to execute nco_pa_stop.
      	-- Replace 'pass1' with password to execute nco_pa_stop.
      	-- Replace 'AGG_PA' with PA server name to connect.
      	 execute procedure ext_shutdown ( 'MasterObjectServer', 'user1', 'pass1', 'AGG_PA' );
       end if;

      Example values are formatted in bold text. On the execute procedure ext_shutdown line, replace the MasterObjectServer, user1, pass1, and AGG_PA placeholders with the ObjectServer process name defined in the process agent configuration file, the user credentials for running nco_pa_stop, and the name of the process agent that the ObjectServer uses to run the external automation.

  3. Apply the controlled shutdown customization to a new or existing ObjectServer as follows:
    • When creating the ObjectServer by using the nco_dbinit command, apply the customization to the ObjectServer database:

      $NCHOME/omnibus/bin/nco_dbinit -server server_name -customconfigfile $NCHOME/omnibus/extensions/control_shutdown/control_shutdown.sql

    • If the ObjectServer already exists, apply the customization as follows:

      For UNIX operating systemFor Linux operating system$NCHOME/omnibus/bin/nco_sql -server server_name -user user_name –password password < $NCHOME/omnibus/extensions/control_shutdown/control_shutdown.sql

      For Windows operating system"%NCHOME%\omnibus\bin\isql" -S server_name -U user_name -P password -i "%NCHOME%\omnibus\extensions\control_shutdown\control_shutdown.sql"

    In these commands, server_name is the name of the ObjectServer, user_name is a valid user name used to log on to the ObjectServer, and password is the corresponding password.

  4. Set up the ObjectServer to run under process control. Within the ObjectServer properties file, set the following properties for process control:
    • Set PA.Name to the name of the process agent that the ObjectServer uses to run external automations.
    • Set PA.Username and PA.Password to a valid user name and password combination required for connecting to a process agent to run the ext_shutdown procedure.
  5. Start the process agent that runs the ObjectServer process under process control.
  6. Assuming clients (both IDUC and non-IDUC) have been started within your environment, you can perform a controlled shutdown at any time by running the following SQL commands from the SQL interactive interface:

    execute procedure control_shutdown;
    go

    Note: After flushing all the IDUC clients, the ObjectServer waits for a GET IDUC response from the notified clients. If any IDUC client does not respond, the ObjectServer remains in a restricted state. In this state, only the SQL interactive interface utility (nco_sql) is allowed to connect. You can query the ObjectServer by using nco_sql to check which client is not responding. If a client does not respond within the expected time, you can forcefully disconnect the client and try to execute the control_shutdown procedure again. For example:
    select ConnectionId, AppName, AppDesc from iduc_system.temp_connections where Pending =1 ;
    alter system drop connection 'connectionid';
    execute procedure control_shutdown;
    go

Sample process agent configuration file: AGG_PA.conf

This example shows sample values in a process agent configuration file named AGG_PA.conf, which map to the placeholder values that you need to complete in the control_shutdown.sql file (as discussed in the preceding steps).

The code shows sample values in the AGG_PA.conf file, for the process agent named AGG_PA. In the file, an ObjectServer process called MasterObjectServer has been set up to run the AGG_P ObjectServer under process control. The user credentials user1 and pass1 will be used to connect to AGG_PA in order to run the external procedure ext_shutdown to shut down the ObjectServer, which is running on host nchost1.

====================================
Example Process agent config file  AGG_PA.conf
======================================
#
# Process Agent Daemon Configuration File 1.1
#
#
# List of Processes.
#
nco_process 'MasterObjectServer'
{
        Command '$OMNIHOME/bin/nco_objserv -name AGG_P -pa AGG_PA -pausername user1 -papassword pass1 run as 0
        Host            =       'nchost1'
        Managed         =       True
        RestartMsg      =       '${NAME} running as ${EUID} has been restored on ${HOST}.'
        AlertMsg        =       '${NAME} running as ${EUID} has died on ${HOST}.'
        RetryCount      =       0
        ProcessType     =       PaPA_AWARE
}

#
# List of Services.
#
nco_service 'Core'
{
        ServiceType     =       Master
        ServiceStart    =       Auto
        process 'MasterObjectServer' NONE
}

nco_service 'InactiveProcesses'
{
        ServiceType     =       Non-Master
        ServiceStart    =       Non-Auto
}


#
# Routing Table Entries.
#
# 'user'        -  (optional) only required for secure mode PAD on target host
#                  'user' must be member of UNIX group 'ncoadmin'
# 'password'    -  (optional) only required for secure mode PAD on target host
#                  use nco_pa_crypt to encrypt.
nco_routing
{
        host 'nchost1' 'AGG_PA' 'user1' 'pass1'
}