InfoSphere Information Server 8.5 Logging Subsystem Changes (Log Helper, Logging Agent, and log purging)

Answer

In InfoSphere Information Server, Version 8.5, several changes were made to the Logging Subsystem:

InfoSphere DataStage Administration Client

• Option to enable or disable Operational Repository (OR) logging on a per project basis.
• Options to limit the number and type of log events that are logged to the OR, per job run.

These features are described in InfoSphere DataStage Administration Client documentation, but not described here.

Log Helper, Logging Agent, and purging logs

These features are described in the present technote.

• Log Helper’s usage of temporary, intermediate log files for processing log events generated by an InfoSphere DataStage job.
• Option to set a maximum log file size and create a rotating set of historical files for the Logging Agent log files.
• Events that are purged by using the InfoSphere DataStage Director client are purged in both the RT and OR databases.
• Events that are purged by using the InfoSphere Information Server Web Console or InfoSphere Information Server Console are purged only in the OR database.

Note: The present technote applies only to Version 8.5 and reflects changes after Version 8.1. For Version 8.1-specific information, see DataStage logging changes introduced at 8.1 can impact performance and can also result in missing log entries.

Log Helper new features

When DataStage is configured for OR logging, log messages from running DataStage jobs are written to the Operational Repository using the Log Helper. In Version 8.1 and earlier, the Log Helper would connect to the Logging Agent to log events. However, if the Logging Agent, Logging Service, or Application Server was not available, the Log Helper would fail and subsequently cause the job to fail.

To remove this dependency, now in Version 8.5, the Log Helper queues log events to intermediate log files on the disk. The Logging Agent reads events from an intermediate log event file and sends the events to the Logging Service to insert them into the OR. Finally, after all events have been logged, the intermediate log event file is deleted from the disk.

The intermediate log event files are written to the <ISInstallDir>/ASBNode/logs directory, by default. You can change the location by modifying the logFilePath property, as described in the “Configuring Logging Agent properties” section, below.

Possible performance implications:

• OR log event latency and engine disk usage can be affected, depending on how many events a job logs, and how many jobs are running simultaneously.
• Because log events are first queued to the disk, there is some delay before the events are available for viewing in InfoSphere Information Server Web console. (Note: Viewing DataStage events using the Director Client will not be affected.)
• Engine disk usage can be affected in terms of storage space and I/O activity.

Logging Agent new features

The Logging Agent logs informational, status and error messages to its own log files. In Version 8.5, the Logging Agent log files are maintained as a rotating historical set. You can set optional properties to specify the maximum size of each file and the maximum number of historical files to maintain, as described in the “Configuring Logging Agent properties” section, below.

Also, the Logging Agent log files were renamed in Version 8.5, as follows:

Log file names in Version 8.1	Log file names in Version 8.5
logging-agent.out logging-agent.err	logging-agent-N.out logging-agent-N.err Note: Where N is the generation number (0-19 for the default limit of 20 files).

Configuring Logging Agent properties by using the RegistrationCommand script

The values for the following properties can be changed by using the RegistrationCommand script which is in ASBNode/bin (property names are case sensitive):

logFilePath=<ISInstallDir>/ASBNode/logs

Type a fully qualified directory where the intermediate log files write to. The InfoSphere DataStage primary user group (dstage) must have read and write permissions to this directory. The default value for this directory is <ISInstallDir>/ASBNode/logs.

agentLogLimit=1000000

Optional. Set the maximum size of Logging Agent log files to be retained (maximum size is 1 MB by default).

agentLogCount=20

Optional. Set the maximum number of Logging Agent log files to be retained (maximum count is 20 files by default)/

Syntax:
RegistrationCommand.<sh|bat> -setproperty <propertyName> 
-value <propertyValue> -app LoggingAgent -user <isAdminUserid> -pw <password> 

Where <propertyName> is one of the property names listed above, <propertyValue> is the value you want to set, and <isAdminUserid> and <password> are the credentials for a suite administrator.

Example:

To set the directory for the log files on a Microsoft Windows system, run the command:
RegistrationCommand.bat -setproperty logFilePath 
-value "d:\InformationServer\LoggingAgent\logs” -app LoggingAgent
-user <isAdminUserid > -pw <password>

You can use a similar invocation of RegistrationCommand to change the values of the log file properties, agentLogLimit and agentLogCount.

You must restart the Logging Agent for the new properties to take affect. See "Starting services" in the IBM InfoSphere Information Server Administration Guide:

• Starting services (Windows)
• Starting services (Linux, UNIX)

Log Purging

In Version 8.1, InfoSphere DataStage log events are written either to the RT database or the OR database (before Version 8.1 there was no option to write events to the OR database). In Version 8.5, the Infosphere DataStage Director always logs/queries to/from the RT database. A subset of InfoSphere DataStage events can also be logged to the OR database (which is viewable from the Web Console and Information Services Console).

When log events are purged by using the DataStage Director, they are purged from both the RT database and OR database. When log events are purged by using the Information Server Web Console or Information Services Console, they are purged only from the OR database.