Log file considerations

Log rotation

Linux provides logrotate to configure rotation. The global options are specified in /etc/logrotate.conf. Options for specific files (which can over ride the global options) are in /etc/logrotate.d for each log file.

When the logs are rotated, the log file is renamed with a .1 extension (assuming the dateext option is not included) and truncated to zero length. The rotation configuration also determines how often old logs are archived, that is, old *.n are removed or archived. The log locations are defined in /etc/rsyslog.conf (which is by default /var/log).

The recommended method for loading data is to use the IBM Tivoli Monitoring Log File Agent (LFA) installed on the remote system where rsyslogd is executing to push data, or use LFA installed where IBM® Operations Analytics - Log Analysis is installed to pull data. Create .conf and .fmt files specific to each log file, the destination of which is specified in /etc/rsyslog.conf. The data source definition should specify <filename>.1. This ensures all log records are processed. However, they will only be sent when the rotation occurs. The user can configure rotation to occur more frequently to minimize the time lag from the current log. Alternatively, the user can monitor the <filename>. When you monitor <filename>, there is a window where log entries will not be forwarded if log entries are rotated before LFA has polled the data. If the user has configured daily or hourly rotation, they can monitor the *.1 file name to avoid a window where log entries are not forwarded.

The best practice is to rotate the logs frequently so <filename>.1 has recent data for IBM Operations Analytics - Log Analysis to ingest. The default log rotation is weekly. You can change the rotation for syslog in /etc/logrotate.d/syslog. To change it to daily, add the daily option in /etc/logrotate.d/syslog configuration file. If the logs are large, you can rotate them based on a size with the size option.

The following two sections describe configuration changes to SyslogInsightPack-lfasyslog.conf and SyslogInsightPack-lfasyslog.fmt only when dealing with rotating log files.

Logfile Agent Configuration - SyslogInsightPack-lfasyslog.conf

The following parameters should be specified to monitor the rotating files:

LogSources=<syslog directory to monitor>/<logfilename>.* 
FileComparisonMode=CompareByAllMatches

Logfile Agent Configuration - SyslogInsightPack-lfasyslog.fmt

Use the following specification to avoid defining multiple data sources because of the file name changes when the log rotates. This allows a fixed file name in the log path specification.

// Matches records for any Log file:
//

REGEX AllRecords
(.*)
hostname LABEL
-file <logfilename>.log
RemoteHost DEFAULT
logpath PRINTF("%s",file)
text $1
END

Centralized logging

If you are streaming logs to a central server, the best practice is to stream to one consolidated log for ingestion by IBM Operations Analytics - Log Analysis. The same best practices are applicable to the consolidated file as for the logs in the non-server scenario. The logs should be rotated frequently so <filename>.1 has recent data, and the Log File Agent should be used to pull or push the data to the IBM Operations Analytics - Log Analysis server.

To configure rsyslog to stream logs to a central server (for example, 192.168.1.1), do the following:

1. Add the following to each client (or edge systems) /etc/rsyslog.conf file to stream to the central server:

   	$ModLoad imuxsock
   
   	$ModLoad imklog
   
   	# Provides UDP forwarding. The IP is the server's IP address
   	*.* @192.168.1.1:514
   	# Provides TCP forwarding. But the current server runs on UDP
   	# *.* @@192.168.1.1:514

2. On the central server (for example, with IP address 192.168.1.1) add the following to rsyslog.conf:

   	$ModLoad imuxsock 
   
   	# provides kernel logging support (previously done by rklogd)
   	$ModLoad imklog
   
   	# Select the syslog reception of UDP or TCP. For TCP, load imtcp by 
   uncommenting $ModLoad imtcp.
   #$ModLoad imudp
   #$ModLoad imtcp
   
   # Select the syslog reception port.  For TCP, uncomment InputServerRun 514
   #$UDPServerRun 514
   #$InputTCPServerRun 514
   # This FILENAME template generates the log filename dynamically.  
   # You can replace the specification with variables applicable to 
   # your site.
   # The scalaLogFormat template formats the message required 
   # for ingestion by SCALA.
   	$template FILENAME,"/var/log/scala-syslog.log"
   	$template scalaLogFormat,"%TIMESTAMP:::date-rfc3339% host=%HOSTNAME%, 
   	relayHost=%FROMHOST%, tag=%syslogtag%, programName=%programname%, 
   procid=%P		ROCID%, 	facility=%syslogfacility-text%, 
   sev=%syslogseverity-text%, appName=%APP-NAM	E%, msg=%msg%\n"
   
   
   	# Log all messages to the dynamically formed file.
   	*.* ?FILENAME;scalaLogFormat

3. Decide whether you are going to use either the UDP or TCP configuration and comment out the other. For example, to use TCP update the code section as follows:

   # Select the syslog reception of UDP or TCP. For TCP, load imtcp 
   # by uncommenting $ModLoad imtcp.
   #$ModLoad imudp
   $ModLoad imtcp
   
   # Select the syslog reception port.  For TCP, uncomment 
   # InputServerRun 514
   #$UDPServerRun 514
   $InputTCPServerRun 514