IBM Operations Analytics - Log Analysis, Version 1.3.2

WebSphere Application Server data loading best practice

There are different data loading recommendations based on how WebSphere® Application Server logging is configured.

To set up the data loading, you need to consider:

WebSphere Application Server configuration

Is WebSphere Application Server configured to produce rolling logs or single logs?

Data loading

Determine if you want to use the IBM® Tivoli® Monitoring Log File Agent (LFA) installed on the remote WebSphere Application Server to push data or to use the LFA installed on the local IBM Operations Analytics - Log Analysis server to pull data. In some scenarios, you must use the Data Collector as described in scenario 2.

Logfile agent configuration

If you choose to use the LFA, use the logfile agent configuration files to specify the log files you want to monitor. The WASInsightpack-lfawas.conf and WASInsightPack-lfawas.fmt files are located in the directory:

<HOME>/IBM-LFA-6.30/config/lo

The log file scenarios here describe the specific settings for these files.

Scenario 1 - Log file rotation on one WebSphere Application Server server

WebSphere Application Server configuration

WebSphere Application Server is configured for rolling log files. For example, records are written to SystemErr.log. When it reaches a defined log file size it is renamed to SystemErr_13.05.10_05.22.05.log and a new SystemErr.log is created for more data. A similar flow occurs for SystemOut.log and trace.log.

Data Loading Method

The recommended method for loading data is to use the IBM Tivoli Monitoring Log File Agent (LFA) installed on the remote WebSphere Application Server server to push data or to use the LFA installed on the local IBM Operations Analytics - Log Analysis server to pull data. Create individual .conf and .fmt files that are specific to each log file. The LogSources definition will specify the name of the renamed file. This approach should insure that all log records are processed (no loss of log records) but with the trade off that the log records forwarded to IBM Operations Analytics will be sent only once the rollover occurs. The log forwarding will be one log file behind real-time resulting in some time lag for search results. The amount of time is dependent on the environment, that is, what the defined log size is for rollover and how frequently this occurs based on logging volumes.

Loading logs using the LFA is a CPU bound process. If your system does not meet the minimum requirements you will need to increase the MaxEventQueueDepth. On some systems, altering this value may produce a noticeable impact on performance. This will buffer additional LFA events while they are waiting to be processed. The required value for MaxEventQueueDepth may vary depending on the size of the rolled log and the number/speed of your CPU's. If you choose not to increase this value, then older events may be replaced on the event queue by newer events and not sent to the IBM Operations Analytics server.

To minimize the chance of data loss due to CPU bottlenecks, and to reduce the latency between when a log record is written to the file and when it is loaded, we recommend that the maximum size of a WebSphere Application Server log be small enough so that you system does not fall behind while processing the logs.

An alternative method is to always monitor the current log file (for example, SystemErr.log) and not the renamed log file. This would result in log records being forwarded immediately by the LFA. The trade off is that this configuration may result in log entries not being forwarded if those log entries were written during the LFA polling interval (sleep time) and a rollover occurs. In this case the LFA would start processing the new logfile.

Logfile Agent Configuration - lfawas.conf file

You must create additional .conf files for each log type that you monitor. For example, if want to monitor the SystemOut.log and trace.log, then you need a .conf file for each log file.

Specify the following parameters to monitor the rotating SystemErr.log files:

LogSources=<was log directory to monitor>/SystemErr_*.log 
FileComparisonMode=CompareByAllMatches

Logfile Agent Configuration - lfawas.fmt file

You must create additional .fmt files for each log type that you monitor. For example, if want to monitor the SystemOut.log and trace.log, then you need a .fmt file for each log file.

Use the following .fmt file to specify a fixed SystemErr.log name and avoid the need to define multiple logsources because of the rolling log file name changes. This allows a fixed file name in the logpath.

// Matches records for any Log file:
//

REGEX AllRecords
(.*)
hostname LABEL
-file SystemErr.log
RemoteHost DEFAULT
logpath PRINTF("%s",file)
text $1
END

The same pattern can be used to define the .conf and .fmt files for the other logs:

<was log directory to monitor>/SystemOut_*.log   OR   
<was log directory to monitor>/trace_*.log</p>

Scenario 2 - Collecting Log Files from multiple WAS Servers

WebSphere Application Server Configuration: WebSphere Application Server is configured for single log files (non-rolling) on multiple servers and the server logs are collected for data loading on the IBM Operations Analytics - Log Analysis server.
Data Loading Method: The recommended method for loading data is to use the Data Collector client. Remove the previous log files before creating or copying new versions into the directory from which you will load data. The order of processing logs in this manner is important to handle any split records. When using the Data Collector client you need to set the flushflag so that split records can be merged. This is set in the javaDatacollector.properties file located in the <HOME>/utilities/datacollector-client/ directory.

Feedback