Binary logging

Binary logging is a high performance log and trace facility based on the High Performance Extensible Logging (HPEL) technology in WebSphere Application Server traditional.

Overview

Note: You must enable the binary logging facility to use it.

Binary logging provides a convenient mechanism for storing and accessing log, trace, System.err, and System.out information produced by the application server or your applications. It is an alternative to the default log and trace facility, which provides the JVM logs and diagnostic trace files commonly named messages.log and trace.log.

Log and trace storage

Binary logging provides a log data repository and a trace data repository. See the following figure to understand how applications and the application server store log and trace information.

In the previous figure:

Logging for the application code is collected in a root logger, which sends the logging data to a log/trace data handler in the HPEL logging framework and a log/trace data handler in the default logging framework.
Logging data from the application server is collected by a logger and sent to a serial peripheral interface (SPI), which sends the logging data to a log/ trace service in the HPEL logging framework and a log/ trace service in the default logging framework.
The log/trace service in each framework sends the data to the log/trace data handler for that framework.
The HPEL logging framework log/trace data handler sends the data to the log data repository, trace data repository, and text log within that framework.
The default logging framework log/trace data handler sends the data to the messages log and trace log within that framework.
The data from each framework is also sent to the console log.

Log data repository: The log data repository is a storage facility for log records. Log data is typically intended to be reviewed by administrators. This includes any information applications or the server write to System.out, System.err, OSGi logging service at level LOG_INFO or higher (including LOG_INFO, LOG_WARNING, and LOG_ERROR), or java.util.logging at level Detail or higher (including Detail, Config, Info, Audit, Warning, Severe, Fatal, and any custom levels at level Detail or higher).
Trace data repository: The trace data repository is a storage facility for trace records. Trace data is typically intended for use by application programmers or by the WebSphere® Application Server support team. This includes any information applications or the server write to the OSGi logging service at level LOG_DEBUG or java.util.logging at levels less than level Detail (including Fine, Finer, Finest, and any custom levels less than level Detail).

Log and trace performance

Binary logging has been designed and tested to significantly outperform the default log and trace facility. The influence on performance of running the application server with trace enabled is greatly improved by using binary logging rather than the default log and trace framework. In addition, using binary logging results in applications that frequently write to the logs running faster.

Log and trace events are each stored in only one place: Log events, System.out, and System.err are stored in the log data repository. Trace events are stored in the trace data repository. Storing each type of event in only one place ensures that performance is not wasted on redundant data storage.

Note: The console log must be disabled in cases where logging performance is important. Any content written to the console log will already be stored in the log data repository.
Data is not formatted unless it is needed: Formatting data for a user to read uses processor time. Rather than format log event and trace event data at run time, log and trace data are stored more rapidly in a proprietary binary representation. This improves the performance of the log and trace facility. By deferring log and trace formatting until the binaryLog command is run, sections of the log or trace that are never viewed are never formatted.
Log and trace data are buffered before being written to disk: Writing large blocks of data to a disk is more efficient than writing the same amount of data in small blocks. The binary logging facility provides the capability to buffer log and trace data before writing it to disk. By default, log and trace data are stored in an 8 KB buffer before being written to disk. If the buffer is filled within 10 seconds, the buffer is written to disk. If the buffer is not filled within that time it is automatically written to disk to ensure that the logs have the most current information.

Administration of log and trace

Binary logging has been designed to be easy to configure and understand. For example, administrators can easily configure how much disk space to dedicate to logs or trace, or how long to retain log and trace records, and leave the management of log and trace content up to the server. As another example all log, trace, System.out, and System.err content can be accessed using one easy-to-use command (binaryLog), avoiding any possible confusion over which file to access for certain content.

Reading from the log data and trace data repositories: The log data and trace data repositories are stored in a WebSphere Application Server proprietary format and cannot be read using text file editors such as Notepad or VI. You can copy the log data and trace data repositories in to a plain text format using the binaryLog command.
binaryLog command: binaryLog is an easy-to-use command-line tool provided for users to work with the log data and trace data repositories. binaryLog provides filtering and formatting options that make finding important content in the log data and trace data repositories easy. For example, a user might filter any errors or warnings, then filter all log and trace entries that occurred within 10 seconds of a key error message on the same thread.
Filtering using log and trace record extension content: The binary logging facility provides the capability for developers to add custom extensions to log and trace records using a log record context API (com.ibm.websphere.logging.hpel.LogRecordContext). You can use the binaryLog command-line tool to filter records based on the content of log and trace record extensions.

Development resources

Binary logging has been designed to make working with log and trace content more flexible and effective than the default logging facility. Log and trace content can be easily filtered to show only the records that are of interest. You can use the command line (see the description of the binaryLog command), or developers can create powerful log handling programs using the HPEL API.

Image showing the log and trace data repositories connecting to com.ibm.websphere.logging.hpel API.

Reading the log data and trace data

An API has been provided to make it easy for developers to develop tools to consume content from the binary log and trace repositories. For example, a developer might write a Java™ program to search the log and trace content to find any messages with message IDs that match a known list of important message IDs. This API is in the com.ibm.websphere.logging.hpel package. Refer to the API documentation for details on the HPEL log reading API.

Log and trace record extensibility

Developers can add custom extensions to log and trace records through a log record context API (com.ibm.websphere.logging.hpel.LogRecordContext). When binary logging stores log and trace records, it includes any extensions present in the log record context on the same thread. For example, a developer might write a servlet filter to add important HTTP request parameters to the log record context. While that servlet runs, HPEL API adds those extensions to any log and trace records created on the same thread.

As with other log and trace record fields, developers can access the record extensions using the HPEL API. This is useful when writing tools to read from log and trace repositories. Developers can also make use of the log record context API to access extensions in custom log handlers, filters, and formatters at run time.