Analyzing logs with Elastic Stack

You can use the Elastic Stack to analyze your Liberty logs. The Elastic Stack is a third-party collection of log analysis products and is not included with Liberty.

Open Liberty The most current documentation for Analyzing Liberty logs with Elastic Stack is available on the Open Liberty website.

About this task

You can view your Liberty logs by using Kibana dashboards.

Procedure

  1. Set the messages.log file to JSON format.
    1. Use the bootstrap.properties file, as shown in the following example. 
      com.ibm.ws.logging.message.format=json
com.ibm.ws.logging.message.source=message,trace,accessLog,ffdc,audit
    2. Alternatively, use the following environment variables. 
      WLP_LOGGING_MESSAGE_FORMAT=json
WLP_LOGGING_MESSAGE_SOURCE=message,trace,accessLog,ffdc,audit
  2. Start the server.
  3. Set up the Elastic Stack to use and visualize logs.

    Complete the following steps to configure your Elasticsearch, Logstash, Kibana, and Filebeat.

    1. Download the sample Logstash configuration file, liberty_logstash.conf, from this repository.
    2. In the Logstash configuration file, customize the Beats port, port:port_number, and the Elasticsearch hosts value Elasticsearch_host_name:port_number .
    3. Download the sample Filebeat configuration file for the appropriate version from the repository. For Filebeat 5.x, use the liberty_filebeat5.yml file. For Filebeat 6.x / 7.x, use the liberty_filebeat6_7.yml file.
      • In the Filebeat configuration file, change the path of the log to the location of the messages.log file.
    4. Start Elasticsearch, Logstash, Kibana, and Filebeat.

      See the Elastic website for instructions.

    5. Open Kibana in a browser and create an index.
      • For Kibana 6 and 7, click Management > Index Patterns.
        • Enter logstash-* as the Index Pattern.
        • Click Next > Advanced Options and enter logstash-* as the Index Pattern ID.
        • Select ibm_datetime as the Time filter field name, and click Create.
      • For Kibana 5.6 or later, click Management > Index Patterns.
        • Enter logstash-* as the Index Pattern.
        • Click Advanced Options, and enter logstash-* as the Index Pattern ID.
        • Select ibm_datetime as the Time filter field name, and click Create.
      • For Kibana 5.0-5.5, click Management > Index Patterns.
        • Select ibm_datetime as the Time filter field name, and click Create.
    6. Download one or more of the sample dashboards from this repository. For Kibana 5.x, download JSON files that start with kibana5. For Kibana 6.x and 7.x, download JSON files that start with kibana6_7.
    7. Import the dashboard into Kibana.
      • For Kibana 5, 6, and 7, click Management > Saved Object > Import
    8. View the dashboard.
      • For Kibana 5, click Dashboard > Open, and select the dashboard.

Results

You can now send your Liberty logs to the Elastic Stack and analyze them by using the provided Kibana dashboards.