Using the Logstash collector version 1.0

Use the Logstash collector feature in Liberty to collect logs and other events from your Liberty servers and send them to a remote Logstash server. The collected events can be used for log analysis and troubleshooting purposes.

The most current documentation for using Logstash collector with Liberty is available on the Open Liberty website.

Before you begin

The logstashCollector-1.0 feature was tested with the following products:

Logstash V2.x, Elasticsearch V2.x, and Kibana V4.x
Logstash V5.3.x, Elasticsearch V5.3.x, and Kibana V5.3.x
Logstash V6.4.x, Elasticsearch V6.4.x, and Kibana V6.4.x
Logstash V7.x, Elasticsearch V7.x, and Kibana V7.x

You can use the logstashCollector-1.0 feature with a Logstash server that runs with any of the available output plug-ins from Logstash. However, many users choose to use Logstash with Elasticsearch and Kibana to provide a complete log consolidation and analysis facility. For more information, see the Elastic website.

Logstash collector garbage collection events are obtained from IBM Health Center in IBM JDKs. IBM JDKs usually include IBM Health Center with the exception of IBM Semeru JDKs.

Procedure

Set up Logstash by following the instructions from Elastic.
Create or acquire certificate and key pair files for SSL for Logstash.
The following example command for openSSL generates a certificate and key pair. Customize the number of days the keys are valid as needed.
```
openssl req -x509 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -days 365 -nodes
```
For Logstash and Elasticsearch users, download a sample Logstash configuration file and an index template file from this repository.
1. Download the Logstash configuration file, liberty_logstash.conf, and the index template file, liberty_logstash_template.json, for your Elastic stack version according to the readme file.
2. In the liberty_logstash.conf file, customize the lumberjack ssl_certificate and ssl_key paths and the Elasticsearch hosts (Elasticsearch_host_name:port_number) value.
3. For Elasticsearch V2.x users, optionally customize the _ttl defaults to indicate the number of milliseconds to keep records of each event type in the liberty_logstash_template.json file.
Complete the following steps for each Liberty server from which you want to collect events.
1. Acquire or create a keystore for the Liberty server. To create a self-signed certificate, use the following command. Customize the server name, password, and subject as needed.
```
d:\wlp\bin\securityUtility createSSLCertificate --server=myServerName --password="Liberty" --subject=CN=myHostname,OU=defaultServer,O=ibm,C=us
```
2. Import the logstash.crt file from step 2 into the trust.jks file in your server. Customize the wlp_install_dir and server name as needed. When prompted for a password, use the certificate password from step 4a.
```
d:\java\bin\keytool -import -noprompt -alias logstash -file logstash.crt -keystore wlp_install_dir\usr\servers\myServerName\resources\security\trust.jks -storepass Liberty
```
3. Run the following command to install the logstashcollector-1.0 feature:
```
d:\wlp\bin\installUtility install logstashcollector-1.0
```
4. Configure Logstash collector in the server.xml file in Liberty by adding the following content. Customize the logstashCollector list of sources, host name, and port as needed.
  You can change the maximum number of characters that are allowed for the message field in a message or trace event or for the stack trace in an FFDC event. The default maximum is 2048 characters. The following example sets the maxFieldLength parameter to 3000 characters in the logstashCollector element. For an unlimited number of characters, set maxFieldLength to 0 (zero) or -1.
  
  You can limit the maximum number of events that are sent for each type of event (message, trace, accessLog, ffdc, garbageCollection, audit) per second. The default maximum number of events is 0, which means that no throttling is applied. If maxEvents is set to greater than 0, there is a limit to the number of events that are sent per second. The maxEvents value must be an integer between 0 and 2147483647.
  
  Also, you can add custom tags to decorate events that are sent from the server. The following example has two tag elements in the logstashCollector element.
  Note: Tags that are used in Admin Center have no relation to tags that are used in logstashCollector/bluemixLogCollector. If you are using the audit source, then enable the audit feature. For more information, see the Audit topic.
```
<featureManager>
    <feature>logstashCollector-1.0</feature>
</featureManager>

<keyStore id="defaultKeyStore" password="Liberty" />
<keyStore id="defaultTrustStore" location="trust.jks" password="Liberty" />
<ssl id="mySSLConfig" trustStoreRef="defaultTrustStore" keyStoreRef="defaultKeyStore" />

<logstashCollector 
	source="message,trace,garbageCollection,ffdc,accessLog,audit" 
	hostName="localhost" 
	port="5043" 
	sslRef="mySSLConfig"
	maxEvents="10"
	maxFieldLength="3000">
	<tag>serverRackA5</tag>
	<tag>billingAppTeam</tag>
</logstashCollector>
```
  Trace and access logs are high volume logs and require more network, CPU, and storage resources to collect.
5. Enable HTTP access logging and use the '%h %u %t "%r" %s %b %D %{User-agent}i' log format.
Start Elasticsearch, Logstash, and Kibana. See the Elastic website for instructions.
Start the Liberty server and generate some events.
Open Kibana in a browser and create an index.
- For Kibana 7, 6, or 5.6, click Management > Index Patterns.
  - Enter logstash-* as the Index Pattern.
  - Click Advanced Options, and enter logstash-* as the Index Pattern ID.
  - Select datetime as the Time filter field name, and click Create.
- For Kibana 5.0-5.5, click Management > Index Patterns. and select datetime as the Time filter field name. Click Create.
- For Kibana 4, click Settings > Index Patterns, and select datetime as the Time filter field name. Click Create.
Download a sample dashboard from this repository.
Import the dashboard into Kibana.
- For Kibana 7, 6, or 5 , click Management > Saved Object > Import.
- For Kibana 4, click Settings > Objects > Import.
View the dashboard.
- For Kibana 7, 6, or 5 , click Dashboard > Open and then select the dashboard.
- For Kibana 4, click Dashboard > Load Saved Dashboard and then select the dashboard.

Results

You configured your Liberty servers to send events to your Logstash server and can now view your events in the Liberty dashboard by using Kibana.

For z/OS platforms If you enable Logstash collector to acquire garbage collection events, late attach might not enable properly for IBM® tools against the z/OS® Java™ virtual machine (JVM). To fix the problem, add the system property and value -Dcom.ibm.tools.attach.enable=yes for the JVM invocation. Either add it directly as part of the Java execution or within a jvm.options file.