IBM Operations Analytics - Log Analysis, Version 1.3.2

Configuring the Web Access Logs Insight Pack

Instructions on how to configure the Web Access Logs Insight Pack.

Procedure

In the IBM® Operations Analytics - Log Analysis Administrative Settings workspace, create a new log source for the log file to be monitored. The source type should be WebAccessLog.
On the web server, customize the access log format to a delimiter-separated value output (DSV) that can be consumed by the Web Access Log Insight Pack and IBM Operations Analytics - Log Analysis. The syntax to customize the log format is different for each web server, but the generated log will be the same. Following is the log format directive for the supported web servers:
For Apache/IHS
1. Edit <ServerRoot>/conf/httpd.conf file, where <ServerRoot> is the root installation path.
  1. Add the following log format directive:
    LogFormat "Apache/IHS,%h,%l,%u,%t,%m,\"%r\",%>s,%b,%D, \"%{Referer}i\",\"%{User-Agent}i\"" scalaAccessLog
  2. Update the access log directory specification to use the LogFormat directive:
    CustomLog logs/access_log scalaAccessLog
  3. Comment out the following line by prefixing it with #:
    CustomLog logs/access_log common
2. Restart the web server.
3. The generated access files are at <ServerRoot>/logs.
For JBoss
1. Edit the file <JBOSS_HOME>/jboss-eap-6.1/standalone/configuration/standalone.xml
2. Find the XML element subsystem xmlns="urn:jboss:domain:web:1.4" and add the following <access_log> element:
```
<subsystem xmlns="urn:jboss:domain:web:1.4" 
	default-virtual-server="default-host" native="false>
		<connector name="http" protocol="HTTP/1.1" scheme="http" 
			socket-binding="http"/>
		<virtual-server name="default-host" enable-welcome-root="true">
				<alias name="localhost"/>
				<alias name="example.com"/>
				<access-log prefix="access-log." pattern="JBoss,%h,%l,%u,%t,
					%m,&quot;	%r&quot;,%s,%b,%D,&quot;%{Referer}i&quot;
					,&quot;%{User-Agent}i&quot;">
						<directory path="." relative-to="jboss.server.log.dir"/>
				</access-log>
		</virtual-server>
</subsystem>
```
3. Restart the JBoss App Server
4. Look for the access log file in <JBOSS_HOME>/standalone/log
  Where <JBOSS_HOME> is the directory where you installed JBoss
For Apache Tomcat
1. Edit the file <tomcat-dir>/conf/server.xml where <tomcat-dir> is the installation root and add the following log format:
```
<Valve className="org.apache.catalina.valves.AccessLogValve" 
 directory="logs"
	prefix="localhost_access_log." suffix=".txt"
	pattern="Tomcat,%h,%l,%u,%t,%m,&quot;%r&quot;,%s,%b,%D,
 &quot;%{Referer}i&quot;,&quot;%{User-Agent}i&quot;" 
/>
```
2. Restart the web server using the scripts in <tomcat-dir>/bin
3. The log files are written on <tomcat-dir>/logs/localhost_access_log.<date>.txt
(Optional) Configure the Log File Agent to monitor rotated logs. This step is only required if your web server is configured to rotate log files and you are using the Log File Agent to ingest the log files.
Note: Access logs are rotated by default for Apache Tomcat and JBoss. Access Logs are not rotated by default for Apache/IHS. For instructions on how to configure log rotation for Apache/IHS, see Web Access Logs Best Practices.

Each web server has different syntax on how to specify rotation and the generated filename. By default, a rotated log has a timestamp or a number in the filename. Specify the log filename pattern in the WebAccessLogInsightPack-lfadsv.conf file that is applicable to your web server.
1. In WebAccessLogInsightPack-lfadsv.conf, update LogSources to monitor all the files in the directory:
```
LogSources=<web server log directory to monitor>/
<access_log_filename_without_timestamp>* 
FileComparisonMode=CompareByAllMatches
```
2. Update WebAccessLogInsightPack-lfadsv.fmt to specify a fixed filename so you can use the same fixed name in the path of the IBM Operations Analytics - Log Analysis logsource configuration. You only need to define one logsource with this path, and LFA will monitor all the files in the directory because you specified wildcard file naming in the WebAccessLogInsightPack-lfadsv.conf specification.
```
// Matches records for any Log file:
// REGEX AllRecords
(.*) hostname LABEL
-file web_server_access.log
RemoteHost DEFAULT logpath PRINTF("%s",file)
text $1
END
```
LFA will monitor all the log records in the directory (as specified by the LogSources value). This ensures no data will be lost as logs are rotated. However, LFA is allocating resources to monitor each file. This results in unnecessary resources since the rotated logs will not be updated again. It is a best practice to periodically archive old logs so LFA can release resources monitoring static files. For Unix, you can use tools like logrotate and cron to schedule archiving of old logs.
If you want to collect logs from multiple web servers, or want to ingest an archive of rotated logs, the recommended method for loading data is to use the Data Collector client.

Feedback