Configure your agent to receive data from a log file data
source.
Before you begin
Note: The agent monitors log files
that are in the same locale and code page that the agent runs in.
Procedure
- On the Agent Initial Data Source page
(Figure 1) or the Data Source Location page, click Logged Data in the Monitoring Data Categories area.
Figure 1. Adding
a log file
- In the Data Sources area, click A Log File.
- Click Next.
- On the Log File Information page (Figure 2), type the name of the
log file you want to monitor in the Log File Information area. The file name must be fully qualified.
Figure 2. Log File Information page
- Optional: Part of the log file name can
come from a runtime configuration property. To create a log file name,
click Insert Configuration Property and select
a configuration property (Figure 1).
- Optional: The file can also be a dynamic
file name. For more information, see (Dynamic file name support).
- In the Field Identification area, click one of the following options:
- Fixed number of characters
- When selected, limits the number of characters.
With this
option, each attribute is assigned the maximum number of characters
it can hold from the log file. For example, if there are three attributes
A, B, and C (in that order), and each attribute is a String of maximum
length 20. Then, the first 20 bytes of the log record go into A, the
second 20 into B, and the next 20 into C.
- Tab separator
- When selected, you can use tab separators.
- Space separator
- When selected, multiple concurrent spaces can be used as a single
separator.
- Separator Text
- When selected, type in separator text.
- Begin and End Text
- When selected, type in both Begin and End text.
- XML in element
- When selected, type the name of the XML element to use as the
record, or click Browse to define the element.
If you clicked
Browse, the
XML Browser window is displayed (
Figure 3). If you use the browse function, the Agent Builder identifies all
possible attributes of the record by looking at the child tags and
their attributes.
Figure 3. XML Browser window
Note: Unless you click
Advanced and fill out the information in that window, the following assumptions
are made about information that you complete:
- Only a log file at a time is monitored.
- Each line of the log file contains all the fields necessary to
fill the attributes to be defined.
For more information about log file parsing
and separators, see (Log file parsing and separators).
- Optional: Click Advanced on the Log File Information page to do the following by using the Advanced Data Source
Properties page (Figure 4):
- Monitor more than one file, or monitor files with different names
on different operating systems or monitor files with names that match
regular expressions.
- Draw a set of fields from more than one line in the log file.
- Choose Event Filtering and Summarization
Options.
- Produce output summary information. This summary produces an additional
attribute group at each interval. For more information about this
attribute group, see (Log File Summary). This function
is deprecated by the options available in the Event Information tab.
Figure 4. Advanced Data Source Properties page, File Information
- To monitor more than 1-log file, click Add and type the name. If more than one file is listed, a
unique label must be entered for each file. The label can be displayed
as an attribute to indicate which file generated the record. It must
not contain spaces.
- Optional: To select the operating systems
on which each log file is to be monitored, follow these steps:
- Click in the Operating systems column for
the log file.
- Click Edit.
- In the Operating Systems window, select the
operating systems.
- Click OK to save your changes and return
to the Advanced Data Source Properties page.
- Optional: Select File names match
regular expression if the file name you are providing
is a regular expression that is used to find the file instead of being
a file name. For more information, see (ICU regular expressions). If you do not check
this box, the name must be an actual file name. Alternatively it must
be a pattern that follows the rules for file name patterns that are
described in (Dynamic file name syntax).
- Optional: Select One
directory element matches regular expression to match
one subdirectory of the file name path with a regular expression. You can select this option only if you also selected File names match regular expression in the previous step.
If regular expression meta characters are used
in the path name, the meta characters can be used in only one subdirectory
of the path. For example, you can specify /var/log/[0-9\.]*/mylog.* to have meta characters in one subdirectory. The [0-9\.]* results in matching any subdirectory of /var/log that consists solely of numbers and dots (.).
The mylog.* results in matching any file names in
those/var/log subdirectories that begin with mylog and are followed by zero or more characters.
Because some operating systems use the backslash (\) as a directory separator it can be confused with a regular expression
escape meta character. Because of this confusion forward slashes must
always be used to indicate directories. For example, Windows files that are specified as C:\temp\mylog.* might mean the \t is
a shorthand tab character. Therefore, always use forward slashes (/) on all operating systems for directory separators. The C:/temp/mylog.* example represents all files in the C:/temp directory that start with mylog.
- In the When multiple files match list, select one of the following options:
- The file with the highest numerical value in the file
name
- The biggest file
- The most recently-updated file
- The most recently-created file
- All files that match
Note: When you select
All files that
match, the agent identifies all files in the directory
that match the dynamic file name pattern. The agent monitors updates
to all of the files in parallel. Data from all files is intermingled
during the data collection process. Its best to add an attribute by
selecting
Log file name in
Record
Field Information to correlate log messages to the log
files that contain the log messages. Ensure that all files that match
the dynamic file name pattern can be split into attributes in a consistent
manner. If the log files selected cannot be coherently parsed, then
its best to select
Entire record in
Record Field Information to define a single attribute.
For more information about specifying
Record Field Information for attributes, see step (
8).
- Choose how the file is processed. With Process all records when the file is sampled, you can process all records in the entire file every time the defined
sampling interval for the log monitor expires. The default interval
is 60 seconds. This interval can be modified by using the KUMP_DP_COPY_MODE_SAMPLE_INTERVAL environment variable
(specifying a value in seconds). The same records are reported every
time unless they are removed from the file. With this selection,
event data is not produced when new records are written to the file.
With Process new records appended to the file, you can process new records that are appended to the file while
the agent is running. An event record is produced for every record
added to the file. If the file is replaced (first record changes
in any way), the file is processed and an event is produced for each
record in the file.
Note: If appending records to an XML log file,
the append records must contain a complete set of elements that are
defined within the XML element you selected as Field Identification.
- If you chose to process new records that are appended
to the file, you can also choose how new records are detected. With Detect new records when record count increases, new records can be detected when the number of records in the file
increases, whether the size of the file changes. This feature is useful
when an entire log file is pre-allocated before any records are written
to the file. This option can be selected for files that are not pre-allocated,
but it is less efficient than monitoring the size of the file. With Detect new records when the file size increases, you
can determine when a new entry is appended to a file in the typical
way. There might be a brief delay in recognizing that a monitored
file is replaced.
- If you selected Detect new records when the
file size increases, you can also choose how to process
a file that exists when the monitoring agent starts. Ignore existing records disables event production for
any record in the file at the time agent starts. Process
___ existing records from the file specifies production
of an event for a fixed number of records from the end of the file
at the time the agent starts. Process records not previously
processed by the agent: Specifies for restart data to
be maintained by the monitoring agent so the agent knows which records
were processed the last time that it ran. Events are produced for
any records that are appended to the file since the last time the
agent was running. This option involves a little extra processing
each time a record is added to the file.
- If you selected Process records not previously
processed by the agent, you can choose what to do when
the agent starts and apparently the existing file was replaced. Process all records if the file has been replaced: If information about the monitored file and the restart data information
do not match, events are produced for all records in the file. Examples
of mismatches include: The file name is different, the file creation-time
is different, the file-size decreased, the file last modification
time is earlier than before. Do not process records if
the file has been replaced: If the information about the
monitored file and the restart data information do not match, disables
processing of existing records in the file.
- Click the Record Identification tab (Figure 5) to interpret
multiple lines in the log file as a single logical record.
Note: If you select XML in element as the
field identification on the Log File Information page, the Record Identification tab does
not display.
Figure 5. Advanced Data Source Properties page, Record
Identification
- If you did select Process all records when
the file is sampled earlier, click the Filter
Expression tab. By clicking Filter Expression you can filter the data that is returned as rows based on the values
of one or more attributes, configuration variables or both. If you selected Process new records appended to the file earlier you cannot create a filter expression. For more information
about filtering data from an attribute group, see (Filtering attribute groups).
Figure 6. Advanced Data Source Properties page, Filter Expression
tab
- If you selected Process new records appended
to the file earlier, click the Event Information tab Figure 7 to select Event Filtering and Summarization Options. For more information, see (Event filtering and summarization).
Note: The Summary tab can be present if the agent was created
with an earlier version of Agent Builder. The summary tab is now deprecated
by the Event Information tab
Figure 7. The Event Information tab
of the Advanced Data Source Properties page,
- Optional: Click Test
Log File Settings on the Log File Information pageto start and test the agent (Figure 2). Click Test
Log File Settings after you select the options for the
log source. For more information about testing, see (Testing log file attribute groups).
- Use the following steps if you did
not use the test function earlier and you typed the log file name
in the Log File Information area of the Log File Information page:
- Click Next to display the Attribute Information page and define the first attribute
in the attribute group.
- Specify the information, on the Attribute
Information page, and click Finish.
Note: When a log file attribute group is added to an agent
at the default minimum Tivoli® Monitoring version (6.2.1) or later, a
Log File Status attribute group is included. For more information about the
Log File Status attribute group, see (
Log File Status attribute group).
Figure 8. Attribute Information
page
Along with the fields applicable to all data sources, the Attribute Information page for the log file data source
has some additional fields in the Record Field Information area.
The
Record Field Information fields are:
- Next field
- Shows the next field after parsing, by using the delimiters from
the attribute group (or special delimiters for this attribute from
the Advanced dialog).
- Remainder of record
- Shows the rest of the record after previous attributes are parsed.
This attribute is the last attribute, except for possibly the log
file name or log file label.
- Entire record
- Shows the entire record, which can be the only
attribute, except for possibly the log file name or log file label.
- Log file name
- Shows the name of the log file.
- Log file label
- Shows the label that is assigned to the file on the advanced panel.
Note: Use the Derived Attribute
Details tab only if you want a derived attribute, and
not an attribute directly from the log file.
- Click Advanced in
the Record Field Information area to display
the Advanced Log File Attribute Information page
(Figure 9).
Figure 9. Advanced Log File Attribute Information page
- In the Attribute Filters section,
specify the criteria for data to be included or excluded. Filtering attributes can enhance the performance of your solution
by reducing the amount of data processed. Click one or more of the
attribute filters:
- Inclusive indicates that the attribute
filter set is an acceptance filter, meaning that if the filter succeeds,
the record passes the filter, and is output.
- Exclusive indicates that the attribute
filter set is a rejection filter, meaning that if the attribute filter
succeeds, the record is rejected, and is not output.
- Match all filters indicates that all filters
defined to the filter must match the attribute record in order for
the filter to succeed.
- Match any Filter indicates that if any
of the filters that are defined to the filter match the attribute
record, the filter succeeds.
- Use Add, Edit, and Remove to define the individual filters
for an attribute filter set.
Figure 10. Add Filter window
- To add a filter, follow these steps:
- Click Add, and complete the options in
the Add Filter window (Figure 10) as follows:
- The Filter criteria section defines the
base characteristics of the filter, including the following properties:
- Starting offset defines the position in
the attribute string where the comparison is to begin.
- Comparison string defines the pattern string
against which the attribute is defined.
Type a string,
pattern, or regular expression that is used by the agent to filter
the data read from the file. The records that match the filter pattern
are eliminated from the records that are returned to the Tivoli Enterprise Portal, or are the only
records returned. The result depends on whether you choose for the
filter to be inclusive or exclusive.
- Match entire value checks for an exact
occurrence of the comparison string in the attribute string. Checking
starts from the starting offset position.
- Match any part of value checks for the
comparison string anywhere in the attribute string. Checking starts
from the starting offset position.
- The comparison string is a regular expression indicates that the comparison string is a regular expression pattern
that can be applied against the attribute string.
Regular expression-filtering
support is provided by using the International Components for Unicode
(ICU) libraries to check whether the attribute value examined matches
the specified pattern.
To effectively use regular
expression support, you must be familiar with the specifics of how
ICU implements regular-expressions. This implementation is not identical
to how regular expression support is implemented in Perl, grep, sed, Java™ regular expressions, and other implementations.
See ICU regular expressions for guidance
on creating regular expression filters.
- Define an override filter indicates that
you want to provide a more specific filter comparison that overrides
the base characteristics previously defined. This additional comparison
string is used to reverse the filter result. When the filter is Inclusive,
the override acts as an exclusion qualifier for the filter expression.
When the filter is Exclusive, the override acts as an inclusion qualifier
for the filter expression. (For more about Inclusive and Exclusive,
see step 9, and the examples
that follow). The override filter has the following properties:
- Starting offset defines the position in
the attribute string where the comparison is to begin.
- Comparison string defines the pattern string
against which the attribute is matched.
Type a regular expression
that is used by the agent to filter the data read from the file. The
records that match the filter pattern are eliminated from the records
that are returned to the Tivoli Enterprise Portal, or are the only records returned. The result
depends on whether you choose for the filter to be inclusive or exclusive.
- Replacement value can be used to alter
the raw attribute string with a new value. See ICU regular expressions for more details about
special characters that can be used.
- Replace first occurrence replaces the first
occurrence that is matched by the comparison string with new text.
- Replace all occurrences replaces all occurrences
that are matched by the comparison string with new text.
- Click OK.
Figure 11. Add Filter example 1
If the attribute string is
abc is easy as 123, then the replaced string that is displayed in the Tivoli Enterprise Portal as
123 is
not as easy as abc.
Figure 12. Add Filter example 2
If the attribute string is
Unrecoverable Error reading
from disk, and the filter is Inclusive, then the attribute
is displayed in the Tivoli Enterprise Portal. If the attribute string is
No Errors
Found during weekly backup and the filter is Inclusive, then
the attribute is not displayed in the Tivoli Enterprise Portal.
- In the Field Identification section
of the Advanced Log File Attribute Information page (Figure 9),
specify how to override the attribute group field delimiters for this
one attribute only. Click one of the attribute filters, and complete
the required fields for the option:
- Number of characters: Enter the limit for
the number of characters.
- Tab separator specifies the use of tab
separators.
- Separator Text: Enter the separator text
that you want to use.
- Begin and End Text Enter both Begin text and End text.
- In the Summary section of the Advanced Log File Attribute Information page (Figure 9),
click the Include attribute in summary attribute group check box to add the attribute to the summary attribute group. This attribute group is produced when a user turns on log attribute
summarization.
- Click OK.
- If you used the test function in step (7), the Select key
attributes page is displayed. On the Select key
attributes page, select key attributes or indicate that
this data source produces only one data row. For more
information, see (Selecting key attributes).
- Do one of the following steps:
- If you are using the New Agent wizard, click Next.
- Click Finish to save the data source
and open the Agent Editor.
Note: When a log file attribute group is added to an agent with
the default minimum Tivoli Monitoring version (6.2.1) or later, a
Log File Status attribute group is included. For more information about the
Log File Status attribute group, see (
Log File Status attribute group).