IBM Tivoli Monitoring, Version 6.3

Format file specifications

The format file describes the patterns that the agent looks for to match events in the monitored logs. The format file consists one or more format specifications.

You can change the format file while an agent instance is running. The file is read by the agent when it starts, and is monitored for changes to its timestamp every 60 seconds thereafter. If the timestamp of the file changes, the agent reinitializes its configuration dynamically, without requiring a restart. For more information, see Changing the agent configuration and format files.

To create new patterns to match an event, use the new regular expression syntax that consists of the following parts:

Format header
Regular expression
Slot mappings
End statement

The format header contains the keyword REGEX, which informs the agent that you are using a regular expression to match the pattern in the monitored log.

You assign this regular expression to an event class as shown in the following example:

REGEX REExample

If you use the special predefined event class *DISCARD* as your event class, any log records matching the associated pattern are discarded, and no events are generated for them. For example:

REGEX *DISCARD*

As a pattern matched, nothing is written to the unmatch log. The log file status records matched include these discarded events.

Note: You can assign multiple event definitions to either the same event class or to different event classes. The class name is arbitrary and you can use it to indicate the type of event or to group events in various ways.

After the format header, the format content consists of a regular expression on the first line, followed by mappings. Each mapping is shown on a separate line and these mappings are described in the following example.

All lines that match the regular expressions are selected and sent to the monitoring server as events. The regular expression contains sub expressions. You can use the sub expressions to match specific parts of these lines that match, to a variable called a 'slot' in the Event Integration Facility. You can optionally map these slots to specific attributes in IBM® Tivoli® Monitoring.

The monitoring log shown here contains three lines that you might want to monitor:

Error:  disk failure 
Error: out of memory 
WARNING: incorrect login

For example, you generate an event for a specific error, such as the lines that begin with Error: and ignore the line that begins with Warning:. The regular expression must match the lines that begin with Error: and also include a sub expression. The sub expression is denoted by parentheses and it must match only the input text that you want to assign to the msg slot. An example is shown in the following format definition:

REGEX REExample
Error: (.*)
msg $1
END

Note: This example shows a simple regular expression with only one sub expression.

Based on this format specification, and the preceding set of log data, the agent generates two events. Both events are assigned the REEXample event class. In the first event, the disk failure value is assigned to the msg slot. Also, in the second event, the out of memory value is assigned to the msg slot. As the Warning line did not match the regular expression, it is ignored and no event is generated.

When you assign the value of $1 to the msg slot, you assign it the value of the first sub expression.

If you have log text that contains the following errors:

Error: disk failure on device /dev/sd0: bad sector
Error: disk failure on device /dev/sd1: temperature out of range

you might want to assign these error messages to their own event class so that you are informed immediately if there is a disk failure. You can include a description of the disk on which the error occurred and more specifically the disk error in the event.

The following regular expression contains two sub expressions that identify this information:

REGEX DiskFailure
Error: disk failure on device (/dev/sd[0-9]):(.*)
device $1 CustomSlot1
msg $2
END

You assign these two sub expressions to event slots. The two events that are generated contain the following values:

"device=/dev/sd0" and "msg=bad sector"
"device=/dev/sd1" and "msg=temperature out of range"

If you use EIF to generate the first event, it displays as shown in the following example:

DiskError;device='/dev/sd0';msg='bad sector';END

If the event is sent to IBM Tivoli Monitoring, the slot named msg is assigned to the IBM Tivoli Monitoring attribute of the same name. But there is no pre-defined IBM Tivoli Monitoring attribute for the slot that we called device.

If you need to see the value assigned to device directly on the Tivoli Enterprise Portal, or write situations against it, you must assign it to an IBM Tivoli Monitoring attribute.

The Tivoli Log File agent includes the following 13 predefined attributes:

10 string type attributes ranging from

CustomSlot1 to CustomSlot10.

Three integer type attributes ranging from

CustomInteger1 to CustomInteger3.

Using these attribute names in the format file populates IBM Tivoli Monitoring attributes of the same name. Using these attributes does not affect the content of the EIF event sent directly to OMNIbus.

Note: The CustomSlot and CustomInteger attribute names are case-sensitive and so you must enter the names exactly as shown.

You assign a slot from the event definition to one of these custom IBM Tivoli Monitoring attributes in the format file.

You assign the device slot to the IBM Tivoli Monitoring string type attribute called CustomSlot1 as shown in the following example:

REGEX DiskFailure
Error: disk failure on device (/dev/sd[0-9]):(.*)
device $1 CustomSlot1
msg $2
END

When the event displays in IBM Tivoli Monitoring, the value assigned to the device slot is assigned to the IBM Tivoli Monitoring attribute called CustomSlot1. You view this value on the Tivoli Enterprise Portal or use it to define situations. You can assign any slot in the event definition to any of the 10 custom IBM Tivoli Monitoring attributes in the same manner, by putting "CustomSlot<n>", where <n> is a number from 1 to 10, next to the slot definition.

In this example, the first sub expression is defined specifically as (/dev/sd[0-9]), but the second sub expression is defined generally as (.*). In defining the regular expression as specifically as possible, you improve performance. Therefore, if you enter a search for an error on a device that does not match the specific error message defined here, the search procedure stops immediately when the error is not found. Time is not wasted looking for a match.

The END keyword completes the format specification. The format header, regular expression, and the END keyword must each begin on a new line, as shown in the following example:

REGEX REExample 
Error:
msg $1
END <EOL>
<EOF>

Note: For the last format in the file, you must insert a newline after the END keyword as shown in the example. Otherwise, you get a parsing error.

The custom integer attributes CustomInteger1 to CustomInteger3 are 64-bit integer attributes. You can use them in the same manner as the string type CustomSlot attributes. You can use these attributes to map individual slots, or sub expressions, from the log file to individual IBM Tivoli Monitoring attributes. Because these attributes are numeric, you can use arithmetic comparisons on them, such as < and >, which is not possible with the string attributes.

Note: Although these values are evaluated as integers by the IBM Tivoli Monitoring infrastructure, for EIF purposes, and within the format file, they are still treated as strings. So for example, to use an integer slot in a PRINTF statement, you still identify it with “%s”, not “%d”.

The following example illustrates the use of a custom integer attribute. Suppose that a periodic UNIX syslog message is received that reports the percentage of a file system that is free, such as the following hypothetical log record:

Oct 24 11:05:10 jimmy fschecker[2165]: Filesystem /usr is 97% full.

You can use the following statement in the format file to check for the percentage of the file system that is free:

REGEX FileSystemUsage
^([A-Z][a-z]{2}) ([ 0-9][0-9]) ([0-9]{2}:[0-9]{2}:[0-9]{2}) (.*?) (.*?): 
Filesystem (.*?) is ([0-9]+)% full\.$
Month   $1 CustomSlot1
Date    $2 CustomSlot2
Time    $3 CustomSlot3
Host    $4 CustomSlot4
Service $5 CustomSlot5
Filesystem     $6 CustomSlot6
PctFull	     $7  CustomInteger1
msg          PRINTF("%s: %s% full", Filesystem, PctFull)
END

Note: In the preceding statement, everything between the ^ and $ symbols on the second and third lines must be on a single line.

Because you might have other events that put values in CustomInteger1, you can avoid confusing the different event types by using the value of the Class attribute to limit its effect to the correct type of events. For example, the following situation formula causes the situation to fire only when an event of the FileSystemUsage event class has a value greater than or equal to 95 in CustomInteger1:

(  Class == 'FileSystemUsage' AND CustomInteger1 >= 95)

A different event can then use CustomInteger1 for a different purpose and not trigger this situation accidentally.

In summary, you can now write a situation in IBM Tivoli Monitoring that uses arithmetic operators on the CustomInteger attributes, which is not possible with the CustomSlots attributes.

Note: If you map non-integer data to the CustomInteger attributes the resulting value might be zero or some unexpected value.

Feedback