filemon Command

Purpose

Monitors the performance of the file system, and reports the I/O activity on behalf of logical files, virtual memory segments, logical volumes, and physical volumes.

Syntax

filemon [ -d ] [ -i Trace_File -n Gensyms_File] [ -o File] [-O Levels ] [ -w ] [-I count:interval] [-P ] [ -T n] [-u ] [-v ] [-@ [WparList | ALL ] [ -r RootString ] [ -A -x User_Command ]

Description

The filemon command monitors a trace of file system and I/O system events, and reports on the file and I/O access performance during that period.

In its normal mode, the filemon command runs in the background while one or more application programs or system commands are being run and monitored. The filemon command automatically starts and monitors a trace of the program file system and I/O events in real time. By default, the trace is started immediately; optionally, tracing might be deferred until you issue a trcon command. You can issue the trcoff and trcon commands while the filemon command is running to turn the monitoring off and on, as required. When tracing is stopped by a trcstop command, the filemon command generates an I/O activity report and exits.

The filemon command can also process a trace file that is previously recorded by the trace facility. The file and I/O activity report are based on the events recorded in that file.

To provide a complete understanding of file system performance for an application, the filemon command monitors file and I/O activity at four levels:

Any combination of the four levels can be monitored, as specified by the command-line flags. By default, the filemon command only monitors I/O operations at the virtual memory, logical volume, and physical volume levels. These levels are all concerned with requests for real disk I/O.

The filemon command also generates a hotness report on the files, logical volumes, and physical volumes. The hotness report can be generated by using –O hot option. This report is supported only in automated offline and manual offline modes. Hotness report contains statistics of I/O operations of files, logical volumes, and physical volumes. This report helps you decide which files or logical volumes to move to any drive, with a different I/O characteristic based on the hotness of the file/logical volume. The hotness is determined based on number of read operations, average number of bytes read per read operation, number of read sequences and the average sequence length.

The filemon command writes its report to standard output or to a specified file. By default the report contains a summary of the I/O activity for each of the levels being monitored. Detailed report is printed only if the -O detailed flag is enabled. The summary and detailed report contents are described in the Reports section.

System Trace Facility

The filemon command obtains raw I/O performance data using the system trace facility. Currently, the trace facility only supports one output stream. Consequently, only one filemon or trace process can be active at a time. If another filemon or trace process is already running, the filemon command responds with the message:

/dev/systrace: Device busy

While monitoring the I/O-intensive applications, the filemon command might not be able to consume trace events as fast as they are produced in real time. When that happens, the error message:

Trace kernel buffers overflowed, N missed entries

is displayed on stderr, indicating how many trace events were lost while the trace buffers were full. The filemon command continues monitoring the I/O activity, but the accuracy of the report is diminished to some unknown degree. One way to prevent overflow is to monitor fewer levels of the file and I/O subsystems: the number of trace events generated is proportional to the number of levels monitored. Additionally, the trace buffer size can be increased using the -T option, to accommodate larger bursts of trace events before overflow. Remember that increasing the trace buffer size results in more pinned memory, and therefore might affect I/O and paging behavior.

In memory-constrained environments (where demand for memory exceeds supply), the -P option can be used to pin the text and data pages of the real-time filemon process in memory so the pages cannot be swapped out. If the -P option is not used, letting the filemon process to be swapped out, the progress of the filemon command might be delayed to the point where it cannot process trace events fast enough. This situation leads to trace buffer overflow as described previously. Consequently, pinning this process takes memory away from the application (although the filemon command is not a large program, its process image can consume up to 500KB).

The -i Trace_File and -n Gensyms_File flags let offline processing by filemon of trace data files created by the trace command. Both flags must be supplied if either is present. These flags are useful when it is necessary to postprocess a trace file from a remote machine or perform the trace data collection at one time and postprocess it at another time. The flags are also useful when system load is high and trace hooks are being missed by filemon. You can use these flags for automated offline mode.

The -r RootString flag deprecates the -i Trace_File flag and the -n Gensyms_File flag. Apart from using the -r RootString flag for offline processing, the same can be used along with the -A flag which enables automated offline mode.

The gensyms file (containing file system information) must be used from the machine that the trace came from. Also, it is wise to run gensyms at close to the same time that the system trace file is created, so that the system configuration is the same for both.

Trace hooks relevant to filemon must be collected by the trace command and are specified by the trace -j flag. The relevant trace hooks are listed when filemon is started with the -v flag. The gensyms command with -F option is then run, with its output saved in Gensyms_File to collect additional information for filemon. The -F option is used with the gensyms command to collect the device information for physical and logical volumes. It is also used to get the virtual file system information used by offline filemon. Then this file and the Gensyms_File might be provided to filemon.

Reports

Each report generated by the filemon command has a header that identifies the date, the machine ID, and the length of the monitoring period, in seconds. The processor utilization during the monitoring period is also reported.

Next, summary reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, as measured by the total amount of data transferred. If the -v flag is specified, activity for all files and segments is reported. There is one row for each reported file, segment, or volume. The columns in each row for the four summary reports are described in the following lists:

Most Active Files Report

Most Active Segments Report

svmon -S <segid>

Most Active Logical Volumes Report

Most Active Physical Volumes Report

Most Active Files Process-Wise Report

Most Active Files Thread-Wise Report

Finally, detailed reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, as measured by the total amount of data transferred. If the -v flag is specified, activity for all files and segments is reported. There is one entry for each reported file, segment, or volume.

Some of the fields report a single value, others report statistics that characterize a distribution of many values. For example, response time statistics are kept for all read or write requests that were monitored. The average, minimum, and maximum response times and the standard deviation of the response times are reported. The standard deviation is used to show how much the individual response times deviated from the average. Roughly two-thirds of the sampled response times are between average - standard deviation and average + standard deviation. If the distribution of response times is scattered over a large range, the standard deviation will be large compared to the average response time. The four detailed reports are described in the following lists:

Detailed File Statistics Report

Detailed VM Segment Statistics Report

Detailed Logical/Physical Volume Statistics Reports

Detailed Process-wise Statistics Report

Detailed Thread-wise Statistics Report

Collated Report Format

Hotness Report

The hotness report consists of three sections: information section, summary section, and hotness reports section. The information section contains the system model, the filemon command used, and the trace command used. The summary section contains: total number of read or write operations, total time taken, total data read or written, and the CPU utilization.

Hot files report

Hot Logical Volumes Report

Hot Physical Volumes Report

Each of the described hotness reports is repeated multiple times based on the sort field.

The different hotness reports based on different sort fields are:

1. hotness report sorted on key factor
2. hotness report sorted on CAP_ACC
3. hotness report sorted on IOP/#
4. hotness report sorted on #ROP
5. hotness report sorted on #WOP
6. hotness report sorted on RTIME
7. hotness report sorted on WTIME

Each of the reports is sorted in descending order of the corresponding sort field.

If you specify the –O hot=r option then only read operations-based reports and report based on key factor are generated, that is, report number 1, 4, and 6 are generated.

If the user specifies –O hot=w option then only write operations-based reports and report based on key factor are generated, that is, report number 1, 5, and 7 are generated.

The key factor is determined by the values of following columns: #ROP, B/ROP, Seqlen and #Seq.

Flags

gensyms -F > file

Examples

1. To monitor the physical I/O activity of the virtual memory, logical volume, and physical volume levels of the file system, enter:

   filemon

   The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:

   trcstop

   After the trcstop command is issued, the I/O activity report is displayed on standard output (but probably scrolls off the screen). The virtual memory I/O report is limited to the 20 segments that incurred the most I/O.
2. To monitor the activity at all file system levels, and write the report to the fmon.out file, enter:

   filemon -o fmon.out -O all

   The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:

   trcstop

   After the trcstop command is issued, the I/O activity report is written to the fmon.out file. All four levels of the file and I/O system (the logical file, virtual memory, logical volume, and physical volume levels) are monitored. The logical file and virtual memory I/O reports are limited to the 20 files and segments (respectively) that incurred the most I/O.
3. To monitor the activity at all file system levels and write a verbose report to the fmon.out file, enter:

   filemon -v -o fmon.out -O all

   The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:

   trcstop

   This example is similar to the previous example, except a verbose report is generated on the fmon.out file. The primary difference is that the filemon command indicates the steps that it is taking to start the trace, and the summary and detailed reports include all files and segments that incurred any I/O (there might be many), instead of just the top 20.
4. To report on I/O activity captured by a previously recorded trace session, enter:

   filemon -i trcfile | pg

   In this example, the filemon command reads file system trace events from the input file trcfile. Since the trace data is already captured on a file, the filemon command does not put itself in the background to let application programs run. After the entire file is read, an I/O activity report for the virtual memory, logical volume, and physical volume levels will be displayed on standard output (which, in this example, is piped to pg).
5. To monitor the I/O activity for logical and physical volumes only, while controlling the monitored intervals using the trcon and trcoff commands, enter:

   filemon -d -o fmon.out -O pv,lv

   The filemon command automatically starts the system trace and puts itself in the background. After this command, you can enter the unmonitored application programs and system commands to be run at this time, then enter:

   trcon

   After this command, you can enter the monitored application programs and system commands to be run at this time, then enter:

   trcoff

   After this command, you can enter the unmonitored application programs and system commands to be run at this time, then enter:

   trcon

   After this command, you can enter the monitored application programs and system commands to be run at this time, then enter:

   trcstop

   In this example, the -O flag is used to restrict monitoring to logical and physical volumes only. Only those trace events that are relevant to logical and physical volumes are enabled. Also, as a result of using the -d flag, monitoring is initially deferred until the trcon command is issued. System tracing can be intermittently disabled and reenabled using the trcoff and trcon commands, so that only specific intervals are monitored.
6. To run filemon in offline mode, run the trace and gensyms commands separately, then use the output from those commands as input to the filemon command, as follows:

   trace -a -T 768000 -L 10000000 -o trace.out -j 000,000,001,002,003,
   005,006,139,102,10C,106,00A,107,101,104,10D,15B,12E,130,163,19C,154,
   3D3,1BA,1BE,1BC,10B,221,1C9,222,228,232,45B

   Run the monitored application programs and system commands, then enter:

   trcstop

   Create the gensyms file:

   gensyms -F > gensyms.out

   Then run filemon with both -i and -n flags:

   filemon -i trace.out -n gensyms.out -O all

7. To generate hotness report in automated offline mode, with unit of data as megabytes, use the following command:

   filemon -O hot,unit=MB -r <rootstring> -A-x "<user command>" 

8. To generate hotness report with three snapshots of trace in 5-seconds interval, run the following command:

   filemon -O hot -r <rootstring> -A-x "<user command>" -I 3:5

9. To generate hotness report in offline mode:

   filemon -r <rootstring> -O hot