zfsslower(8)

NAME
zfsslower - Trace slow zfs file operations, with per-event details.

SYNOPSIS
zfsslower [-h] [-j] [-p PID] [min_ms]

DESCRIPTION
This tool traces common ZFS file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).

This uses kernel dynamic tracing of the ZPL interface (ZFS POSIX Layer), and will need updates to match any changes to this interface.

This is intended to work with the ZFS on Linux project:
    http://zfsonlinux.org

Since this uses BPF, only the root user can use this tool.

REQUIREMENTS
CONFIG_BPF and bcc.

OPTIONS
-p PID Trace this PID only.

min_ms
    Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.

EXAMPLES

Trace synchronous file reads and writes slower than 10 ms:
    # zfsslower

Trace slower than 1 ms:
    # zfsslower 1

Trace slower than 1 ms, and output just the fields in parsable format (csv):
    # zfsslower -j 1

Trace all file reads and writes (warning: the output will be verbose):
    # zfsslower 0

Trace slower than 1 ms, for PID 181 only:
    # zfsslower -p 181 1

FIELDS

TIME(s)
    Time of I/O completion since the first I/O seen, in seconds.

COMM
    Process name.

PID Process ID.

T   Type of operation. R == read, W == write, O == open, S == fsync.

OFF_KB
    File offset for the I/O, in Kbytes.

BYTES
    Size of I/O, in bytes.

LAT(ms)
    Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.

FILENAME
    A cached kernel file name (comes from dentry->d_name.name).

ENDTIME_us
    Completion timestamp, microseconds (-j only).

OFFSET_b
    File offset, bytes (-j only).

LATENCY_us
    Latency (duration) of the I/O, in microseconds (-j only).

OVERHEAD
This adds low-overhead instrumentation to these ZFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.

Note that the overhead of this tool should be less than fileslower(8), as this tool targets zfs functions only, and not all file read/write paths (which can include socket I/O).

SOURCE
This is from bcc.

    https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

OS
Linux

STABILITY
Unstable - in development.

AUTHOR
Brendan Gregg

SEE ALSO
biosnoop(8), funccount(8), fileslower(8)