A five-minute ZFS pool health check

The handful of commands I run on every ZFS box: pool status, error counters, capacity, scrubs and alerting.

ZFS tells you a lot about the state of a pool, but only if you ask. This is the short routine I run on any machine I inherit. The examples use a pool named tank.

Is anything wrong right now

zpool status -x

On a healthy system the answer is a single line: all pools are healthy. Anything else deserves the long form:

zpool status -v tank

The -v flag lists files affected by unrecoverable errors, which is the first thing you want to know before deciding what to restore.

Reading the error counters

Each device in zpool status has three counters.

Column Meaning Usual suspect
READ The device returned an I/O error on read Failing disk, cable, backplane
WRITE The device returned an I/O error on write Failing disk, controller
CKSUM Data was read but did not match its checksum Cable, HBA firmware, bad RAM

After the cause is fixed, reset the counters so new errors stand out:

zpool clear tank

Capacity and fragmentation

zpool list -o name,size,alloc,free,cap,frag,health

Keep cap under roughly 80%. A nearly full pool becomes slow, because the allocator has to work harder to find free space, and that slowdown does not fully go away after you free space again.

Scrubs

A scrub reads every block and verifies it against its checksum. It is the only way to find silent corruption on data nobody has read for months.

zpool scrub tank
zpool status tank | grep scan

OpenZFS 2.1.3 and later ship systemd timers, so a monthly scrub is one command:

systemctl enable --now zfs-scrub-monthly@tank.timer

Getting told about problems

The ZFS Event Daemon can send mail when a device faults or a scrub finishes with errors.

ZED_EMAIL_ADDR="root"
ZED_NOTIFY_INTERVAL_SECS=3600
ZED_NOTIFY_VERBOSE=1
systemctl restart zfs-zed

With ZED_NOTIFY_VERBOSE=1 you also get a message after every clean scrub. That monthly mail doubles as proof that alerting still works.