A five-minute ZFS pool health check
The handful of commands I run on every ZFS box: pool status, error counters, capacity, scrubs and alerting.
ZFS tells you a lot about the state of a pool, but only if you ask. This is the short routine I run on any machine I inherit. The examples use a pool named tank.
Is anything wrong right now
zpool status -x
On a healthy system the answer is a single line: all pools are healthy. Anything else deserves the long form:
zpool status -v tank
The -v flag lists files affected by unrecoverable errors, which is the first thing you want to know before deciding what to restore.
Reading the error counters
Each device in zpool status has three counters.
| Column | Meaning | Usual suspect |
|---|---|---|
READ |
The device returned an I/O error on read | Failing disk, cable, backplane |
WRITE |
The device returned an I/O error on write | Failing disk, controller |
CKSUM |
Data was read but did not match its checksum | Cable, HBA firmware, bad RAM |
After the cause is fixed, reset the counters so new errors stand out:
zpool clear tank
Capacity and fragmentation
zpool list -o name,size,alloc,free,cap,frag,health
Keep cap under roughly 80%. A nearly full pool becomes slow, because the allocator has to work harder to find free space, and that slowdown does not fully go away after you free space again.
Scrubs
A scrub reads every block and verifies it against its checksum. It is the only way to find silent corruption on data nobody has read for months.
zpool scrub tank
zpool status tank | grep scan
OpenZFS 2.1.3 and later ship systemd timers, so a monthly scrub is one command:
systemctl enable --now zfs-scrub-monthly@tank.timer
Getting told about problems
The ZFS Event Daemon can send mail when a device faults or a scrub finishes with errors.
ZED_EMAIL_ADDR="root"
ZED_NOTIFY_INTERVAL_SECS=3600
ZED_NOTIFY_VERBOSE=1
systemctl restart zfs-zed
With ZED_NOTIFY_VERBOSE=1 you also get a message after every clean scrub. That monthly mail doubles as proof that alerting still works.