feat(sysctl-tuner): cap the ZFS ARC at a quarter of RAM -- the ARM roll needs it

sean commented

2026-09-10 09:54:11 +00:00

Owner

Measured on a live worker: MemTotal 22.9 GiB, arcstats c_max 21.9 GiB, zfs_arc_max 0 -- ZFS was entitled to 96% of the node. Harmless at 24 GiB with 4.2 GiB in use; on an 8 or 16 GiB CAX node the same default starves the kubelet. Computed from the node's own MemTotal via the sysfs knob (CONFIG_MODULES=n, ZFS in-tree, so no modprobe.d; a kernel argument could only carry a constant), so one expression serves every size and both architectures and it lands without a kernel bake. New guard with six controls, all run; the first draft of the guard was itself inert and the control caught it. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit

2026-09-10 09:54:11 +00:00

feat(sysctl-tuner): cap the ZFS ARC at a quarter of RAM -- the ARM roll needs it

binjovi/ci Binjovi completed the frozen plan

Details

3b24a32819

Measured on a live worker, 2026-09-10: MemTotal 22.9 GiB and arcstats c_max
21.9 GiB. `zfs_arc_max` was 0, which means "all but a little", so ZFS was
entitled to 96% of the node for cache. Nothing broke, because ARC was holding
4.2 GiB -- and that is exactly why it went unnoticed for the life of the
cluster.

THIS IS A PREREQUISITE OF THE ARM ROLL, not a tuning nicety. CAX nodes are 8
or 16 GiB. There the same default lets ARC take most of RAM before the kubelet
notices and starts evicting pods, and the first symptom would be pods dying on
a fresh node with no obvious cause.

A quarter is the share this cluster can defend. ARC sits at 4.2 GiB now that
MinIO's per-object xl.meta files and its 300k-object log bucket have left
tank -- down from 8.3 GiB while they were there -- so the cap is above current
use on the nodes we have, and it still leaves an 8 GiB node 6 GiB for its
actual job.

COMPUTED, NOT CONSTANT, and a runtime knob rather than a boot parameter. The
kernel is CONFIG_MODULES=n with ZFS in-tree, so there is no modprobe.d to
write -- verified on the node, where /sys/module/zfs/parameters/zfs_arc_max
exists and is writable even though nothing is a module. A kernel command line
could carry only a fixed byte count, which would mean a value per node class
and a kernel bake to change it; one expression in the DaemonSet that already
nsenters every node serves every size and both architectures, and it lands
without a bake, which is what makes it ready BEFORE the ARM roll rather than
gated on it.

The boot window is real and self-correcting, and the comment says so: ZFS
imports the pool in the initramfs and ARC starts filling before this
DaemonSet runs, so ARC can briefly exceed the cap; lowering arc_max makes ZFS
evict down to it. If that window ever matters, a per-node-class kernel
argument is the belt-and-braces answer as well, not instead.

tests/host-arc-cap-contract.sh: the cap must be arithmetic over the node's own
MemTotal, must name its fraction, must be guarded on the knob existing so a
node without in-tree ZFS is a clean no-op, and must not use a module path.
Six controls run and confirmed: a hardcoded byte count, MemTotal not read, the
knob guard removed, a modprobe call, arc_fraction=1 (a cap of 100%, which is
the default this replaces) and arc_fraction=16; arc_fraction 2 and 8 are
accepted so the value stays retunable.

The first draft of that guard accepted the hardcoded byte count, because it
fell back to finding the word `arc_fraction` anywhere in the script -- a guard
reading the explanation instead of the code. Caught by running the control.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 7d8d82af3b into trunk

2026-09-10 10:02:25 +00:00

Rows
Columns

feat(sysctl-tuner): cap the ZFS ARC at a quarter of RAM -- the ARM roll needs it #543