fix(rustfs): 4 x 210Gi, and sweep stale multipart uploads before every pass

sean commented

2026-09-10 00:17:13 +00:00

Owner

The zot copy filled the 110Gi drives to 100% with 70 GiB landed: 742 incomplete multipart uploads from CrashLoops and a killed transfer held ~400 GiB nothing could see. PVCs grown in place to 210Gi (template follows); every mirror now sweeps stale uploads before its first pass, guarded statically and behaviourally. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit

2026-09-10 00:17:13 +00:00

fix(rustfs): 4 x 210Gi, and sweep stale multipart uploads before every pass

binjovi/ci Binjovi completed the frozen plan

Details

46ccff5156

The zot copy filled the store to 100% on 2026-09-10 with loki, binjovi-logs
and forgejo live on it. It was aborted and purged; nothing was lost, and MinIO
was still authoritative for zot throughout. Two causes, two fixes.

THE DRIVES WERE SIZED FROM A RATIO, NOT FROM A FAILURE. 110Gi came from a
2.004x erasure ratio against a 136 GiB bucket. The bucket was 165 GiB by the
time it moved, and the ratio described a clean transfer, which this was not.
The four PVCs were grown in place to 210Gi -- zfs-nvme-128k allows expansion
and each drive is a ZFS dataset, so it was a quota bump with no restart --
and the template follows so a rebuild creates them at the same size. Tank has
about 250 GiB free per RustFS node before MinIO retires and returns ~110 GiB
more to each. tests/rustfs.sh carries the new pin and the reason.

THE WRECKAGE WAS NOT THE RATIO. When the pool was full, 70 GiB of zot objects
had landed and ~406 GiB of raw space was gone. The difference was 742
INCOMPLETE MULTIPART UPLOADS: six CrashLoops on a wrong access key and one
killed transfer had each left every in-flight container layer half-uploaded,
holding raw space that no `mc du` shows and that no later pass reclaims.
Purging them returned the drives to 7 GiB. Every mirror Job now runs
`mc rm --incomplete --recursive --force` on its destination BEFORE the first
mirror call. They are the destination's own uploads, written by this Job's
scoped key, so the sweep is always safe -- and it must come first, or a Job
that dies mid-pass hands the next run the space it left behind.

Both guards learned it. tests/rustfs-mirror.sh requires the sweep and requires
it to precede the mirror, with a negative control. tests/rustfs-mirror-behaviour.sh
runs each Job against the stub, which now records the mirror-call count at
every sweep, and asserts the first recorded count is zero; removing the sweep
from a Job was confirmed red. 50 behaviours across five mirrors.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 46ccff5156 into trunk

2026-09-10 00:23:32 +00:00

Rows
Columns

fix(rustfs): 4 x 210Gi, and sweep stale multipart uploads before every pass #512