fix(rustfs-mirror): zot is copied AND proved per repository; the bucket is never listed whole

sean commented

2026-09-10 03:38:46 +00:00

Owner

RustFS returns a large recursive listing short with HTTP 200 under load (dst/zot: 1,543 of 21,600; build/: 1,543 of 13,264; every repository correct on its own). The bucket-level diff reported 20,095 present objects as missing and each bucket pass re-uploaded them for 20 minutes. The zot Job now mirrors and diffs per repository prefix and sums the counts; a unit never copied or compared fails the run by name. 82 behaviours in the fake-mc suite. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit

2026-09-10 03:38:46 +00:00

fix(rustfs-mirror): zot is copied AND proved per repository; the bucket is never listed whole

binjovi/ci Binjovi completed the frozen plan

Details

1518ed1ac1

Measured 2026-09-10 03:30 UTC. A recursive listing of dst/zot on RustFS
returned 1,543 objects for a bucket holding 21,600, and the listing of its
`build/` prefix returned the same 1,543 for 13,264 -- while every
repository under it listed correctly on its own: build/binjovi 9,888 to
MinIO's 9,908 (the gap is live pushes), node-bootc 638 = 638, and every
other namespace equal on both sides. RustFS ends a recursive listing
SILENTLY -- HTTP 200, IsTruncated false -- where a sub-tree walk fails
under load (`Metacache walk_dir failed`, BrokenPipe). The bucket-level
`mc diff` therefore reported 20,095 missing objects that were present, and
each bucket-level pass re-uploaded them for twenty minutes until a GOAWAY
burst ended it, with the drives at the same 105 GiB after every pass.

The zot Job now does everything per repository prefix, which lists
reliably: mirror (six attempts), then diff (six attempts), summing the
directional counts; nothing is copied or concluded from a listing above a
repository. A unit never copied or never compared fails the run by name --
its objects are absent from the sums, and absence must not read as parity.
The bounded bulk gate and the zero cutover gate are unchanged in meaning.

tests/rustfs-mirror.sh: the rate-cap anchor matches a unit path.
tests/rustfs-mirror-behaviour.sh: the stub records diff arguments and lists
two repositories for zot by default; zot's cases assert two unit mirrors
and two unit diffs with no bucket-level call, an in-place retry, and that a
unit whose listing never completes fails the run. 82 behaviours.

RECORDED FOR RUSTFS: a large recursive ListObjects under write load comes
back short with a success status. Any consumer that lists a big prefix --
GC, backup, `mc du` -- must not trust one listing.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 1518ed1ac1 into trunk

2026-09-10 03:44:53 +00:00

Rows
Columns

fix(rustfs-mirror): zot is copied AND proved per repository; the bucket is never listed whole #526