fix(rustfs-mirror): copy zot in repository units; a GOAWAY burst costs one unit #524
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/rustfs-mirror-zot-units"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
RustFS's internode HTTP/2 connection receives GoAway(too_many_internal_resets) -- hyper's rapid-reset guard, fed by cancelled walk_dir listing streams -- and every in-flight shard write dies; mc ends a whole mirror pass on one failed object and the restart re-lists 21k objects, so each burst pays for the next (measured 2026-09-10 02:13, four passes, no net progress). zot is now copied one repository prefix at a time, six attempts each, before the bucket passes; a burst costs one unit's attempt and completed objects stay. Behaviour suite asserts unit order and in-place retry. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
Measured 2026-09-10 02:13 UTC, single-stream PUTs and the rate cap in place: RustFS's internode HTTP/2 connection received hyper::Error(Http2, GoAway("too_many_internal_resets", ENHANCE_YOUR_CALM)) -- hyper's rapid-reset guard -- and closed. Every shard write in flight on that connection died at once ("error reading a body from connection", "body size mismatch: expected 14318374 bytes, received 10486400"), a dozen PutObjects across all four nodes answered "internal error" within two seconds, and mc, which ends a whole `mc mirror` pass on one failed object, ended the pass. The resets are cancelled `walk_dir` listing streams (metacache BrokenPipe precedes each burst), and a restarted pass re-lists 21,000 objects on both sides, so every failure paid for the next one: passes 1-4 each ended in a burst, 1.5 to 2.5 minutes apart, and the drives sat between 53 and 61 GiB throughout -- the in-flight objects a burst discards are the pass's whole gain. The zot copy now runs in units first, one repository prefix (`<namespace>/<repository>/`) at a time, six attempts each: a burst costs one unit's attempt, the retry re-lists one repository, and every object a unit completed stays. The bucket-level passes follow with little left to copy and one listing to make; the diff gate is unchanged. The behaviour suite lists two repositories for zot and asserts the unit mirrors run before the bucket pass, in order, and that a failed unit is retried in place while the run still passes. 78 behaviours. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76