fix(rustfs-mirror): cap the upload rate; mc adds workers until RustFS breaks #520
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/rustfs-mirror-upload-cap"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
mc mirror has no worker-count flag: it starts one worker per CPU and adds one every 4 s while throughput rises, up to 128. Measured on the zot copy 2026-09-10: 109 slow PUTs in flight on one node, RustFS closing connections mid-PUT, every pass failing. A global --limit-upload 100MiB/s (the rate the uncapped probe measured) makes throughput plateau, which is what stops the ramp. Guarded in tests/rustfs-mirror.sh with a confirmed negative control. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
mc mirror has no worker-count flag. It starts one worker per CPU (12 on these nodes) and adds one every 4 s while the measured throughput still rises, up to 128. Against RustFS that ramp does not stop where the store saturates; it stops where the store breaks. Measured on the zot copy, 2026-09-10 01:14 UTC, single-stream PUTs and no cap: 109 PUTs in flight on one node flagged slow, 543 slow internode RPCs, metacache walks failing on every bucket, and RustFS closing connections mid-PUT ("Connection closed by foreign host") -- one per pass, a different blob each time, so every pass failed and the copy made ~50 MiB/s of net progress between failures. A global rate cap makes the throughput plateau, and a plateau is what stops the ramp; it is the only concurrency control mc exposes. 100 MiB/s is what the uncapped probe measured on a 1.2 GiB prefix, so the cap costs nothing on the fast path and bounds the store's exposure on the slow one. At that rate the 165 GiB zot bucket is ~30 minutes. tests/rustfs-mirror.sh requires the cap on every mirror; the negative control strips it and is confirmed detected. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76