fix(rustfs-mirror): one stream per object; parallel multipart saturates RustFS #517
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/rustfs-mirror-single-stream"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Six failed zot passes read as MinIO truncating the source; MinIO's log shows it timing out writing to a client stalled on RustFS, whose internode locks were saturated by parallel multipart parts. Each object copies in a second alone; --disable-multipart copied a 1.2 GiB prefix in 12 s, faster than the two throttled forms. Applied to every mirror, guarded. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
The zot copy failed six of its passes on four different objects, each with the same shape of error: Put ".../blobs/sha256/490bdc5e...?partNumber=4&uploadId=...": http: ContentLength=12073893 with Body length 1470884 That reads as MinIO truncating the source. It was not. MinIO's own log at the same moments says "write tcp 10.42.3.202:9000->10.42.4.197: i/o timeout" -- MinIO gave up writing the object to a client that had stopped reading. The client had stopped reading because it was stalled writing the same part to RustFS, and RustFS at that moment reported 5,777 slow in-flight requests, 170 internode lock-RPC timeouts and 214 failed distributed unlocks. mc mirror uploads every large object as parallel 16 MiB parts across many objects at once, and a four-node erasure set serialises those through its internode locks. The stall propagates backwards until the source stream is cut, and mc blames the source. Each of the four objects copied in one second on its own, which is how the cause was separated from the symptom. Three forms of the mirror were then tried on a real 1.2 GiB, 91-object prefix: --limit-upload 60MiB --disable-multipart 62 s --limit-upload 60MiB 21 s --disable-multipart 12 s Single-stream PUTs were the fastest of the three, about 100 MiB/s, and need no bandwidth cap. Every mirror Job now passes --disable-multipart. Objects here are container layers well under the single-PUT ceiling. tests/rustfs-mirror.sh requires the flag, with a negative control. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76