fix(rustfs-mirror): sit through a node reboot instead of amplifying it #496
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/fix-mirror-reboot"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A kernel-roll reboot took rustfs-1 away for two minutes; the mirror turned that into twenty, by crash-restarting and re-listing 300k objects into a store whose RPC replay cache was already at capacity. A failed pass is now waited out in-process. The fix then had the same class of bug it was fixing -- a run that compared nothing reported a pass -- caught by a new fake-mc harness that executes each Job script and is kept as tests/rustfs-mirror-behaviour.sh. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
MEASURED 2026-09-09, during the binjovi-logs bulk transfer. The kernel roll rebooted seanfarm-worker-07b866, which holds rustfs-1. RustFS lost a peer for about two minutes. What followed was not two minutes of degradation: mc mirror got HTTP 503 and exited 1 set -eu killed the script restartPolicy: OnFailure restarted it from pass 1 pass 1 re-listed the whole bucket on both sides -- ~300k objects that listing storm drove RustFS's internode RPC replay cache to capacity (failure_reason: replay_cache_capacity, rpc_path ReadVersion) a full replay cache makes RustFS reject LEGITIMATE internode RPCs as signature verification failures -- 1331 on rustfs-0, 1879 on rustfs-2 which produced more 503s, which restarted the container again Five restarts in twenty minutes, against a reboot that lasted two. The mirror was not the cause of the outage and it was most of its duration. Rolling a patched kernel onto every node with no drama is what this cluster is for, so a node reboot is ROUTINE, and a mirror that cannot sit through one is a mirror that will do this again on every roll. A failed pass is now caught, waited out and retried IN THE SAME PROCESS. `mc mirror` copies only what differs, so the wait costs nothing and the next pass resumes where the last one stopped. A failed `mc diff` is treated the same way: the exit status is still not the parity gate -- the output is, and that has not changed -- but a listing that never COMPLETED must not be read as agreement between the stores. tempo.yaml gets the same shape though its bucket is 268 objects. The property is not about size; it is that no file here should have to be the one that learns this again. AND THE FIX HAD THE BUG IT WAS FIXING. The counters are initialised before the loop so `set -u` cannot abort at the gate. If every pass then failed on a transient, the counters still held their initial values, the bounded gate read missing=1 as a measurement, and the Job reported a successful bulk transfer with one object in flight -- a run that compared nothing, reporting a pass. Caught by a fake-mc harness before it ran anywhere. Each mirror now carries a `compared` flag and fails loudly if parity was never read. tests/rustfs-mirror-behaviour.sh is that harness, kept. It extracts each Job's script and EXECUTES it with mc and sleep stubbed, so a node reboot, a store outage and a corrupt copy are each reproducible in a second with no cluster. Eight behaviours per mirror, 24 in all. The static guard reads these Jobs as text and catches a missing gate; it cannot catch a gate that is present and wrong, and it did not catch this one. Four negative controls on the new guard, each confirmed red: the compared-nothing gate removed, a failed mirror pass exiting the script, an incomplete listing counted as parity, and the differing-object gate removed. The harness also proves its own stub can fail, because a stub that cannot go red makes every case above inert. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76c3749df6ba9ab7ea8dd89ab7ea8dd8e55e6d1614e55e6d1614f3ebc76ef1