fix(rustfs-mirror): gate on what is missing, not on what changed #484
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/loki-mirror-gate"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The loki mirror failed four times in ten minutes with nothing wrong. mc mirror --quiet still prints one line per object -- measured in the Job's own log -- so it fed its output into the log store whose bucket it was copying; stdout now goes to /dev/null and the guard checks the redirection rather than the flag. And mc diff is directional: an empty diff is unachievable for a bucket whose compactor deletes from the source, so the gate is now '<' and '!' with '>' counted and reported. Four controls, each confirmed red. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
The loki mirror ran and failed four times in ten minutes with nothing wrong. Two mistakes, both mine, and both only visible once a real bucket went through it. `--quiet` DOES NOT DO WHAT I SAID IT DOES. `mc mirror --quiet` drops the progress meter and still prints one line per object copied. The Job emitted a line per object with the flag set -- measured, in its own log. Alloy ships that stdout into Loki, whose chunks are the bucket being copied, so it fed itself exactly as the flag was supposed to prevent. The mirror's stdout now goes to /dev/null. Errors are on stderr and a non-zero exit still stops the script. The guard checks the redirection, which is the property that matters, instead of a flag that proved nothing. AN EMPTY DIFF IS NOT ACHIEVABLE FOR A BUCKET WITH RETENTION. `mc diff` is directional and the direction is the whole point: < only on MinIO the mirror is behind. A real gap. ! contents differ a corrupt or truncated copy. > only on RustFS the source deleted it after we copied it. Loki runs with `retention_enabled: true` and `delete_request_store: s3`, and `mc mirror` does not propagate deletions, so `>` lines appear on every run forever. Requiring an empty diff meant requiring the compactor to stop. The gate is now `<` and `!`, and `>` is counted and reported. Objects written during a pass show up as missing, so the mirror repeats up to five times until that set is empty; it converges because loki's write rate is far below the copy rate. tempo gets the same gate -- its traces are append-only so `>` should stay at zero there, but the gate should mean one thing across the directory rather than something each file decides. Four negative controls, each confirmed to turn the guard red on its own: removing the missing-object count, removing the differing-object count, un-redirecting the mirror, and going back to the empty-diff gate. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76