perf(k3s-registries): the guard runs in a pool, and halves the GitOps step #624
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/parallel-k3s-guard"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This guard was 114.9 s of a 125.3 s tests/check.sh, and tests/check.sh is what the 108.9 s GitOps validation step spends its time on: about a hundred executions of the real reconcile script under the fakes, each about a second, every one independent. The guard goes 117.0 s to 63.1 s and the whole suite 118.6 s to 73.9 s, measured on a machine already carrying a load average of 11. The fixture is per WORKER, not per unit -- a tree cannot be copied because the fakes embed its absolute path, and one unit per worker was tried first and was SLOWER than serial (219 s) because it paid a fixture per scenario. Two scenarios stay out of the pool, and so do the three mutations that replay them: s_sigterm and s_sigterm_lease assert a one-second timing margin, and for a mutation contention would make it look RED when the property had not caught it -- a silent weakening, not a visible flake. Proved: the 30 mutations are the same set diffed against the serial run; three consecutive runs green at 64.6/63.2/63.1 s; injected failures in both lanes exit 1 and are named. K3S_GUARD_JOBS sets the width. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
This guard was 114.9 seconds of a 125.3-second `tests/check.sh`, and `tests/check.sh` is what the 108.9-second GitOps validation step spends its time on. It is about a hundred executions of the real reconcile script under the fakes -- 35 scenarios per shell dialect, then 30 mutation checks replaying them -- each about a second, almost all of it process starts, and every one of them independent. | | before | after | |---|---|---| | this guard | 117.0 s | 63.1 s | | whole `tests/check.sh` | 118.6 s | 73.9 s | Measured on a machine carrying a load average of 11 from other work, so a quiet builder should do better than this. THE FIXTURE IS PER WORKER, NOT PER UNIT. A tree cannot be copied -- the fakes embed its absolute path, so every fake in a copy points back at the original -- and building one costs about as much as a scenario. So the fixture block became `setup_fixture`, and a worker builds one tree and runs a CHUNK of the work in it, exactly as the serial section always did: every scenario starts with `fresh`, which is what makes that safe. One unit per worker was tried first and was SLOWER than serial, 219 s against 117 s, because it paid a fixture per scenario. TWO SCENARIOS DO NOT GO IN THE POOL, and neither do the three mutations that replay them. `s_sigterm` and `s_sigterm_lease` assert a timing margin: the fake containerd sleeps `SLOW_SLEEP=6` and the watchdog fires at 5, so a script that ignores SIGTERM waits out the child and is caught. One second of margin does not survive a machine running the rest of this guard beside it. Widening the pair is not the fix -- every one of those scenarios then pays the difference in wall time, which cost 100 seconds when tried. They run in this shell, after the pool drains. The mutations matter more than the flake: a mutation is RED when its replay FAILS, so contention there would make one look red when the property had not caught it. That is a silent weakening, not a visible failure, and it is why a mutation replaying a timing-margin scenario runs in the serial lane too. Output is held per slot and replayed IN ORDER, so a parallel run reads exactly like a serial one, and each worker keeps its own `failures` and `diagnosed` so a failure is reported by the collector rather than raced into the parent's counter. Proved, not assumed: * the 30 mutations are the SAME SET, `diff`ed against the serial run * three consecutive runs: 64.6 s, 63.2 s, 63.1 s, all green, 30 red mutations each * an injected failure in a pooled scenario exits 1 and names its chunk under both dialects; an injected failure in the serial lane exits 1 and is counted `K3S_GUARD_JOBS` sets the width, which defaults to `nproc`. The one free change to the timing was the poll before SIGTERM, from 10 s to 60 s: it breaks the moment the line appears, so a wider bound costs a healthy run nothing. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76