perf(guards): run the guard suite in parallel, and unpin the renamed cache claim #569

Merged

binjovi-bot merged 2 commits from sean/parallel-guard-suite into trunk

2026-09-10 13:36:22 +00:00

AGit

sean commented

2026-09-10 13:27:42 +00:00

Owner

Two commits. First, trunk is red and the wiring it guards is correct: dc078fbc renamed the buildkit cache claim to binjovi-buildkit-cache-arm-evac for the ARM worker replacement and moved tests/binjovi-buildkit.sh to a derived name, but check-kernel-build-cache.sh kept pinning the literal, so every full run of tests/check.sh has failed since. Its neighbouring probe survived only by accident, because claimName: binjovi-buildkit-cache is a PREFIX of the new name. Both are replaced by deriving the claim from the Deployment and pinning the shape: the daemon mounts a cache claim, that claim is declared in the same file, and it is on zfs-nvme-128k. Seven controls now, including that a CONSISTENT rename must still pass while a half rename must fail. Second, the suite runs in parallel. Serial it is 323 seconds, and three guards are 70 percent of that. A change touching tests/check.sh runs the whole suite inside one gitops_validate task with a 600 second deadline, and seanfarm #552 was killed at 602 seconds by the clock rather than by a failing guard. In parallel it finishes in 118 seconds with the same 97 guards reporting. The guards are hermetic; the five that use a fixed /tmp path were checked and every name is distinct. Every guard now runs even after one fails, and output is replayed in list order. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Two commits. First, trunk is red and the wiring it guards is correct: dc078fbc renamed the buildkit cache claim to binjovi-buildkit-cache-arm-evac for the ARM worker replacement and moved tests/binjovi-buildkit.sh to a derived name, but check-kernel-build-cache.sh kept pinning the literal, so every full run of tests/check.sh has failed since. Its neighbouring probe survived only by accident, because claimName: binjovi-buildkit-cache is a PREFIX of the new name. Both are replaced by deriving the claim from the Deployment and pinning the shape: the daemon mounts a cache claim, that claim is declared in the same file, and it is on zfs-nvme-128k. Seven controls now, including that a CONSISTENT rename must still pass while a half rename must fail. Second, the suite runs in parallel. Serial it is 323 seconds, and three guards are 70 percent of that. A change touching tests/check.sh runs the whole suite inside one gitops_validate task with a 600 second deadline, and seanfarm #552 was killed at 602 seconds by the clock rather than by a failing guard. In parallel it finishes in 118 seconds with the same 97 guards reporting. The guards are hermetic; the five that use a fixed /tmp path were checked and every name is distinct. Every guard now runs even after one fails, and output is replayed in list order. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 2 commits

2026-09-10 13:27:43 +00:00

fix(guards): derive the buildkit cache claim, do not pin its name 1a9a88e72b

TRUNK IS RED, AND THE WIRING IT GUARDS IS CORRECT. dc078fbc renamed the cache
claim to binjovi-buildkit-cache-arm-evac, because the 2026-09-10 ARM worker
replacement copied the cache to a surviving worker and the bound PVC has to be
kept. tests/binjovi-buildkit.sh moved to a derived claim name in that same
commit. This guard did not, so `^  name: binjovi-buildkit-cache$` stopped
matching and every full run of tests/check.sh has failed since.

THE OTHER PROBE SURVIVED BY ACCIDENT, WHICH IS WORSE. `claimName:
binjovi-buildkit-cache` still passed, because it is a PREFIX of
binjovi-buildkit-cache-arm-evac. One probe went red for a correct change and
its neighbour stayed green for no reason at all. Both are gone.

A claim name is operational. It changes whenever a volume is replaced, and the
ARM migration will replace more of them. So derive it from the Deployment and
pin the shape instead: the daemon mounts a cache claim, that claim is declared
in this same file, and it is on zfs-nvme-128k. That is what must not change.

Three controls join the existing four. A CONSISTENT rename must still pass --
that is the whole point of deriving the name -- while a half rename, where the
Deployment points at a claim the file does not declare, must fail, and so must
moving the cache off ZFS. Two more controls were run from outside the guard:
the daemon swapping its claim for an emptyDir, and the declaration renamed
while the mount is not.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

perf(guards): run the guard suite in parallel

binjovi/ci Binjovi completed the frozen plan

Details

4e87756c60

MEASURED, NOT GUESSED. Run one after another, the 97 guards take 323 seconds.
Three of them are 70 percent of that: k3s-registries-contract.sh 128 s,
workflows-secret-declarations.sh 51 s, binjovi-collaborator-bootstrap.sh 47 s.
The other 94 total about 97 s.

THIS WAS ALREADY FAILING A BUILD. A change that touches tests/check.sh selects
the repository-guard-catalog group, which runs the whole suite inside one
gitops_validate task, and that task has a 600 second deadline. On 2026-09-10
seanfarm #552 ran 602 seconds and was killed by the clock -- not by a failing
guard. Run in parallel the suite finishes in 118 seconds, measured, with the
same 97 guards reporting.

WHY THIS IS SAFE. The guards are hermetic: they read the tree and write only
to their own temporary files. Twenty-two use mktemp. The five that use a fixed
path under /tmp were checked one by one and every name is distinct, so no two
collide.

The three slowest run first. Order is a scheduling hint and nothing more, since
no guard depends on another, but with a tail this long, starting the longest
guard last would add its whole runtime to the end of the run.

EVERY GUARD NOW RUNS, EVEN AFTER ONE FAILS. `set -e` used to stop at the first
failure and hide the rest, so a broken change was found and fixed one guard per
run. Output is replayed in list order once the pool drains, so the log reads
the same every time whatever order they finished in, and each guard's output is
under its own `::: path :::` header.

tests/every-guard-has-a-caller.sh reads this file for its callers and still
counts all 97 as gated by CI: it matches a guard path on any line that is not a
comment, and each path keeps a line of its own.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 87a52d8900 into trunk

2026-09-10 13:36:22 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!569

No description provided.

Rows
Columns