feat(binjovi): grant the mirror fleet, bind the two ownership lists, and fix the guard that blamed the script #650
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/mirror-grant-ledger"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Replaces #649, which carried a third commit restoring the
miniomirror declarations. MinIO is being retired outright instead, so that commit is gone and the grant here is sized for the fleet that remains.The grant: 38 names, read-only
Binjovi must tell an absent
ForgejoPushMirrorfrom a denied read. WithresourceNamesscoping, a name in the list with no CR returns404 → absent; a name not in the list returns403 → denied. The grant is not permission to act, it is permission to distinguish, and it staysgetonly.While the forgejo-operator CR is live the read returns
200and the legacy gate refuses — so adding a name is a no-op until that CR is deleted. That is what makes it safe to grant the whole fleet in one change and pace the handover by CR deletion.It also has to land first. The bootstrap
createis the only automatically-queued operation: five attempts,30 * 2^(n-1)backoff, terminal after ~7m30s, and nothing re-queues it —bootstrap_manyreturns{:ok, :existing}once the row exists and the drift scan only selectsownership='managed'. Declaring in binjovi before the grant exists strands the resource with no automatic recovery.38 = the 36 mirrors forgejo-operator still owns, plus
binjovi-plansandhetzner-node-image. The twominionames leave with the rest of MinIO; they are still carried by theforgejorepositoriesandforgejocollaboratorsrules and come out in that change, not this one.The two lists prove each other
tests/binjovi-owned-mirrors.txtis the ledger: column 1 the CR name, column 2 the gitmirrors file allowed to carry no mirror document (-when binjovi created the mirror outright). Two rows today.tests/binjovi-service-handoff.shnow requiresas set equality in both directions, plus disjointness, plus: a ledger row's column 2 must name a real file whose mirror document is already gone.
So a declaration cannot leave the tree without its CR name arriving in the ledger, and a name cannot arrive in the ledger while its document is still declared.
This is the check
dc450644did not haveMeasuring the fleet against the cluster for this migration found 38 live CRs against 36 declarations.
dc450644deletedminio.yamlandminio-console.yamlfrom aprune: falseKustomization; both CRs kept running undeclared for a day, still pushing successfully, and nothing went red.tests/gitmirror-fleet.shreads the same ledger, so the two cannot drift, and thecrons >= 28floor is replaced byThe old floor would have had to be lowered at every batch until it stopped meaning anything. This one gets stricter as mirrors move.
The two names a generator gets wrong
ibmi-examplesis the mirror ofsean/rpg-hello;zerotieroneis the mirror ofsean/ZeroTierOne. Both are pinned by hand andrpg-hello/ZeroTierOneare refuted.rpg-hellois the dangerous one: the gate would GET a CR that does not exist, read404as absent, and let binjovi act whileForgejoPushMirror/ibmi-examplesis live. Two owners, neither aware.Second commit: the k3s guard blamed the script for its own watchdog
tests/k3s-registries-contract.shhas gone red intermittently all week — green standalone, red insidetests/check.sh:It was chased three times as a contract failure. It is the harness killing its own fixture.
This gates builds here.
tests/check.shis abuild_checkof theseanfarmproject, in theplatform-fallbackpolicy group, whose paths includekubernetes/**andtests/**— so almost every PR in this repository runs it.The cause.
run_scriptused a 30 s watchdog that sendsTERM. The reconciler trapsTERMwithstop(), which releases the Lease andexit 0. So a watchdog kill and a healthy early return arrive atrun_onceas the same thing —rc=0, output truncated wherever the kill landed.check.shruns guardsnprocways parallel, this one is the longest at ~128 s with an internal pool of its own, and under that oversubscription a one-second run reaches 30 s.The fix, in two parts — the first matters more. The watchdog writes a marker and
run_oncereads it before the exit status, so a kill is reported as a kill and says the output is truncated. Raising the timeout alone would only make the next starved run lie again, more rarely and more confusingly. The marker is written only when the kill lands on a live process.Second, 30 s → 300 s, overridable with
K3S_GUARD_WATCHDOG. It is a deadlock detector, not a time budget: every sleep in the fixture is a fake that returns at once.The control.
watchdog_controlruns serially, likes_sigterm, because its assertion is a timing margin. With the marker check deleted it reports:— the historical message, byte for byte.
Negative controls
ibmi-examplesreplaced byrpg-hellotests/check.sh: exit 0, zero FAIL lines, under the parallel load that was flaking it.https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8
Phase 2 of the push-mirror migration. RBAC only -- no mirror changes owner here. WHAT THE GRANT IS FOR. Binjovi must tell an ABSENT ForgejoPushMirror from a DENIED read. With resourceNames scoping a name IN the list that has no CR returns 404 -> absent, and a name NOT in the list returns 403 -> denied. The grant is not permission to act; it is permission to DISTINGUISH, and it stays `get` only. While the forgejo-operator CR is live the read returns 200 and the legacy gate REFUSES, so adding a name is a no-op until that CR is deleted. That is what makes it safe to grant all 38 at once and pace the handover by CR deletion instead. It also has to land FIRST. The bootstrap create is the only automatically-queued operation: five attempts, 30 * 2^(n-1) backoff, terminal after ~7m30s, and nothing re-queues it -- bootstrap_many returns {:ok, :existing} once the row exists and the drift scan only selects ownership='managed'. Declaring in binjovi before the grant exists strands the resource with no automatic recovery. THE LEDGER. tests/binjovi-owned-mirrors.txt names the mirrors binjovi owns. Column 1 is the CR name, column 2 the gitmirrors file allowed to carry no ForgejoPushMirror document (or `-` when binjovi created the mirror outright and there is no file). Two rows today, the two from3b9ee46c. THE TWO LISTS PROVE EACH OTHER. tests/binjovi-service-handoff.sh now requires (mirror documents in gitmirrors/) + (the ledger) == the resourceNames as set equality in both directions, plus disjointness, plus: a ledger row's column 2 must name a real file whose mirror document is already GONE. So a declaration cannot leave the tree without its CR name arriving in the ledger, and a name cannot arrive in the ledger while its document is still declared. THAT IS THE CHECKdc450644DID NOT HAVE. It deleted minio.yaml and minio-console.yaml from a prune: false Kustomization; both CRs kept running undeclared for a day and nothing went red. Measuring the fleet against the cluster for this migration is what found them -- 38 live CRs against 36 declarations. MinIO is being retired outright rather than re-declared, so the grant here covers the 36 the operator still owns plus the two binjovi already owns; the two minio names leave the other rules with the rest of MinIO. tests/gitmirror-fleet.sh reads the same ledger, so the two cannot drift: a file listed there must have no mirror document, a file not listed must have one, and the `crons >= 28` floor is replaced by operator-owned + handed-to-binjovi == repositories The old floor would have had to be lowered at every batch of the migration until it stopped meaning anything. This one gets STRICTER as mirrors move: a document deleted with no ledger line fails it, and so does a ledger line whose file was deleted outright. The RFC 1123 name check moved ahead of the ownership branch so it still runs for a file that has been handed over. Two CR names are not their repository name -- `ibmi-examples` is sean/rpg-hello and `zerotierone` is sean/ZeroTierOne -- and both are pinned by hand, with `rpg-hello` and `ZeroTierOne` refuted. rpg-hello is the dangerous one: a generator that assumes CR name == repo name makes the gate GET a CR that does not exist, read 404 as absent, and let binjovi act while ForgejoPushMirror/ ibmi-examples is live. Two owners, neither aware. Negative controls, all measured red against this tree: a name dropped from the grant; a bogus name added; ibmi-examples replaced by rpg-hello; a mirror document deleted with no ledger line; a ledger line added while the document is present; a ledger column 2 naming a file that does not exist; and a file listed in the ledger that still declares a mirror. The unmodified tree is green. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8This guard has gone red intermittently all week -- green standalone, red inside tests/check.sh -- with FAIL: script exited 0 instead of finishing its passes: k3s-registries on test-node: reconciling every 60 s (server restart: never) and it was chased three times as a contract failure. It is not one. It is the harness killing its own fixture. IT GATES THIS REPOSITORY'S BUILDS. tests/check.sh is a build_check of the seanfarm project, in the `platform-fallback` policy group, whose paths include `kubernetes/**` and `tests/**` -- so almost every pull request here runs it, and this flake can turn any of them red. THE CAUSE. run_script ran the script under a 30 s watchdog that sends TERM. The reconciler traps TERM with stop(), which releases the Lease and `exit 0` (k3s-registries/manifest.yaml:391-395). So a watchdog kill and a healthy run that returned early arrive at run_once as the SAME thing: rc=0, with the output truncated wherever the kill landed. run_once had only the status to go on, so it reported a contract violation that had not happened, quoting a log that was cut short -- which is why it read like the script never reached its first pass. check.sh runs the guards `nproc` ways parallel and this one is the longest at ~128 s, with an internal worker pool of its own. Under that oversubscription a run that normally takes about a second reaches 30 s and the watchdog fires. THE FIX IS IN TWO PARTS, AND THE FIRST MATTERS MORE. The watchdog now writes a marker, and run_once reads the marker BEFORE the exit status, so a kill is reported as a kill and says the output below is truncated. Raising the timeout without this would only make the next starved run lie again, more rarely and more confusingly. The marker is written only when the kill LANDS on a live process, so a script that finished in the same instant the timer expired is not blamed for it. Second, the watchdog goes 30 s -> 300 s and takes K3S_GUARD_WATCHDOG. It is a DEADLOCK detector, not a time budget: every sleep the script makes is a fake that returns at once, so a healthy run is about a second and nothing honest is anywhere near either number. 300 s still ends a hang well inside the CI step. THE CONTROL. watchdog_control runs serially, for the same reason s_sigterm and s_sigterm_lease do -- its assertion IS a timing margin. SLOW_SLEEP=3 holds the script inside its first pass and K3S_GUARD_WATCHDOG=1 fires under it, so the kill lands every time. It requires run_once to fail AND to name the watchdog, and it refuses to claim proof if the run failed for any other reason. Measured with the marker check deleted, the control reports: FAIL: the watchdog control failed for another reason, so it proves nothing: FAIL: script exited 0 instead of finishing its passes: k3s-registries on test-node: reconciling every 60 s (server restart: never) -- the historical message, byte for byte. That is the diagnosis confirmed and the control proven load-bearing in one run. tests/check.sh now completes green under the parallel load that was flaking it. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8Pull request closed