fix(gitmirrors): two backups were running with no declaration; grant binjovi the fleet #649
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/orphaned-minio-mirrors"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 2 of the push-mirror migration to binjovi — plus a hole the measurement fell into.
Two backups were running with no declaration
Counting the fleet against the cluster found 38 live
ForgejoPushMirrorCRs and 36 declarations. The extras areminioandminio-console.dc450644("feat(minio): delete the operator, the CRDs and the namespace") deleted those two files deliberately — it lists "theminioandminio-consolegit mirrors" among what it removed. The removal did not take. That commit reasoned about prune correctly:…but that was about the minio operator and instance Kustomizations.
gitmirrorsisprune: false. Deleting the files removed the declarations and orphaned the CRs, so for a day both mirrors kept pushing with nothing in this repository describing them — last success2026-09-10T21:09:27Z, emptylast_error, still labelledkustomize.toolkit.fluxcd.io/name=gitmirrors.They come back rather than getting finished off
sean/miniois 139 MB andsean/minio-consoleis 696 MB, both non-empty, neither archived. Deleting the push mirror leaves a live repository with no off-cluster copy, andmake rebuildwipes Forgejo and reseeds from Codeberg — an unbacked repo does not survive the next rebuild. Retiring the repository is the deliberate act that ends this cleanly (deletionPolicy: Delete, pergitmirrors/README.md). Orphaning it from its backup is not that act.No cluster change. The restored specs were diffed against the live CRs first: both
ForgejoPushMirrorspecs match live exactly, and bothForgejoRepositoryspecs differ only bydeletionPolicy,importSource.workflowNamespaceandimportSource.workflowServiceAccount— the three CRD defaults the operator stamps on every repository, shown identically by a declared control file (jitsi.yaml). Flux adopts two objects and writes nothing.The
ForgejoCollaboratordocs are not restored; the pipeline-bot collaborators were retired inf4404df5.The grant: all 40, read-only
Binjovi must tell an absent
ForgejoPushMirrorfrom a denied read. WithresourceNamesscoping, a name in the list with no CR returns404 → absent; a name not in the list returns403 → denied. The grant is not permission to act, it is permission to distinguish, and it staysgetonly.While the forgejo-operator CR is live the read returns
200and the legacy gate refuses — so adding a name is a no-op until that CR is deleted. That is what makes it safe to grant the whole fleet in one change and pace the handover by CR deletion.It also has to land first. The bootstrap
createis the only automatically-queued operation: five attempts,30 * 2^(n-1)backoff, terminal after ~7m30s, and nothing re-queues it —bootstrap_manyreturns{:ok, :existing}once the row exists and the drift scan only selectsownership='managed'. Declaring in binjovi before the grant exists strands the resource with no automatic recovery.The two lists prove each other
tests/binjovi-owned-mirrors.txtis the ledger: column 1 the CR name, column 2 the gitmirrors file allowed to carry no mirror document (-when binjovi created the mirror outright). Two rows today.tests/binjovi-service-handoff.shnow requiresas set equality in both directions, plus disjointness, plus: a ledger row's column 2 must name a real file whose mirror document is already gone.
So a declaration cannot leave the tree without its CR name arriving in the ledger, and a name cannot arrive in the ledger while its document is still declared. This is the check
dc450644did not have.tests/gitmirror-fleet.shreads the same ledger, so the two cannot drift, and thecrons >= 28floor is replaced byThe old floor would have had to be lowered at every batch until it stopped meaning anything. This one gets stricter as mirrors move.
The two names a generator gets wrong
ibmi-examplesis the mirror ofsean/rpg-hello;zerotieroneis the mirror ofsean/ZeroTierOne. Both are pinned by hand andrpg-hello/ZeroTierOneare refuted.rpg-hellois the dangerous one: the gate would GET a CR that does not exist, read404as absent, and let binjovi act whileForgejoPushMirror/ibmi-examplesis live. Two owners, neither aware.Negative controls
All measured red against this tree:
ibmi-examplesreplaced byrpg-hellotests/check.sh: every guard green exceptk3s-registries-contract.sh, the known load-sensitive flake — green standalone, and untouched by this change.https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8
Added a third commit: the
k3s-registries-contract.shflake that failed this PR's first build is fixed, and it was never a contract failure.That guard has gone red intermittently all week — green standalone, red inside
tests/check.sh— withIt was chased three times as a contract failure. It is the harness killing its own fixture.
The cause
run_scriptran the script under a 30 s watchdog that sendsTERM. The reconciler trapsTERMwithstop(), which releases the Lease andexit 0(k3s-registries/manifest.yaml:391-395). So a watchdog kill and a healthy run that returned early arrive atrun_onceas the same thing:rc=0, with the output truncated wherever the kill landed.run_oncehad only the status to go on, so it reported a violation that had not happened and quoted a log that was cut short — which is exactly why it read like the script never reached its first pass.check.shruns the guardsnprocways parallel, this one is the longest at ~128 s, and it has an internal worker pool of its own. Under that oversubscription a run that normally takes about a second reaches 30 s.The fix, in two parts — the first matters more
The watchdog now writes a marker, and
run_oncereads it before the exit status, so a kill is reported as a kill and says the output is truncated. Raising the timeout without this would only make the next starved run lie again — more rarely, and more confusingly. The marker is written only when the kill lands on a live process, so a script that finished in the same instant the timer expired is not blamed.Second, 30 s → 300 s, overridable with
K3S_GUARD_WATCHDOG. It is a deadlock detector, not a time budget: every sleep the script makes is a fake that returns at once, so a healthy run is about a second and nothing honest is near either number.The control
watchdog_controlruns serially, for the same reasons_sigtermands_sigterm_leasedo — its assertion is a timing margin.SLOW_SLEEP=3holds the script inside its first pass andK3S_GUARD_WATCHDOG=1fires under it, so the kill lands every time. It requiresrun_onceto fail and to name the watchdog, and refuses to claim proof if the run failed for any other reason.With the marker check deleted, the control reports:
— the historical message, byte for byte. The diagnosis confirmed and the control proven load-bearing in one run.
tests/check.shnow completes green, exit 0, zero FAIL lines under the same parallel load that was flaking it.https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8
Phase 2 of the push-mirror migration. RBAC only -- no mirror changes owner here. WHAT THE GRANT IS FOR. Binjovi must tell an ABSENT ForgejoPushMirror from a DENIED read. With resourceNames scoping a name IN the list that has no CR returns 404 -> absent, and a name NOT in the list returns 403 -> denied. The grant is not permission to act; it is permission to DISTINGUISH, and it stays `get` only. While the forgejo-operator CR is live the read returns 200 and the legacy gate REFUSES, so adding a name is a no-op until that CR is deleted. That is what makes it safe to grant all 40 at once and pace the handover by CR deletion instead. It also has to land FIRST. The bootstrap create is the only automatically-queued operation: five attempts, 30 * 2^(n-1) backoff, terminal after ~7m30s, and nothing re-queues it -- bootstrap_many returns {:ok, :existing} once the row exists and the drift scan only selects ownership='managed'. Declaring in binjovi before the grant exists strands the resource with no automatic recovery. THE LEDGER. tests/binjovi-owned-mirrors.txt names the mirrors binjovi owns. Column 1 is the CR name, column 2 the gitmirrors file allowed to carry no ForgejoPushMirror document (or `-` when binjovi created the mirror outright and there is no file). Two rows today, the two from3b9ee46c. THE TWO LISTS PROVE EACH OTHER. tests/binjovi-service-handoff.sh now requires (mirror documents in gitmirrors/) + (the ledger) == the resourceNames as set equality in both directions, plus disjointness, plus: a ledger row's column 2 must name a real file whose mirror document is already GONE. So a declaration cannot leave the tree without its CR name arriving in the ledger, and a name cannot arrive in the ledger while its document is still declared. That is the checkdc450644did not have. It deleted minio.yaml and minio-console.yaml from a prune: false Kustomization; both CRs kept running undeclared for a day and nothing went red. tests/gitmirror-fleet.sh reads the same ledger, so the two cannot drift: a file listed there must have no mirror document, a file not listed must have one, and the `crons >= 28` floor is replaced by operator-owned + handed-to-binjovi == repositories The old floor would have had to be lowered at every batch of the migration until it stopped meaning anything. This one gets STRICTER as mirrors move: a document deleted with no ledger line fails it, and so does a ledger line whose file was deleted outright. The RFC 1123 name check moved ahead of the ownership branch so it still runs for a file that has been handed over. Two CR names are not their repository name -- `ibmi-examples` is sean/rpg-hello and `zerotierone` is sean/ZeroTierOne -- and both are pinned by hand, with `rpg-hello` and `ZeroTierOne` refuted. rpg-hello is the dangerous one: a generator that assumes CR name == repo name makes the gate GET a CR that does not exist, read 404 as absent, and let binjovi act while ForgejoPushMirror/ ibmi-examples is live. Two owners, neither aware. Negative controls, all measured red against this tree: a name dropped from the grant; a bogus name added; ibmi-examples replaced by rpg-hello; a mirror document deleted with no ledger line; a ledger line added while the document is present; a ledger column 2 naming a file that does not exist; and a file listed in the ledger that still declares a mirror. The unmodified tree is green. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8This guard has gone red intermittently all week -- green standalone, red inside tests/check.sh -- with FAIL: script exited 0 instead of finishing its passes: k3s-registries on test-node: reconciling every 60 s (server restart: never) and it was chased three times as a contract failure. It is not one. It is the harness killing its own fixture. THE CAUSE. run_script ran the script under a 30 s watchdog that sends TERM. The reconciler traps TERM with stop(), which releases the Lease and `exit 0` (k3s-registries/manifest.yaml:391-395). So a watchdog kill and a healthy run that returned early arrive at run_once as the SAME thing: rc=0, with the output truncated wherever the kill landed. run_once had only the status to go on, so it reported a contract violation that had not happened, quoting a log that was cut short -- which is why it read like the script never reached its first pass. check.sh runs the guards `nproc` ways parallel and this one is the longest at ~128 s, with an internal worker pool of its own. Under that oversubscription a run that normally takes about a second reaches 30 s and the watchdog fires. THE FIX IS IN TWO PARTS, AND THE FIRST MATTERS MORE. The watchdog now writes a marker, and run_once reads the marker BEFORE the exit status, so a kill is reported as a kill and says the output below is truncated. Raising the timeout without this would only make the next starved run lie again, more rarely and more confusingly. The marker is written only when the kill LANDS on a live process, so a script that finished in the same instant the timer expired is not blamed for it. Second, the watchdog goes 30 s -> 300 s and takes K3S_GUARD_WATCHDOG. It is a DEADLOCK detector, not a time budget: every sleep the script makes is a fake that returns at once, so a healthy run is about a second and nothing honest is anywhere near either number. 300 s still ends a hang well inside the CI step. THE CONTROL. watchdog_control runs serially, for the same reason s_sigterm and s_sigterm_lease do -- its assertion IS a timing margin. SLOW_SLEEP=3 holds the script inside its first pass and K3S_GUARD_WATCHDOG=1 fires under it, so the kill lands every time. It requires run_once to fail AND to name the watchdog, and it refuses to claim proof if the run failed for any other reason. Measured with the marker check deleted, the control reports: FAIL: the watchdog control failed for another reason, so it proves nothing: FAIL: script exited 0 instead of finishing its passes: k3s-registries on test-node: reconciling every 60 s (server restart: never) -- the historical message, byte for byte. That is the diagnosis confirmed and the control proven load-bearing in one run. tests/check.sh now completes green under the parallel load that was flaking it. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8Pull request closed