feat(minio): eradicate sean/minio and sean/minio-console #651
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/eradicate-minio"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stacked on #650. MinIO is retired; this removes the two source repositories.
They were still running, undeclared
dc450644removed the MinIO tenant, operator, CRDs and namespace, and said it removed "theminioandminio-consolegit mirrors" too. It did not.gitmirrorsisprune: false, so deleting the two files removed the declarations and orphaned the CRs. Both repositories kept running and kept pushing to Codeberg for a day — last success2026-09-10T21:09:27Z, emptylast_error.Measuring the fleet against the cluster for the push-mirror migration is what found them: 38 live
ForgejoPushMirrorCRs against 36 declarations. Nothing in the tree consumes either repository.This tombstone destroys, and it is the first one here that does
Every other tombstone releases a CR and proves the repository survives;
require_retain()exists to make that impossible to get wrong. This one flipsspec.deletionPolicytoDeleteand then deletes theForgejoRepository, which destroys it at Forgejo. The fence is inverted with it:require_retainis the wrong shape when refusing to destroy is exactly what must not happen. The Job reads the live object and requiresspec.owner == seanandspec.name ==the expected name. A retargeted or unreadable CR halts the whole Job before any policy is flipped — the second name is not safe to touch if the first is not what it claims.repositoryIDis printed, not enforced (minio=8,minio-console=11). A constant id fence went stale once already, on hello-lib, and a stale fence that refuses is only marginally better than one that permits.jq, not by pattern-matching the PATCH response. hello-lib's tombstone refused four times with "policy did not take" while the policy had taken, because it discarded the response body and grepped for a substring.patchappears onforgejorepositoriesonly. The push-mirror migration's tombstones getget, deleteand must never be able to destroy a repository. The two must not share a Role.Codeberg and GitHub are not touched from here.
someara/minioandsomeara/minio-consoleexist on both and have to go, but an irreversible external write does not belong in a Job that Flux can re-run. They go by hand once this Job reports the Forgejo side gone.The counts were measured, not guessed
Removing two names moves six pinned numbers, and the sixth is the one that caught it —
repository-bootstrap.yaml's Jobcompletionsis asserted equal to the record count.forgejorepositoriesresourceNamesforgejocollaboratorsresourceNamesrepository-bootstrap.jsonrecordssourcemanagement-grants-repositories.txtrepository-bootstrap.yamlcompletionsminio-resource-operatorkeeps its names in all of these. It isdc450644's "Stage 5b" — still deployed, still wired intocrds,image-automationandshipwright, managing zero custom resources cluster-wide — and it is its own change.The tombstone directory was unguarded
tests/retirement-jobs-are-replaceable.shwatched onlypipelines-retirement. A Job inrepository-retirement— the directory this tombstone and the whole push-mirror migration use — could lose itskustomize.toolkit.fluxcd.io/forceannotation and nothing would say so. Without it a Job's immutable pod template cannot be corrected by Flux: the dry-run fails, the Kustomization goesReady: False, and nothing in it reconciles.It now covers both directories, requires at least one Job across them so an empty glob cannot pass vacuously, and lets a directory be empty only if it holds no Job at all.
forceannotationrepository-retirementemptied againtests/check.sh: exit 0, zero FAIL lines.https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8
Phase 2 of the push-mirror migration. RBAC only -- no mirror changes owner here. WHAT THE GRANT IS FOR. Binjovi must tell an ABSENT ForgejoPushMirror from a DENIED read. With resourceNames scoping a name IN the list that has no CR returns 404 -> absent, and a name NOT in the list returns 403 -> denied. The grant is not permission to act; it is permission to DISTINGUISH, and it stays `get` only. While the forgejo-operator CR is live the read returns 200 and the legacy gate REFUSES, so adding a name is a no-op until that CR is deleted. That is what makes it safe to grant all 38 at once and pace the handover by CR deletion instead. It also has to land FIRST. The bootstrap create is the only automatically-queued operation: five attempts, 30 * 2^(n-1) backoff, terminal after ~7m30s, and nothing re-queues it -- bootstrap_many returns {:ok, :existing} once the row exists and the drift scan only selects ownership='managed'. Declaring in binjovi before the grant exists strands the resource with no automatic recovery. THE LEDGER. tests/binjovi-owned-mirrors.txt names the mirrors binjovi owns. Column 1 is the CR name, column 2 the gitmirrors file allowed to carry no ForgejoPushMirror document (or `-` when binjovi created the mirror outright and there is no file). Two rows today, the two from3b9ee46c. THE TWO LISTS PROVE EACH OTHER. tests/binjovi-service-handoff.sh now requires (mirror documents in gitmirrors/) + (the ledger) == the resourceNames as set equality in both directions, plus disjointness, plus: a ledger row's column 2 must name a real file whose mirror document is already GONE. So a declaration cannot leave the tree without its CR name arriving in the ledger, and a name cannot arrive in the ledger while its document is still declared. THAT IS THE CHECKdc450644DID NOT HAVE. It deleted minio.yaml and minio-console.yaml from a prune: false Kustomization; both CRs kept running undeclared for a day and nothing went red. Measuring the fleet against the cluster for this migration is what found them -- 38 live CRs against 36 declarations. MinIO is being retired outright rather than re-declared, so the grant here covers the 36 the operator still owns plus the two binjovi already owns; the two minio names leave the other rules with the rest of MinIO. tests/gitmirror-fleet.sh reads the same ledger, so the two cannot drift: a file listed there must have no mirror document, a file not listed must have one, and the `crons >= 28` floor is replaced by operator-owned + handed-to-binjovi == repositories The old floor would have had to be lowered at every batch of the migration until it stopped meaning anything. This one gets STRICTER as mirrors move: a document deleted with no ledger line fails it, and so does a ledger line whose file was deleted outright. The RFC 1123 name check moved ahead of the ownership branch so it still runs for a file that has been handed over. Two CR names are not their repository name -- `ibmi-examples` is sean/rpg-hello and `zerotierone` is sean/ZeroTierOne -- and both are pinned by hand, with `rpg-hello` and `ZeroTierOne` refuted. rpg-hello is the dangerous one: a generator that assumes CR name == repo name makes the gate GET a CR that does not exist, read 404 as absent, and let binjovi act while ForgejoPushMirror/ ibmi-examples is live. Two owners, neither aware. Negative controls, all measured red against this tree: a name dropped from the grant; a bogus name added; ibmi-examples replaced by rpg-hello; a mirror document deleted with no ledger line; a ledger line added while the document is present; a ledger column 2 naming a file that does not exist; and a file listed in the ledger that still declares a mirror. The unmodified tree is green. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8This guard has gone red intermittently all week -- green standalone, red inside tests/check.sh -- with FAIL: script exited 0 instead of finishing its passes: k3s-registries on test-node: reconciling every 60 s (server restart: never) and it was chased three times as a contract failure. It is not one. It is the harness killing its own fixture. IT GATES THIS REPOSITORY'S BUILDS. tests/check.sh is a build_check of the seanfarm project, in the `platform-fallback` policy group, whose paths include `kubernetes/**` and `tests/**` -- so almost every pull request here runs it, and this flake can turn any of them red. THE CAUSE. run_script ran the script under a 30 s watchdog that sends TERM. The reconciler traps TERM with stop(), which releases the Lease and `exit 0` (k3s-registries/manifest.yaml:391-395). So a watchdog kill and a healthy run that returned early arrive at run_once as the SAME thing: rc=0, with the output truncated wherever the kill landed. run_once had only the status to go on, so it reported a contract violation that had not happened, quoting a log that was cut short -- which is why it read like the script never reached its first pass. check.sh runs the guards `nproc` ways parallel and this one is the longest at ~128 s, with an internal worker pool of its own. Under that oversubscription a run that normally takes about a second reaches 30 s and the watchdog fires. THE FIX IS IN TWO PARTS, AND THE FIRST MATTERS MORE. The watchdog now writes a marker, and run_once reads the marker BEFORE the exit status, so a kill is reported as a kill and says the output below is truncated. Raising the timeout without this would only make the next starved run lie again, more rarely and more confusingly. The marker is written only when the kill LANDS on a live process, so a script that finished in the same instant the timer expired is not blamed for it. Second, the watchdog goes 30 s -> 300 s and takes K3S_GUARD_WATCHDOG. It is a DEADLOCK detector, not a time budget: every sleep the script makes is a fake that returns at once, so a healthy run is about a second and nothing honest is anywhere near either number. 300 s still ends a hang well inside the CI step. THE CONTROL. watchdog_control runs serially, for the same reason s_sigterm and s_sigterm_lease do -- its assertion IS a timing margin. SLOW_SLEEP=3 holds the script inside its first pass and K3S_GUARD_WATCHDOG=1 fires under it, so the kill lands every time. It requires run_once to fail AND to name the watchdog, and it refuses to claim proof if the run failed for any other reason. Measured with the marker check deleted, the control reports: FAIL: the watchdog control failed for another reason, so it proves nothing: FAIL: script exited 0 instead of finishing its passes: k3s-registries on test-node: reconciling every 60 s (server restart: never) -- the historical message, byte for byte. That is the diagnosis confirmed and the control proven load-bearing in one run. tests/check.sh now completes green under the parallel load that was flaking it. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8