chore(argo): delete the Argo CRDs and the RBAC that only Workflows used #462

Closed
sean wants to merge 1 commit from sean/retire-argo-crds into main AGit
Owner

Eleven CRDs -- three Argo Events, eight Argo Workflows. The three ArgoCD CRDs share the argoproj.io group and stay; 28 live Applications need them. The crds Flux Kustomization is prune: false on purpose (its own comment: CRITICAL, never prune CRDs), so a tombstone Job does the delete with the eleven names pinned in resourceNames, and it WAITS up to fifteen minutes for both controllers to be gone rather than refusing, so this can be released alongside the controller retirement. Also fourteen RBAC objects that existed only to run a Workflow pod, including a standing flux-system write grant held by an unused ServiceAccount. binjovi-deploy-argocd stays: it shares their naming but the live native binjovi-argocd-agent runs as it, and the tombstone cannot reach it. While rewriting the guard: every '! grep' refutation in tests/binjovi-argo-execution.sh was INERT, because bash's set -e does not fire when a return value is inverted with '!'. They now go through a helper that exits -- which caught a mistake in this very commit on its first run. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Eleven CRDs -- three Argo Events, eight Argo Workflows. The three ArgoCD CRDs share the argoproj.io group and stay; 28 live Applications need them. The crds Flux Kustomization is prune: false on purpose (its own comment: CRITICAL, never prune CRDs), so a tombstone Job does the delete with the eleven names pinned in resourceNames, and it WAITS up to fifteen minutes for both controllers to be gone rather than refusing, so this can be released alongside the controller retirement. Also fourteen RBAC objects that existed only to run a Workflow pod, including a standing flux-system write grant held by an unused ServiceAccount. binjovi-deploy-argocd stays: it shares their naming but the live native binjovi-argocd-agent runs as it, and the tombstone cannot reach it. While rewriting the guard: every '! grep' refutation in tests/binjovi-argo-execution.sh was INERT, because bash's set -e does not fire when a return value is inverted with '!'. They now go through a helper that exits -- which caught a mistake in this very commit on its first run. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
Argo Events reconciles nothing. `kubectl get sensors,eventsources,eventbus -A`
returns no resources anywhere in the cluster: the Pipelines EventSource that
used to trigger builds went with Pipelines, and Binjovi never used Argo Events.
One `controller-manager` Deployment has been idling in ns `argo-events` since.

Removing the wrapper is not enough on its own. The `cluster` Flux Kustomization
that delivers it is `prune: false`, so deleting the file stops Flux MANAGING
the live `argo-events` Kustomization without deleting it, and the controller
would keep running with nothing in the tree declaring it. That is exactly what
happened to the four observability agents in 2026-08, whose Kustomization
objects had to be deleted separately afterwards.

So this is two halves that must land together: the tree stops declaring
argo-events, and a tombstone Job deletes the live Kustomization object. That
one delete is enough, because the `argo-events` Kustomization is itself
`prune: true` -- Flux garbage-collects the controller, its RBAC and the
namespace around them. The Job's namespace grant is READ ONLY on purpose: Flux
performs the delete and the Job only waits for the namespace to disappear, so a
prune that never ran cannot hide behind a namespace the Job removed itself.

The CRDs stay for a later release. Deleting a CRD while its controller still
runs leaves a finalizer wedge, so `crds/argo-events-*.yaml` goes only after
this Job has run and the controller is confirmed gone.

`tests/argo-events-retired.sh` pins both halves and gives every assertion a
negative control: a widened RBAC grant, a missing prune proof and a namespace
delete grant each turn it red on their own.

The upstream inventory loses the `argo-events` row and the `natsio/` allow
line, whose only subject was the vendored manifest this deletes; no image under
that prefix is left in the tree.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
ArgoCD Application `argo-workflows` syncs the engine into namespace
`workflows`: two `workflows-argo-workflows-server` replicas, two
`workflows-argo-workflows-workflow-controller` replicas, the
`argo-otlp-exporter`, the workflows.sean.farm HTTPRoutes and twelve
Pipelines-era ExternalSecrets -- 70 objects.

Nothing creates a Workflow any more. `builder-images` went native, the
`zot-src` shadow entry never created a Workflow object at all, and the
`argo-workflows` project left `sean/binjovi-plans` in the release before this
one, so Binjovi has already stopped tending the Application. In ArgoCD's own
terms it was orphaned long ago: its tracking-id names `pipelines-argocd`, an
app-of-apps deleted with Pipelines.

THE FINALIZER IS THE POINT. The Application carries no
`resources-finalizer.argocd.argoproj.io` and runs `syncPolicy.automated` with
`selfHeal`, so deleting it as it stands orphans all 70 objects and leaves them
running, and deleting them first just makes ArgoCD put them back. Adding the
finalizer before the delete makes ArgoCD cascade, which also means the
Application's own resource list decides what goes -- safer than this Job
enumerating seventy names and getting one wrong, and it keeps the RBAC to a
single object.

THE NAMESPACE SURVIVES. `workflows` also holds the five live Binjovi agents
and the registry-package-prune CronJob, none of them owned by this
Application. `CreateNamespace=true` made the namespace without tracking it, so
the cascade cannot reach it -- verified against the live Application, whose
resource list names no Namespace at all. The Job asserts that before it patches
anything and refuses if it ever stops being true, asserts the five agents are
running before, and asserts they are still running after. Its grant in ns
`workflows` is read-only: a botched run must not be able to remove the very
agents the post-condition protects. Retiring Pipelines broke a ServiceAccount
and two Forgejo tokens in this same namespace, which is the failure these
pre- and post-conditions exist to catch.

The CRDs stay for a later release. A CRD deleted while its controller still
runs leaves a finalizer wedge, so `crds/argo-workflows-*.yaml` and the 64
finished Workflow records go only after this Job has run.

`tests/argo-workflows-retired.sh` pins the cascade, the read-only namespace
grant, the survivor list and the before-and-after check, parses the inline
script under `sh`, and gives each of them a negative control: a lost finalizer,
a delete grant in ns `workflows` and a dropped survivor each turn it red alone.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
chore(pipelines): archive sean/pipelines
All checks were successful
binjovi/ci Binjovi completed the frozen plan
d6639f8a35
Nothing deploys from it. It sources 0 ArgoCD Applications and 0 Flux
GitRepositories, and every target its Binjovi entry named is already gone --
all five Applications (pipeline-base, pipelines, pipelines-operator,
pipelines-project-registry, workflows-infra-bare) and all four image
Deployments (pipelines-operator-controller-manager, pipelines,
pipelines-release-api, pipelines-mcp), verified NotFound against the live
cluster. Its ~25 pipeline templates, ~17 golden fixtures and 6 guards that name
MinIO are dead code; archiving the repo removes every one of those MinIO
references at once, with no endpoint edit.

BACKUP CONFIRMED FIRST, per gitmirrors/README.md. Forgejo and codeberg agree
exactly on main and trunk (3f02028f) and NO ref exists only on Forgejo. 261
refs exist only on codeberg: the mirror pushes main and trunk with no delete
refspec, so codeberg keeps branches Forgejo has since removed. The README's
`diff ... || STOP` therefore reports a difference for this repo no matter what
-- read the DIRECTION, not the exit code. A codeberg superset is the safe one;
a ref only on Forgejo is the failure that check exists for.

`gitmirrors` is prune: false on purpose -- a bad render under prune: true would
delete every backup mirror in the fleet and stop the codeberg backups silently
-- so removing the file only stops Flux managing the two CRs. A tombstone Job
releases them, push mirror first, because that is the one still writing to
codeberg while the repository CR exists.

THE CODE IS NOT DELETED. Both CRs carry deletionPolicy: Retain, so releasing
them is a no-op on the Forgejo side. The Job re-reads that from the LIVE object
and refuses if either says Delete, rather than trusting the manifest that used
to declare it: before forgejo-operator v0.6.34 the ForgejoRepository finalizer
called DeleteRepo() and destroyed all code, refs and history. What does change
is a cold rebuild -- with no ForgejoRepository there is no importSource, so
`make rebuild` will not re-seed sean/pipelines. That is what archiving it
means, and codeberg keeps the code.

`tests/pipelines-archived.sh` pins the Retain check, the live read, the pinned
RBAC and the release order, parses the inline script under `sh`, and turns red
on a removed policy check or a widened grant.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
RELEASE THIS AFTER the controllers are gone, not with them. Both tombstones
here refuse until they are, but the ordering is real: a CRD deleted while its
controller still watches it wedges on a finalizer.

ELEVEN CRDs. Three from Argo Events, eight from Argo Workflows. The three
ArgoCD CRDs share the argoproj.io group and stay -- 28 live Applications need
them. The `crds` Flux Kustomization is prune: false and says why in its own
comment ("CRITICAL: Never prune CRDs to prevent data loss"), so removing the
files stops Flux re-applying them and nothing more; a tombstone Job does the
delete, with the eleven names pinned in `resourceNames` so RBAC is the fence
whatever the script says. Removing workflows.argoproj.io garbage-collects the
64 finished Workflow records and their pods, on purpose.

FOURTEEN RBAC OBJECTS. Three ServiceAccounts with no running pod, four
`*-executor` RoleBindings bound to a Role the Argo cascade deletes,
Role/RoleBinding binjovi-deploy-flux in ns flux-system -- a standing write
grant on Flux objects held by an account nothing uses, while the native flux
agents have their own -- and five binjovi-argo-* CiliumNetworkPolicies that
select isolation labels NO RUNNING POD carries.

THE ASYMMETRY IS THE RISK. binjovi-deploy-argocd was created for Argo like all
of them and shares their naming, but the live native binjovi-argocd-agent pod
runs as it. It stays; its `-executor` RoleBinding does not. The tombstone
cannot reach it -- it is absent from every resourceNames list -- and the Job
asserts it is present before and after. `binjovi-control-workflows` keeps its
pod and log reads and loses its `argoproj.io/workflows` rule, which is standing
privilege on a kind that no longer exists.

The file is renamed to what it now is: native-deploy-rbac.yaml.

GUARDS. `tests/binjovi-argo-execution.sh` was two guards in one. Its second
half pins the live Kyverno credential policy that protects the package and RPG
lanes, so it survives as tests/binjovi-untrusted-credential-guard.sh; the first
half is replaced by assertions that the retired names cannot return.

While rewriting it: every `! grep` refutation in that guard was INERT. Bash
documents that `set -e` does not exit "if the command's return value is being
inverted with !", so a dozen refutations reported nothing and passed whatever
the file said. They now go through a `refute` helper that exits. It caught a
mistake in this very commit on its first run -- an over-broad refutation of
`apiGroups: [argoproj.io]`, which ArgoCD Applications also need.

tests/argo-crds-retired.sh and tests/binjovi-argo-rbac-retired.sh pin the rest,
each with negative controls: an ArgoCD CRD inside the delete grant, a removed
controller pre-check, a deleted ArgoCD CRD file, a grant widened to the live
agent's ServiceAccount, an unpinned rule, and a dropped post-condition each
turn a guard red on its own.

`tests/argo-workflows-crds/run-all.sh` pinned the v4.0.8 CRD schemas by digest
and goes with them, along with its entries in run-all.sh and
every-guard-has-a-caller.sh.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
sean force-pushed sean/retire-argo-crds from 1b8663a48a to 316e18d1f5
Some checks failed
binjovi/ci Binjovi failed the frozen plan
2026-09-08 18:17:07 +00:00
Compare
sean closed this pull request 2026-09-09 13:13:55 +00:00
Some checks failed
binjovi/ci Binjovi failed the frozen plan

Pull request closed

Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/seanfarm!462
No description provided.