fix(registry-package-prune): move the prune off Argo; it has been dead a day #435
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/package-prune-serviceaccount"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The daily package prune stopped running on 2026-09-07:
Same root cause as the kernel bake: it ran as
build-pipeline, a ServiceAccount this repo never declared -- the Pipelines ArgoCD application created it, Pipelines was retired, and this Flux-managed CronWorkflow kept naming it. Neither repo could catch that. The manifest is valid YAML, kustomize renders it, Flux reported Ready. It surfaced only when the cron next fired, a day later, against the only retention mechanism either registry has -- the knob added after the 2026-08-14 quota incident.Rather than re-declare the ServiceAccount, this moves the prune off Argo, which is being removed from the cluster. The prune is a single container on a schedule: no DAG, no artifact passing, no fan-out. As a native CronJob it also needs no executor RBAC, because there is no executor reporting step results back to a controller -- so the identity is one ServiceAccount and nothing else.
schedulestoschedule,ttlStrategy.secondsAfterCompletiontottlSecondsAfterFinished;concurrencyPolicy: ForbidandstartingDeadlineSecondscarry over unchangedbackoffLimit: 0keeps the old behaviour -- the CronWorkflow carried noretryStrategy, so a failed prune waited for the next day rather than re-running against a registry that just refused itautomountServiceAccountToken: false--prune.pymakes no Kubernetes API call. It mountsforgejo-admin-token, a Forgejo admin credential, so it must not share an identity with the untrusted build lanes eitherThe Kustomization is
prune: true, so Flux garbage-collects the old CronWorkflow. This was the last CronWorkflow in the cluster.Verification
tests/check-package-prune.shgains the identity and no-Argo assertions with three negative controls, all detecting. The guard goes red against the manifest as deployed today, which is the point -- nothing in the tree could see this. Server-side dry run clean;bash tests/check.shexits 0.https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
The daily prune stopped running on 2026-09-07: error in entry template execution: pods "registry-package-prune-1788758220" is forbidden: error looking up service account workflows/build-pipeline: serviceaccount "build-pipeline" not found It ran as `build-pipeline`, a ServiceAccount this repo never declared -- the Pipelines ArgoCD application created it. Pipelines was retired, the ServiceAccount went with it, and this CronWorkflow, which is Flux-managed and lives HERE, kept naming it. Neither repo could catch that: the manifest is valid YAML, kustomize renders it, and Flux reported Ready. It surfaced only when the cron next fired, a day later, against the ONLY retention mechanism either registry has -- the knob added after the 2026-08-14 quota incident. Two other Flux-managed objects name the same dead ServiceAccount, and both are also broken: node-snapshot-bake (the kernel bake -- its Kustomization is Ready=False right now) and pipelines-backup. They are not fixed here. Rather than re-declare the ServiceAccount, this moves the prune off Argo, which is being removed from the cluster. The prune is a single container on a schedule: no DAG, no artifact passing, no fan-out. It never needed a workflow engine. As a native CronJob it also needs no executor RBAC, because there is no executor reporting step results back to a controller -- so the identity is one ServiceAccount and nothing else. * schedules -> schedule, ttlStrategy.secondsAfterCompletion -> ttlSecondsAfterFinished, concurrencyPolicy Forbid and startingDeadlineSeconds carry over unchanged. * backoffLimit: 0 keeps the old behaviour -- the CronWorkflow carried no retryStrategy, so a failed prune waited for the next day rather than re-running against a registry that just refused it. * automountServiceAccountToken: false -- prune.py makes no Kubernetes API call. It mounts forgejo-admin-token, a Forgejo ADMIN credential, so it must not share an identity with the untrusted build lanes either. The Kustomization is prune: true, so Flux garbage-collects the old CronWorkflow. tests/check-package-prune.sh gains the identity and no-Argo assertions with three negative controls, all detecting. The guard goes red against the manifest as it is deployed today, which is the point: nothing in the tree could see this. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76