fix(workflows): declare the Secrets the bake and the prune mount #439

Merged

binjovi-bot merged 1 commit from fix/workflows-forgejo-tokens into trunk 2026-09-07 23:30:25 +00:00

Conversation 0 Commits 1 Files changed 10 +224 -4

sean commented 2026-09-07 23:24:24 +00:00

Owner

Copy link

Moving the bake and the prune off Argo fixed their ServiceAccount and left a second, quieter break behind. The bake reached the cluster and stopped dead:

MountVolume.SetUp failed for volume "forgejo" :
secret "forgejo-pipeline-bot-token" not found

Neither Secret was declared in this repo. Both exist only in ns forgejo, and the ExternalSecrets that mirrored them into ns workflows lived in sean/pipelines and died with that application -- the same cause as the ServiceAccount, one layer down. registry-package-prune had the identical gap on forgejo-admin-token and would have failed at 05:17 rather than now.

Why this one is nastier than the ServiceAccount

A missing ServiceAccount refuses pod creation and says so in the event. A missing Secret lets the pod be created and leaves kubelet retrying the mount, so it sits in PodInitializing with no container, no logs, and no stated cause until activeDeadlineSeconds burns.

node-snapshot-bake's own header blames an earlier death on "the container never started". That is the shape a missing Secret makes -- r3 may well have been this.

A third gap, found while auditing the first two

The codeberg-creds ExternalSecret in ns workflows carries argocd.argoproj.io/tracking-id: argo-workflows:.... It belongs to the ArgoCD application we are retiring, and both Jobs mount the Secret it produces.

Its target.deletionPolicy is Retain, so the Secret would have outlived the ArgoCD prune and hidden the gap until the next make rebuild -- exactly the failure the north star forbids. It is declared here now, under the same name so the existing object is adopted rather than raced by a second ExternalSecret. Expect ArgoCD to show argo-workflows OutOfSync on ownership annotations until that app is deleted.

hcloud-token and bake-ssh-key were checked too and are fine: SOPS sources in this repo, applied to ns workflows by Flux targetNamespace.

r8

The r7 Job is wedged on the missing mount and carries backoffLimit: 0, so it will not recover on its own. An rN bump is the declarative retry this manifest already asks for, in preference to deleting the Job by hand.

Guard

tests/workflows-secret-declarations.sh fails when a Job or CronJob in either component mounts a Secret that no manifest in this repo declares -- the class, not the three instances. Scoped to these two components on purpose: several Secrets here come from SOPS sources or other components, and a scanner that guesses namespaces produces false positives that get silenced, which is worse than no guard.

Three negative controls, each hiding one declaring file from the repo-wide scan. The first version deleted the file from a copy of the component and was inert, because the real declaration was still on disk and still satisfied the scan.

bash tests/check.sh exits 0. Server-side dry run applies all seven objects clean.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Moving the bake and the prune off Argo fixed their ServiceAccount and left a second, quieter break behind. The bake reached the cluster and stopped dead: ``` MountVolume.SetUp failed for volume "forgejo" : secret "forgejo-pipeline-bot-token" not found ``` Neither Secret was declared in this repo. Both exist only in `ns forgejo`, and the ExternalSecrets that mirrored them into `ns workflows` lived in `sean/pipelines` and died with that application -- the same cause as the ServiceAccount, one layer down. `registry-package-prune` had the identical gap on `forgejo-admin-token` and would have failed at 05:17 rather than now. ## Why this one is nastier than the ServiceAccount A missing ServiceAccount **refuses pod creation** and says so in the event. A missing Secret lets the pod be created and leaves kubelet retrying the mount, so it sits in `PodInitializing` with no container, no logs, and no stated cause until `activeDeadlineSeconds` burns. `node-snapshot-bake`'s own header blames an earlier death on *"the container never started"*. That is the shape a missing Secret makes -- r3 may well have been this. ## A third gap, found while auditing the first two The `codeberg-creds` ExternalSecret in `ns workflows` carries `argocd.argoproj.io/tracking-id: argo-workflows:...`. It belongs to **the ArgoCD application we are retiring**, and *both* Jobs mount the Secret it produces. Its `target.deletionPolicy` is `Retain`, so the Secret would have outlived the ArgoCD prune and hidden the gap until the next `make rebuild` -- exactly the failure the north star forbids. It is declared here now, under the **same name** so the existing object is adopted rather than raced by a second ExternalSecret. Expect ArgoCD to show `argo-workflows` OutOfSync on ownership annotations until that app is deleted. `hcloud-token` and `bake-ssh-key` were checked too and are fine: SOPS sources in this repo, applied to `ns workflows` by Flux `targetNamespace`. ## r8 The r7 Job is wedged on the missing mount and carries `backoffLimit: 0`, so it will not recover on its own. An rN bump is the declarative retry this manifest already asks for, in preference to deleting the Job by hand. ## Guard `tests/workflows-secret-declarations.sh` fails when a Job or CronJob in either component mounts a Secret that no manifest in this repo declares -- the **class**, not the three instances. Scoped to these two components on purpose: several Secrets here come from SOPS sources or other components, and a scanner that guesses namespaces produces false positives that get silenced, which is worse than no guard. Three negative controls, each hiding one declaring **file** from the repo-wide scan. The first version deleted the file from a *copy* of the component and was inert, because the real declaration was still on disk and still satisfied the scan. `bash tests/check.sh` exits 0. Server-side dry run applies all seven objects clean. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit 2026-09-07 23:24:24 +00:00

fix(workflows): declare the Secrets the bake and the prune mount ... 98d8c77a6b

Moving both off Argo fixed their ServiceAccount and left a second, quieter
break behind. The bake reached the cluster and stopped dead:

  MountVolume.SetUp failed for volume "forgejo" :
  secret "forgejo-pipeline-bot-token" not found

Neither Secret was declared in this repo. Both exist only in ns forgejo, and the
ExternalSecrets that mirrored them into ns workflows lived in sean/pipelines and
died with that application -- the same cause as the ServiceAccount, one layer
down. registry-package-prune had the identical gap on forgejo-admin-token and
would have failed at 05:17 rather than now.

This failure is much quieter than the ServiceAccount one. A missing
ServiceAccount refuses pod CREATION and says so. A missing Secret lets the pod be
created and leaves kubelet retrying the mount, so it sits in PodInitializing with
no container, no logs, and no stated cause until activeDeadlineSeconds burns.
node-snapshot-bake's own header blames an earlier death on "the container never
started"; that is the shape a missing Secret makes, so r3 may have been this.

A third gap came out of auditing for the first two: the codeberg-creds
ExternalSecret in ns workflows carries
argocd.argoproj.io/tracking-id: argo-workflows:... It belongs to the ArgoCD
application we are retiring, and BOTH Jobs mount the Secret it produces. Its
target.deletionPolicy is Retain, so the Secret would have outlived the prune and
hidden the gap until the next `make rebuild` -- exactly the failure the north
star forbids. It is declared here now, under the same name so the existing
object is adopted rather than raced by a second ExternalSecret. Expect ArgoCD to
show argo-workflows OutOfSync on ownership annotations until it is deleted.

hcloud-token and bake-ssh-key were checked too and are fine: SOPS sources in
this repo, applied to ns workflows by Flux targetNamespace.

The bake goes to r8. The r7 Job is wedged on the missing mount and carries
backoffLimit: 0, so it will not recover on its own; an rN bump is the
declarative retry this manifest already asks for, in preference to deleting the
Job by hand.

tests/workflows-secret-declarations.sh is new. It fails when a Job or CronJob in
either component mounts a Secret that no manifest in this repo declares -- the
class of bug, not the three instances. It is scoped to these two components on
purpose: several Secrets here come from SOPS sources or other components, and a
scanner that guesses namespaces produces false positives that get silenced,
which is worse than no guard. Three negative controls, each hiding one declaring
FILE from the repo-wide scan; the first version deleted the file from a COPY of
the component and was inert, because the real declaration was still on disk.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 98d8c77a6b into trunk 2026-09-07 23:30:25 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!439

No description provided.