fix(node-snapshot-bake): the kernel bake is down; move it off Argo to a Job #434
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/node-snapshot-bake-native"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The kernel bake has not run since 2026-09-06 23:28. Every attempt dies before the first container starts:
It ran as
build-pipeline, a ServiceAccount this repo never declared -- the Pipelines ArgoCD application created it. Pipelines was retired and the ServiceAccount went with it. This is the kernel roll's last automatic step, so the AGENTS.md north star has been broken for a day, andnode-snapshot-bakeis one of only two Kustomizations not Ready cluster-wide.Argo is being removed from the cluster, so this does not re-declare the ServiceAccount. The bake never needed a workflow engine: the DAG is a straight line, and initContainers run in order and stop the pod at the first non-zero exit -- which is exactly
depends: <previous>.Succeeded.The four step scripts are carried over byte-for-byte
Verified by parsing both manifests and diffing normalised script bodies:
missing=0 extra=0on all four. The only addition is the 4-line cost guard in smoke. Three things moved:{{workflow.parameters.kernel-tag}}${KERNEL_TAG}$stays$$, checked for any single-$Flux would eat/tmp/snapid/work: each container has its own/tmp, and on/tmppromote would commit an empty snapshot id intonodes.envwhen: did-bake == 'true'exit 0in smokeThat last one is not cosmetic. Under Argo, promote had to match both
smoke.Succeededandsmoke.Skipped; when it once matchedOmittedinstead, promote and prune were silently skipped on a run Argo still reported Succeeded. Sequential initContainers cannot express that: promote runs if and only if smoke exits 0.The TTL, and a correction
I first asserted the Job must carry no
ttlSecondsAfterFinished, reasoning that a TTL lets Flux re-create and re-bake forever.tests/job-ttl-contract.shcaught it, and that guard is right: a finish timeout does mean a daily re-run, and it is correct only for a Job that converges.This one converges, by the design already in it:
ensure-snapshotfinds the snapshot for this tag and skips the bake -- twohcloud image listcalls, no VMsmokeexits 0, because did-bake is falsepromotesedsnodes.env, sees an empty diff, pushes nothingprunekeeps the newest 4; deleting nothing is its steady stateThe current tag's snapshot is always the newest, so prune can never remove the one
ensure-snapshotlooks for -- which is what stops the daily re-run becoming a daily VM bake. The guard now pins those four properties instead of forbidding the TTL.Verification
tests/node-snapshot-bake.shis new: native Job, no Argo, identity declared in-repo, the DAG order (smoke before promote is what stops a bad snapshot reachingnodes.env), the/workhand-off, and the four convergence properties. Eight negative controls, all detecting. It goes red against the manifest as deployed today:FAIL: the bake must be a native Kubernetes Job.check-url-credentials.shandhouse-helper-images/run-all.shfollow the rename and moved lines. The credential guard usefully flagged both the new sites and the now-stale allow-list entries.job.batch/node-snapshot-bake-seanfarm45-r7 serverside-applied.bash tests/check.shexits 0.Contract revision goes to r7, which is also what gives the fix a fresh run instead of re-checking the r6 corpse.
https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
The bake has not run since 2026-09-06 23:28. Every attempt dies before the first container starts: task 'node-snapshot-bake-seanfarm45-r6.ensure-snapshot' errored: pods "...-ensure-snapshot-3602098703" is forbidden: error looking up service account workflows/build-pipeline: serviceaccount "build-pipeline" not found It ran as `build-pipeline`, a ServiceAccount this repo never declared -- the Pipelines ArgoCD application created it. Pipelines was retired and the ServiceAccount went with it. This is the kernel roll's last automatic step, so AGENTS.md's north star has been broken for a day, and the node-snapshot-bake Kustomization is one of only two that are not Ready cluster-wide. Argo is being removed from the cluster, so this does not re-declare the ServiceAccount. The bake never needed a workflow engine: the DAG is a straight line, and initContainers run in order and stop the pod at the first non-zero exit, which is exactly `depends: <previous>.Succeeded`. ensure-snapshot -> smoke -> promote as initContainers, prune as the container The four step scripts are carried over byte-for-byte; only the plumbing moved: * `{{workflow.parameters.kernel-tag}}` -> ${KERNEL_TAG}, a plain Flux substitution. Shell `$` stays written as `$$`. * the snapshot id was an Argo output parameter captured from /tmp/snapid. A Job has no such mechanism, so it crosses steps as a file on the shared work volume. It MUST be /work: each container has its own /tmp, and on /tmp promote would commit an empty snapshot id into nodes.env. * `when: did-bake == 'true'` on smoke becomes an early exit 0. This also retires a whole bug class: promote previously had to match both smoke.Succeeded and smoke.Skipped, and when it matched Omitted instead, promote and prune were silently skipped on a run Argo reported Succeeded. Sequential initContainers cannot express that -- promote runs if and only if smoke exits 0. The identity is declared here, mounts no API token (nothing calls Kubernetes), and the contract revision goes to r7, which is also what gives the fix a fresh run instead of re-checking the r6 corpse. ttlSecondsAfterFinished: 86400 per tests/job-ttl-contract.sh. That guard is right that a finish timeout means Flux re-creates and re-runs the Job daily, and right that this is only safe for a Job that converges. This one does, by the design already in it: the snapshot for the current tag exists, so ensure-snapshot skips the bake in two API calls, smoke exits, promote finds an empty diff, and prune's steady state is deleting nothing. The current tag's snapshot is always the newest, so prune can never remove the one ensure-snapshot looks for -- which is what stops the daily re-run from becoming a daily VM bake. Flux health-checks batch/v1 Job through kstatus natively, so the Workflow CEL block is gone; the BuildRun still needs its own. tests/node-snapshot-bake.sh is new and pins the shape that broke: native Job, no Argo, identity declared in-repo, the DAG order (smoke before promote is what stops a bad snapshot reaching nodes.env), the /work hand-off, and the four convergence properties. Eight negative controls, all detecting; it goes red against the manifest as deployed today. tests/check-url-credentials.sh and tests/house-helper-images/run-all.sh follow the rename and the moved lines. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76