fix(retirement): repin seven tombstone Jobs off the dead AMD64 image #744

Merged

binjovi-bot merged 1 commit from fix/retire-jobs-arm64 into trunk 2026-09-14 14:07:42 +00:00

Conversation 0 Commits 1 Files changed 7 +14 -14

sean commented 2026-09-14 14:05:55 +00:00

Owner

Copy link

Seven retirement Jobs have never run. Every one fails identically:

exec /bin/sh: exec format error

Job	namespace	failing for
buildkit-fleet-retire	argocd	22h
per-lane-buildkit-retire-v2	binjovi-builds	19h
argo-workflows-retire	argocd	14h
argo-events-retire	flux-system	14h
pipelines-gitmirror-retire	forgejo	14h
binjovi-argo-rbac-retire	workflows	4h
argo-crd-retire	flux-system	4h

All seven pin registry.sean.farm/someara/pipeline-tools:v8 by the digest
6560ba95. A digest names one architecture, not a manifest list, and every
node in the fleet is arm64. The binfmt DaemonSet that would emulate it selects
kubernetes.io/arch=amd64 and has 0 desired, because no amd64 node remains.

CHANGELOG already carries this diagnosis for
binjovi-management-grants-bootstrap, which was repaired. These seven were
missed.

The replacement is verified, not assumed

registry.sean.farm/build/pipeline-tools:v16@sha256:29918dd... is the pin the
already-repaired Jobs use. Six pods carrying it report Succeeded on
seanfarm-worker-arm-* nodes, so it demonstrably runs on this fleet.

Every Job name moves with the image

A Job template is immutable; the apiserver refuses the apply otherwise. The
CHANGELOG states the rule outright: "Jobs are immutable, so the -v2 suffix
has to move when the image is repaired."

Six take -v2; per-lane-buildkit-retire-v2 takes -v3. ServiceAccounts,
Roles and RoleBindings keep their names, so no binding changes and no RBAC
churn.

I checked that each file holds exactly one Job, so no other Job silently
inherits the new image without a matching rename.

bash tests/check.sh exits 0, including tests/argo-workflows-retired.sh,
which independently requires the Job image stay digest-pinned.

https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

**Seven retirement Jobs have never run.** Every one fails identically: ``` exec /bin/sh: exec format error ``` | Job | namespace | failing for | | --- | --- | --- | | `buildkit-fleet-retire` | argocd | 22h | | `per-lane-buildkit-retire-v2` | binjovi-builds | 19h | | `argo-workflows-retire` | argocd | 14h | | `argo-events-retire` | flux-system | 14h | | `pipelines-gitmirror-retire` | forgejo | 14h | | `binjovi-argo-rbac-retire` | workflows | 4h | | `argo-crd-retire` | flux-system | 4h | All seven pin `registry.sean.farm/someara/pipeline-tools:v8` by the digest `6560ba95`. **A digest names one architecture, not a manifest list**, and every node in the fleet is arm64. The `binfmt` DaemonSet that would emulate it selects `kubernetes.io/arch=amd64` and has **0 desired**, because no amd64 node remains. CHANGELOG already carries this diagnosis for `binjovi-management-grants-bootstrap`, which was repaired. These seven were missed. ### The replacement is verified, not assumed `registry.sean.farm/build/pipeline-tools:v16@sha256:29918dd...` is the pin the already-repaired Jobs use. Six pods carrying it report `Succeeded` on `seanfarm-worker-arm-*` nodes, so it demonstrably runs on this fleet. ### Every Job name moves with the image A Job template is immutable; the apiserver refuses the apply otherwise. The CHANGELOG states the rule outright: *"Jobs are immutable, so the `-v2` suffix has to move when the image is repaired."* Six take `-v2`; `per-lane-buildkit-retire-v2` takes `-v3`. **ServiceAccounts, Roles and RoleBindings keep their names**, so no binding changes and no RBAC churn. I checked that each file holds exactly one Job, so no other Job silently inherits the new image without a matching rename. `bash tests/check.sh` exits 0, including `tests/argo-workflows-retired.sh`, which independently requires the Job image stay digest-pinned. https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean added 1 commit 2026-09-14 14:05:55 +00:00

fix(retirement): repin seven tombstone Jobs off the dead AMD64 image ... fbefa31539

Seven retirement Jobs have never run. Every one fails the same way:

  exec /bin/sh: exec format error

  argocd/buildkit-fleet-retire                22h
  binjovi-builds/per-lane-buildkit-retire-v2  19h
  argocd/argo-workflows-retire                14h
  flux-system/argo-events-retire              14h
  forgejo/pipelines-gitmirror-retire          14h
  workflows/binjovi-argo-rbac-retire           4h
  flux-system/argo-crd-retire                  4h

All seven pinned registry.sean.farm/someara/pipeline-tools:v8 by the
digest 6560ba95. A digest names ONE architecture, not a manifest list,
and every node in the fleet is arm64. The binfmt DaemonSet that would
emulate it selects kubernetes.io/arch=amd64 and has 0 desired, because
no amd64 node is left. CHANGELOG already records this diagnosis for
binjovi-management-grants-bootstrap, which was repaired; these were
missed.

The replacement is registry.sean.farm/build/pipeline-tools:v16 at
29918ddf, the pin the repaired Jobs already use. It is verified on
arm64 rather than assumed: six pods carrying it report Succeeded on
seanfarm-worker-arm nodes.

Each Job name moves with the image because a Job template is immutable
and the apiserver refuses the apply otherwise. CHANGELOG states the rule:
"Jobs are immutable, so the -v2 suffix has to move when the image is
repaired." argo-crd-retire, argo-events-retire, argo-workflows-retire,
binjovi-argo-rbac-retire, buildkit-fleet-retire and
pipelines-gitmirror-retire take -v2; per-lane-buildkit-retire-v2 takes
-v3. ServiceAccounts, Roles and RoleBindings keep their names, so no
binding changes.

Each file holds exactly one Job, checked, so no other Job inherits the
new image without a matching rename.

tests/check.sh exits 0.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

binjovi-bot merged commit fbefa31539 into trunk 2026-09-14 14:07:42 +00:00

binjovi-bot deleted branch fix/retire-jobs-arm64 2026-09-14 14:07:43 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!744

No description provided.