sean/seanfarm
Watch
1
0
Fork
You've already forked seanfarm
 0
Code Issues Pull requests Projects Releases 386 Packages Wiki Activity

fix(arm64): finish the amd64 pin migration, and guard it #746

Merged
binjovi-bot merged 1 commit from fix/arm64-house-toolchain into trunk 2026-09-14 15:19:11 +00:00
Conversation 0 Commits 1 Files changed 9 +117 -13
sean commented 2026-09-14 15:17:30 +00:00
Owner
Copy link

minio-eradication.yaml left this note:

"35 other manifests in this repository still pin the amd64-only v8 and will
fail the same way -- the kubelet-path-probe CronJob already does, every five
minutes. That is the ARM migration's to finish, not this file's."

This finishes it.

The cause, proven against the registry

Not read off an error message:

digest media type platforms
6560ba95 someara/pipeline-tools:v8 oci.image.manifest.v1 amd64 only
c9c3a677 build/hetzner-builder:v5 docker manifest.v2 amd64 only
29918ddf build/pipeline-tools:v16 manifest.list.v2 amd64 + arm64

A digest names one architecture unless it names a manifest list, and the fleet
is arm64 only.

Seven manifests move; each Job name moves with it

A Job template is immutable, so the repin never takes unless the name moves --
the apply conflicts instead. binjovi-release-oauth-retire-v1/-v2 name
which OAuth client they retire, not a generation, so they take -r2 rather
than a bumped counter.

Three files that looked like they needed this do NOT

image-gc, sysctl-tuner and openebs-zfs each render two DaemonSets: an
amd64 one held off this fleet by nodeSelector, and an -arm64 one an overlay
rewrites onto build/node-tools-arm64.

The overlay is a kustomize images: transformer keyed on the base image
name. Repinning the base silently stops the rewrite, and the arm64 DaemonSet
renders with the wrong image -- on system-node-critical, hostPID workloads.
tests/host-route-mtu-contract.sh caught it. Nothing in the rendered YAML looks
wrong.

node-snapshot-bake is excluded too: hetzner-builder v5 is its only tag
and is amd64-only, so ensure-snapshot has nowhere to go and the Job cannot run
until that image is rebuilt for arm64. Repinning only its promote container
would fix nothing and break the apply.

The guard

tests/binjovi.sh already held this rule, but only for files containing
admin/clients in one directory -- which is why everything else drifted.
tests/arm64-image-pins.sh is repository-wide.

The rule it encodes is not "never pin amd64" but "never let an amd64 pin
reach an arm64 node", so the four legitimate cases are exempt by path with the
reason recorded, including the transformer trap above.

Two halves, both mutation-tested: a denylist for the verified single-arch
digests, and an allowlist requiring the house toolchain on >=10 image lines --
so reverting every repin turns it red even while the denylist stays green.

One more guard caught me

An earlier draft renamed the ServiceAccount, Role and RoleBinding along with
each Job while leaving serviceAccountName behind.
tests/service-account-references.sh caught it. Only the Job is immutable.

bash tests/check.sh exits 0.

https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

`minio-eradication.yaml` left this note: > *"35 other manifests in this repository still pin the amd64-only v8 and will > fail the same way -- the kubelet-path-probe CronJob already does, every five > minutes. **That is the ARM migration's to finish, not this file's.**"* This finishes it. ### The cause, proven against the registry Not read off an error message: | digest | media type | platforms | | --- | --- | --- | | `6560ba95` someara/pipeline-tools:v8 | `oci.image.manifest.v1` | **amd64 only** | | `c9c3a677` build/hetzner-builder:v5 | `docker manifest.v2` | **amd64 only** | | `29918ddf` build/pipeline-tools:v16 | `manifest.list.v2` | amd64 + arm64 | A digest names one architecture unless it names a manifest list, and the fleet is arm64 only. ### Seven manifests move; each Job name moves with it A Job template is immutable, so the repin never takes unless the name moves -- the apply conflicts instead. `binjovi-release-oauth-retire-v1`/`-v2` name **which OAuth client they retire**, not a generation, so they take `-r2` rather than a bumped counter. ### Three files that looked like they needed this do NOT `image-gc`, `sysctl-tuner` and `openebs-zfs` each render **two** DaemonSets: an amd64 one held off this fleet by `nodeSelector`, and an `-arm64` one an overlay rewrites onto `build/node-tools-arm64`. The overlay is a kustomize `images:` transformer **keyed on the base image name**. Repinning the base silently stops the rewrite, and the arm64 DaemonSet renders with the wrong image -- on `system-node-critical`, `hostPID` workloads. `tests/host-route-mtu-contract.sh` caught it. Nothing in the rendered YAML looks wrong. `node-snapshot-bake` is excluded too: `hetzner-builder` v5 is its **only** tag and is amd64-only, so `ensure-snapshot` has nowhere to go and the Job cannot run until that image is rebuilt for arm64. Repinning only its `promote` container would fix nothing and break the apply. ### The guard `tests/binjovi.sh` already held this rule, but only for files containing `admin/clients` in one directory -- which is why everything else drifted. `tests/arm64-image-pins.sh` is repository-wide. The rule it encodes is **not** "never pin amd64" but **"never let an amd64 pin reach an arm64 node"**, so the four legitimate cases are exempt by path with the reason recorded, including the transformer trap above. Two halves, both mutation-tested: a denylist for the verified single-arch digests, and an allowlist requiring the house toolchain on >=10 image lines -- so reverting every repin turns it red even while the denylist stays green. ### One more guard caught me An earlier draft renamed the ServiceAccount, Role and RoleBinding along with each Job while leaving `serviceAccountName` behind. `tests/service-account-references.sh` caught it. Only the Job is immutable. `bash tests/check.sh` exits 0. https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA
fix(arm64): finish the amd64 pin migration, and guard it
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
7e16cc7e42
 
minio-eradication.yaml left this note: "35 other manifests in this
repository still pin the amd64-only v8 and will fail the same way -- the
kubelet-path-probe CronJob already does, every five minutes. That is the
ARM migration's to finish, not this file's." This finishes it.

The cause is proven against the registry, not read off an error message:

  6560ba95  someara/pipeline-tools:v8   oci.image.manifest.v1  amd64 only
  c9c3a677  build/hetzner-builder:v5    docker manifest.v2     amd64 only
  29918ddf  build/pipeline-tools:v16    manifest.list.v2       amd64+arm64

A digest names one architecture unless it names a manifest list, and the
fleet is arm64 only, so a pod on the first two dies with `exec /bin/sh:
exec format error` before its script runs.

Seven manifests move to v16. Six carry a Job, and a Job template is
immutable, so each Job name moves with its image or the apply conflicts
and the repin never takes. binjovi-release-oauth-retire-v1 and -v2 name
WHICH OAuth client they retire, not a generation, so they take -r2 rather
than a bumped counter.

Three files that looked like they needed this do NOT, and changing them
would have caused an outage. image-gc, sysctl-tuner and openebs-zfs each
render two DaemonSets: an amd64 one held off this fleet by nodeSelector,
and an -arm64 one an overlay rewrites onto build/node-tools-arm64. The
overlay is a kustomize `images:` transformer keyed on the base image
NAME, so repinning the base silently stops the rewrite and the arm64
DaemonSet renders with the wrong image on system-node-critical hostPID
workloads. tests/host-route-mtu-contract.sh caught that. Nothing in the
rendered YAML looks wrong.

node-snapshot-bake is also excluded. hetzner-builder v5 is its only tag
and is amd64-only, so ensure-snapshot has nowhere to go and the Job
cannot run until that image is rebuilt for arm64. Repinning only its
promote container would fix nothing and break the apply.

tests/arm64-image-pins.sh is the new repository-wide guard.
tests/binjovi.sh already held this rule but only for files containing
admin/clients in one directory, which is why everything else drifted.
The rule it encodes is not "never pin amd64", it is "never let an amd64
pin reach an arm64 node", so the four legitimate cases are exempt by path
with the reason recorded. It has two halves: a denylist for the verified
single-arch digests, and an allowlist requiring the house toolchain on at
least ten image lines, so reverting every repin turns it red even though
the denylist would be green. Both halves are mutation-tested.

An earlier draft of this commit renamed the ServiceAccount, Role and
RoleBinding along with each Job while leaving serviceAccountName behind.
tests/service-account-references.sh caught it. Only the Job is immutable.

tests/check.sh exits 0.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA
binjovi-bot deleted branch fix/arm64-house-toolchain 2026-09-14 15:19:11 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/seanfarm!746
No description provided.