fix(minio): the eradication Job cannot start on an ARM node #659
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/minio-tombstone-arm"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The eradication tombstone from #651 ran and died on all four attempts with
and exit 255, before a single line of its script executed.
Why
registry.sean.farm/someara/pipeline-tools:v8is amd64-only, and every node in this cluster is now arm64 — the last amd64 worker was replaced during today's migration, and the control plane with it./bin/shcould not start, so the identity fence never ran and nothing was touched.Verified no-op:
sean/miniostill returns HTTP 200 atid=8,size=139409;sean/minio-consoleatid=11,size=696580; neither archived. BothForgejoRepositoryCRs are stilldeletionPolicy: Retain— the flip never happened. Failing closed like that is the correct outcome; the Job just could not do its work.The replacement
registry.sean.farm/build/pipeline-tools:v16carriescurl,jqandbash, is built withARG TARGETARCH, and is proven on this fleet — thehydra-n8n-client-registrarJob ran it to completion onseanfarm-worker-arm-a40389at13:22:39Ztoday.The three guards that pin pipeline-tools images all pass with it:
tests/kernel-deploy-agent-contract.sh,tests/k3s-registries-contract.sh,tests/house-helper-images/run-all.sh.This is not confined to this file
35 occurrences of the amd64-only v8 digest remain on trunk, across about twenty manifests — most of
kubernetes/flux/infrastructure/binjovi/plus thepipelines-retirementtombstones.kubelet-path-probeis already failing every five minutes withexec /usr/bin/sh: exec format error— three consecutiveErrorpods in the last fifteen minutes.argo-workflows-retireand the other retirement Jobs would fail identically if they ran.Sweeping those belongs to the ARM migration, not to this change. It is recorded in the file and here so it is not rediscovered a third time.
A note on the failure signature
This is the third distinct thing today whose only symptom was exit 255: the GitOps validator,
npm ci, and now this Job. It is BuildKit's and the kubelet's way of saying the process never started, and on a freshly-ARM fleet it almost always means an amd64-only image. It is not reachable from any guard, which all exit 1 or 2.kustomize.toolkit.fluxcd.io/forceis what lets this correction reach a Job whose pod template is immutable — the propertytests/retirement-jobs-are-replaceable.shexists to protect, and which #651 extended to this directory.https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8
The tombstone ran and died on all four attempts with exec /bin/sh: exec format error and exit 255, before a single line of its script executed. registry.sean.farm/someara/pipeline-tools:v8 is amd64-only, and every node in this cluster is now arm64 -- the last amd64 worker was replaced during today's migration and the control plane with it. /bin/sh could not start at all, so the identity fence never ran and nothing was touched. The Job failing closed like that is the correct outcome; it just could not do its work. registry.sean.farm/build/pipeline-tools:v16 replaces it. It carries curl, jq and bash, is built with ARG TARGETARCH, and is proven on this fleet: the hydra-n8n-client-registrar Job ran it to completion on seanfarm-worker-arm-a40389 at 13:22:39Z today. tests/kernel-deploy-agent-contract.sh, tests/k3s-registries-contract.sh and tests/house-helper-images/run-all.sh -- the three guards that pin pipeline-tools images -- all pass with it. THIS IS NOT CONFINED TO THIS FILE. 35 occurrences of the amd64-only v8 digest remain on trunk across about twenty manifests, most of them under kubernetes/flux/infrastructure/binjovi/ plus the pipelines-retirement tombstones. The kubelet-path-probe CronJob is already failing every five minutes with `exec /usr/bin/sh: exec format error`, and argo-workflows-retire and the other retirement Jobs would fail the same way if they ran. Sweeping those belongs to the ARM migration, not to this change, and is recorded here so it is not rediscovered a third time. The `kustomize.toolkit.fluxcd.io/force` annotation is what lets this correction reach a Job whose pod template is immutable -- the property tests/retirement-jobs-are-replaceable.sh exists to protect, and which it now protects in this directory too. Claude-Session: https://claude.ai/code/session_01TdW5FSvRJW8CtGnpd29Xp8