sean/seanfarm
Watch
1
0
Fork
You've already forked seanfarm
 0
Code Issues Pull requests Projects Releases 386 Packages Wiki Activity

fix(bootstrap): repin the dead AMD64 image, and guard every pin #669

Closed
sean wants to merge 1 commit from chore/hetzner-cleanup-and-pin-fix into trunk
pull from: chore/hetzner-cleanup-and-pin-fix
merge into: sean:trunk
Conversation 0 Commits 1 Files changed 11 +470 -6
sean commented 2026-09-12 09:55:36 +00:00
Owner
Copy link

Cleanup of the Hetzner account turned up a live fault, so this PR fixes the fault first and adds the durable half of the cleanup.

The fault

bootstrap/nodes.env pinned WORKER_AMD64_IMAGE to snapshot 422011494, and that snapshot no longer exists. The AMD64 emergency-rollback pool the file documents could not create a server. The surviving image for the same seanfarm45 bake is 424843910.

How it died, and why depth cannot fix it. The node-snapshot-bake prune keeps the newest 4 images labelled seanfarm-node-image=1. A second producer sets the rate of that window: the seanfarm46 and seanfarm47 RELEASE bakes come from the pipelines repository and never write nodes.env. They pushed the promoted TAG bake out of the window, and the pruner deleted it. Raising KEEP buys time and closes nothing.

So prune now holds a Hetzner delete-protection lock on the image promote just wrote, after clearing the lock from every other node-image snapshot. One image is locked at a time, so the window still drains, and a re-run is idempotent.

A second fault, found while proving the first

promote rewrites MASTER_IMAGE as well as WORKER_AMD64_IMAGE, and it always writes the x86 bootc snapshot it just baked. Since Make ARM the durable cluster topology the master is a cax31 pinned to an arm image. The next green bake will repoint the ARM control plane at an x86 snapshot.

This PR does not change promote. It makes the guard compare image architecture against the architecture the pool's own *_TYPE implies, so the break becomes loud instead of silent. What promote should write for an ARM master is a topology decision, not a cleanup.

The guard

scripts/check-node-image-pin.sh proves every *_IMAGE pin exists and matches its pool architecture. A rebuild destroys before it creates, so it runs as a preflight phase in run-rebuild.sh, before destroy. It is non-vacuous: finding no pins is a failure.

make test-node-image-pin runs the hermetic unit test — 8 cases, no token, no network.

The object-store half

The bucket carried no lifecycle configuration, so vmbackup's leaked multipart uploads accumulated forever: 44 of them from one failed run on 2026-08-10, holding 13.48 GB that no object listing shows. The declared document is now bootstrap/hetzner-bucket-lifecycle.json, converged by make bucket-lifecycle.

A script, not a Flux Job, on purpose: the rule lives on the bucket, which is Hetzner account state, so it survives make rebuild. The MinIO ILM rules are the opposite case and are Jobs for that reason.

tests/hetzner-bucket-lifecycle.sh asserts the document aborts incomplete uploads and expires nothing — that bucket holds the barman archive and the VictoriaMetrics mirror, and a blanket expiry rule on the MinIO workflows bucket deleted the macOS SDK once.

Verification

  • tests/check.sh — green, including tests/no-sigpipe-readers.sh, which caught two | grep -q readers in the new guard before they could invert a match.
  • ./scripts/tests/check-node-image-pin-test.sh — 8 passed.
  • ./scripts/check-node-image-pin.sh against the live account — 3 pins live, architectures correct.
  • The lifecycle rule is applied and reads back with no Expiration.

Not run: creating a node from the repinned image. That changes the fleet.

https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

Cleanup of the Hetzner account turned up a live fault, so this PR fixes the fault first and adds the durable half of the cleanup. ## The fault `bootstrap/nodes.env` pinned `WORKER_AMD64_IMAGE` to snapshot **422011494**, and that snapshot no longer exists. The AMD64 emergency-rollback pool the file documents could not create a server. The surviving image for the same seanfarm45 bake is **424843910**. **How it died, and why depth cannot fix it.** The `node-snapshot-bake` prune keeps the newest 4 images labelled `seanfarm-node-image=1`. A second producer sets the rate of that window: the seanfarm46 and seanfarm47 RELEASE bakes come from the pipelines repository and never write `nodes.env`. They pushed the promoted TAG bake out of the window, and the pruner deleted it. Raising `KEEP` buys time and closes nothing. So prune now holds a Hetzner **delete-protection lock** on the image promote just wrote, after clearing the lock from every other node-image snapshot. One image is locked at a time, so the window still drains, and a re-run is idempotent. ## A second fault, found while proving the first `promote` rewrites `MASTER_IMAGE` as well as `WORKER_AMD64_IMAGE`, and it always writes the x86 bootc snapshot it just baked. Since *Make ARM the durable cluster topology* the master is a `cax31` pinned to an **arm** image. The next green bake will repoint the ARM control plane at an x86 snapshot. This PR does **not** change `promote`. It makes the guard compare image architecture against the architecture the pool's own `*_TYPE` implies, so the break becomes loud instead of silent. What `promote` should write for an ARM master is a topology decision, not a cleanup. ## The guard `scripts/check-node-image-pin.sh` proves every `*_IMAGE` pin exists and matches its pool architecture. A rebuild destroys before it creates, so it runs as a preflight phase in `run-rebuild.sh`, **before destroy**. It is non-vacuous: finding no pins is a failure. `make test-node-image-pin` runs the hermetic unit test — 8 cases, no token, no network. ## The object-store half The bucket carried **no lifecycle configuration**, so vmbackup's leaked multipart uploads accumulated forever: 44 of them from one failed run on 2026-08-10, holding **13.48 GB** that no object listing shows. The declared document is now `bootstrap/hetzner-bucket-lifecycle.json`, converged by `make bucket-lifecycle`. A script, not a Flux Job, on purpose: the rule lives on the bucket, which is Hetzner account state, so it survives `make rebuild`. The MinIO ILM rules are the opposite case and are Jobs for that reason. `tests/hetzner-bucket-lifecycle.sh` asserts the document aborts incomplete uploads and **expires nothing** — that bucket holds the barman archive and the VictoriaMetrics mirror, and a blanket expiry rule on the MinIO `workflows` bucket deleted the macOS SDK once. ## Verification - `tests/check.sh` — green, including `tests/no-sigpipe-readers.sh`, which caught two `| grep -q` readers in the new guard before they could invert a match. - `./scripts/tests/check-node-image-pin-test.sh` — 8 passed. - `./scripts/check-node-image-pin.sh` against the live account — 3 pins live, architectures correct. - The lifecycle rule is applied and reads back with no `Expiration`. Not run: creating a node from the repinned image. That changes the fleet. https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
fix(bootstrap): repin the dead AMD64 image, and guard every pin
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
cb7ea530c5
 
bootstrap/nodes.env pinned WORKER_AMD64_IMAGE to Hetzner snapshot 422011494.
That snapshot no longer exists, so the AMD64 emergency-rollback pool the file
documents could not create a server: `hcloud server create` answers
"Image not found". The surviving image for the same seanfarm45 bake is
424843910, and that is what the pin now names.

HOW THE PIN DIED, AND WHY DEPTH CANNOT FIX IT

The prune step in node-snapshot-bake keeps the newest 4 images labelled
seanfarm-node-image=1. A SECOND producer sets the rate of that window: the
seanfarm46 and seanfarm47 RELEASE bakes come from the pipelines repository and
never write nodes.env. They pushed the promoted TAG bake out of the window and
the pruner deleted it. Raising KEEP buys time and closes nothing.

So prune now holds a Hetzner delete-protection lock on the image promote just
wrote, and clears the lock from every other node-image snapshot first. Exactly
one image is locked at a time, so the window still drains, and a re-run is
idempotent. `hcloud image delete` already ends with `|| true`, so a locked
image is skipped without stopping the loop.

A SECOND FAULT, FOUND WHILE PROVING THE FIRST

promote rewrites MASTER_IMAGE as well as WORKER_AMD64_IMAGE, and it always
writes the x86 bootc snapshot it just baked. Since "Make ARM the durable
cluster topology" the master is a cax31 pinned to an arm image, so the next
green bake will repoint the ARM control plane at an x86 snapshot and break
create in a new way. Existence alone cannot catch that, so the guard compares
the image architecture against the architecture the pool's own *_TYPE implies.
This commit does not change promote: the guard turns a silent break into a
loud one, and what promote SHOULD write for an ARM master is a topology
decision, not a cleanup.

THE GUARD

scripts/check-node-image-pin.sh proves every *_IMAGE pin exists and matches its
pool's architecture. A rebuild DESTROYS before it CREATES, so it runs as a
preflight phase in run-rebuild.sh, beside the other guards and before destroy —
a dead pin now stops the rebuild while the cluster is still up. It is
non-vacuous: finding no pins is a failure, not a pass.

`make test-node-image-pin` runs the hermetic unit test (8 cases; it stubs the
Hetzner lookup, so it needs neither a token nor the network).

THE OBJECT-STORE HALF

The Hetzner bucket carried NO lifecycle configuration, so vmbackup's leaked
multipart uploads accumulated forever: 44 of them from one failed run on
2026-08-10, holding 13.48 GB that no object listing shows. The declared
document now lives in bootstrap/hetzner-bucket-lifecycle.json and
`make bucket-lifecycle` converges it.

It is a script and not a Flux Job on purpose. The rule lives on the bucket,
which is Hetzner account state, so it survives `make rebuild` and needs no
in-cluster re-assertion; the MinIO ILM rules are the opposite case and are Jobs
for that reason.

tests/hetzner-bucket-lifecycle.sh asserts the document aborts incomplete
uploads and expires NOTHING. That bucket holds the barman archive and the
VictoriaMetrics mirror, and a blanket expiry rule on the MinIO workflows bucket
deleted the macOS SDK once.

Also corrects a stale lineage path in the cnpg-restore-drill trap comment,
which still named postgres-cnpg/ after OFFSITE_SERVER moved to the arm lineage.

tests/check.sh passes, including tests/no-sigpipe-readers.sh, which caught two
`| grep -q` readers in the new guard before they could invert a match.

Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
sean closed this pull request 2026-09-12 21:18:01 +00:00
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Required
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/seanfarm!669
No description provided.