feat(bootstrap): add refresh RBAC and repair ARM image pins

sean commented

2026-09-12 21:06:34 +00:00

Owner

Combine the two ready seanfarm change sets after Binjovi reported candidate_catalogs_diverged. This head includes PR #669 image-pin repairs, the restricted Binjovi Argo Application refresh Role and RoleBinding, and the AGENTS rule that every pushed feature branch must receive a pull request.

sean added 2 commits

2026-09-12 21:06:34 +00:00

feat(argocd): grant Binjovi bounded application refresh 70d2ca7e74

docs(agents): require a pull request after every branch push

binjovi/ci Binjovi completed the frozen plan

Details

6c84c10984

sean added 1 commit

2026-09-12 21:11:42 +00:00

fix(bootstrap): repin the dead AMD64 image, and guard every pin

binjovi/ci Binjovi completed the frozen plan

Details

e62d2a092f

bootstrap/nodes.env pinned WORKER_AMD64_IMAGE to Hetzner snapshot 422011494.
That snapshot no longer exists, so the AMD64 emergency-rollback pool the file
documents could not create a server: `hcloud server create` answers
"Image not found". The surviving image for the same seanfarm45 bake is
424843910, and that is what the pin now names.

HOW THE PIN DIED, AND WHY DEPTH CANNOT FIX IT

The prune step in node-snapshot-bake keeps the newest 4 images labelled
seanfarm-node-image=1. A SECOND producer sets the rate of that window: the
seanfarm46 and seanfarm47 RELEASE bakes come from the pipelines repository and
never write nodes.env. They pushed the promoted TAG bake out of the window and
the pruner deleted it. Raising KEEP buys time and closes nothing.

So prune now holds a Hetzner delete-protection lock on the image promote just
wrote, and clears the lock from every other node-image snapshot first. Exactly
one image is locked at a time, so the window still drains, and a re-run is
idempotent. `hcloud image delete` already ends with `|| true`, so a locked
image is skipped without stopping the loop.

A SECOND FAULT, FOUND WHILE PROVING THE FIRST

promote rewrites MASTER_IMAGE as well as WORKER_AMD64_IMAGE, and it always
writes the x86 bootc snapshot it just baked. Since "Make ARM the durable
cluster topology" the master is a cax31 pinned to an arm image, so the next
green bake will repoint the ARM control plane at an x86 snapshot and break
create in a new way. Existence alone cannot catch that, so the guard compares
the image architecture against the architecture the pool's own *_TYPE implies.
This commit does not change promote: the guard turns a silent break into a
loud one, and what promote SHOULD write for an ARM master is a topology
decision, not a cleanup.

THE GUARD

scripts/check-node-image-pin.sh proves every *_IMAGE pin exists and matches its
pool's architecture. A rebuild DESTROYS before it CREATES, so it runs as a
preflight phase in run-rebuild.sh, beside the other guards and before destroy —
a dead pin now stops the rebuild while the cluster is still up. It is
non-vacuous: finding no pins is a failure, not a pass.

`make test-node-image-pin` runs the hermetic unit test (8 cases; it stubs the
Hetzner lookup, so it needs neither a token nor the network).

THE OBJECT-STORE HALF

The Hetzner bucket carried NO lifecycle configuration, so vmbackup's leaked
multipart uploads accumulated forever: 44 of them from one failed run on
2026-08-10, holding 13.48 GB that no object listing shows. The declared
document now lives in bootstrap/hetzner-bucket-lifecycle.json and
`make bucket-lifecycle` converges it.

It is a script and not a Flux Job on purpose. The rule lives on the bucket,
which is Hetzner account state, so it survives `make rebuild` and needs no
in-cluster re-assertion; the MinIO ILM rules are the opposite case and are Jobs
for that reason.

tests/hetzner-bucket-lifecycle.sh asserts the document aborts incomplete
uploads and expires NOTHING. That bucket holds the barman archive and the
VictoriaMetrics mirror, and a blanket expiry rule on the MinIO workflows bucket
deleted the macOS SDK once.

Also corrects a stale lineage path in the cnpg-restore-drill trap comment,
which still named postgres-cnpg/ after OFFSITE_SERVER moved to the arm lineage.

tests/check.sh passes, including tests/no-sigpipe-readers.sh, which caught two
`| grep -q` readers in the new guard before they could invert a match.

Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

sean changed title from ~~feat(argocd): grant Binjovi bounded application refresh~~ to feat(bootstrap): add refresh RBAC and repair ARM image pins

2026-09-12 21:11:50 +00:00

binjovi-bot merged commit e62d2a092f into trunk

2026-09-12 21:14:52 +00:00

binjovi-bot deleted branch fix/binjovi-argo-refresh-rbac-20260912

2026-09-12 21:14:52 +00:00

Rows
Columns

feat(bootstrap): add refresh RBAC and repair ARM image pins #687