docs(registries): the codeberg fallback does not cover zot being down #252

Merged

binjovi-bot merged 1 commit from fix/registry-mirror-fallback into trunk

2026-09-04 15:43:34 +00:00

sean commented

2026-09-04 15:31:37 +00:00

Owner

The comment on the codeberg.org mirror claimed the second endpoint covers zot down or empty. Only empty was ever true, and the difference cost ~20 minutes of total CI outage on 2026-09-04.

containerd walks the endpoint list on a 404 and treats any other status as fatal without trying the next endpoint. A zot with no ready backend is a 503 from the gateway, not a 404.

It bites because zot's own image is on codeberg.org and is therefore mirrored through zot. While any node still holds that image spegel serves the replacement pod and nothing shows. Lose it on every node at once (wiping containerd content fleet-wide: a repartition, a cold rebuild) and the cluster cannot start the one service that could serve it.

Records the break-glass k3s ctr pull, including the two things that were actually needed live: tag as well as pull, and do not delete the zot pod.

No rendered config changes — mirror list, endpoints and configs are byte-identical. But the k3s-registries DaemonSet hashes whole file bytes, so merging this still takes the serialized fleet-wide k3s restart while rendering identical containerd config. Worth timing deliberately.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

The comment on the codeberg.org mirror claimed the second endpoint covers `zot down or empty`. Only **empty** was ever true, and the difference cost ~20 minutes of total CI outage on 2026-09-04. containerd walks the endpoint list on a **404** and treats any other status as fatal without trying the next endpoint. A zot with no ready backend is a **503** from the gateway, not a 404. It bites because zot's own image is on codeberg.org and is therefore mirrored through zot. While any node still holds that image spegel serves the replacement pod and nothing shows. Lose it on every node at once (wiping containerd content fleet-wide: a repartition, a cold rebuild) and the cluster cannot start the one service that could serve it. Records the break-glass `k3s ctr` pull, including the two things that were actually needed live: tag as well as pull, and do not delete the zot pod. **No rendered config changes** — mirror list, endpoints and configs are byte-identical. But the k3s-registries DaemonSet hashes whole file bytes, so merging this still takes the serialized fleet-wide k3s restart while rendering identical containerd config. Worth timing deliberately. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit

2026-09-04 15:31:37 +00:00

docs(registries): the codeberg fallback does not cover zot being down

binjovi/ci Binjovi completed the frozen plan

Details

58c7d727cc

The comment on the codeberg.org mirror said the second endpoint covers "zot
down or empty". Only "empty" was ever true, and the difference cost about
twenty minutes of total CI outage on 2026-09-04.

containerd walks the endpoint list on a 404 — "that blob is not here, try the
next one" — and treats any other status as fatal without trying the next
endpoint. A zot with no ready backend is a 503 from the gateway, not a 404, so
the pull fails outright:

  unexpected status from HEAD request to
  https://registry.sean.farm/v2/someara/zot/manifests/sha256:...?ns=codeberg.org:
  503 Service Unavailable

It bites because zot's own image lives on codeberg.org and is therefore
mirrored through zot. While any node still holds that image spegel serves the
replacement pod and nothing is visible, which is why this went unnoticed. Lose
it on every node at once — what wiping containerd's content store fleet-wide
does, so a repartition or a cold rebuild — and the cluster cannot start the one
service that could serve it. registry.sean.farm 503s and every workload image
and CI build stops behind it.

Recorded with the break-glass, which is a direct `k3s ctr` pull: ctr does not
read certs.d, so it bypasses the mirror and takes about three seconds. Both the
tag step and the do-not-delete-the-pod warning are there because both were
needed in the live recovery — the pod asks for tag+digest so IfNotPresent needs
that exact reference, and deleting the pod re-provisions its generic-ephemeral
scratch claim on whichever node the replacement lands on, which need not be the
node just warmed.

Reordering the two endpoints would fix it and is the wrong trade: codeberg rate
limits per client IP, so going direct first puts every node on that limit on
every roll, which is the entire reason zot is in front.

NO RENDERED CONFIG CHANGES. The mirror list, the endpoints and the configs block
are byte-identical; this is a comment. The k3s-registries DaemonSet hashes whole
file bytes, so merging this still writes the file on every node and takes the
serialized fleet-wide k3s restart, rendering containerd config identical to what
is already there. Worth timing deliberately rather than landing mid-operation.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

binjovi-bot merged commit 58c7d727cc into trunk

2026-09-04 15:43:34 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!252

No description provided.

Rows
Columns