feat(registry): warm every StatefulSet image on the zot node before a rollout #23

Merged

binjovi-bot merged 1 commit from feat/zot-image-prepull into trunk 2026-09-02 13:05:47 +00:00

Conversation 0 Commits 1 Files changed 6 +662 -11

sean commented 2026-09-02 13:05:12 +00:00

Owner

Copy link

Add Job/zot-image-warm, an Argo CD Sync hook in wave -1. It pulls every image of the zot-v2 pod template onto the node that runs zot-v2-0 before wave 0 updates the StatefulSet. A bad or unpullable image fails the sync, and the old pod keeps serving.

Why. On 2026-09-02 an image change rolled zot-v2-0 to an image the node could not pull. Nodes pull codeberg.org images through zot itself; the direct fallback failed with 401 on a stale node PAT. The revert could not roll the Pending pod (OrderedReady). The registry was down for 40 minutes.

How.

• The hook pod follows zot through the label registry.sean.farm/image-warm: zot-v2 and a required podAffinity on kubernetes.io/hostname. When no pod carries the label (cold rebuild), the scheduler's self-affinity rule lets the pod run anywhere.
• Namespace/registry moves to wave -1 so the hook has a namespace on a cold rebuild.
• The zot container passes only --version to the image entrypoint: the upstream image has /usr/local/bin/zot-linux-amd64, the seanfarm fork /usr/bin/zot.
• backoffLimit: 0, restartPolicy: Never, activeDeadlineSeconds: 600, no SA token, uid/gid 902, restricted security context, tiny resources.

Guards. New tests/zot-image-warm.sh (image sets equal, digest pins, IfNotPresent, wave placement, pin label on exactly the two templates, pod security, imagePullSecrets parity). tests/zot-binjovi-publisher.sh requires the identity labels as a subset of the pod labels. tests/zot-oidc-ui.sh counts hooks inside the registrar block only.

Rollout note. This sync adds the pin label, so the first hook run is not pinned (the label reaches the live pod in wave 0). Both images are already cached on the zot node, so the one-time roll needs no pull. Pinning applies from the next sync on.

Docs. docs/REGISTRY-AVAILABILITY.md gains "Image warm-up before a rollout": the failure chain, wave and content rules, the one-sync label gap, the cold-rebuild limit (stale node PATs), and how to read a failed hook.

https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

Add `Job/zot-image-warm`, an Argo CD `Sync` hook in wave -1. It pulls every image of the `zot-v2` pod template onto the node that runs `zot-v2-0` before wave 0 updates the StatefulSet. A bad or unpullable image fails the sync, and the old pod keeps serving. **Why.** On 2026-09-02 an image change rolled `zot-v2-0` to an image the node could not pull. Nodes pull `codeberg.org` images through zot itself; the direct fallback failed with 401 on a stale node PAT. The revert could not roll the Pending pod (`OrderedReady`). The registry was down for 40 minutes. **How.** - The hook pod follows zot through the label `registry.sean.farm/image-warm: zot-v2` and a required `podAffinity` on `kubernetes.io/hostname`. When no pod carries the label (cold rebuild), the scheduler's self-affinity rule lets the pod run anywhere. - `Namespace/registry` moves to wave -1 so the hook has a namespace on a cold rebuild. - The zot container passes only `--version` to the image entrypoint: the upstream image has `/usr/local/bin/zot-linux-amd64`, the seanfarm fork `/usr/bin/zot`. - `backoffLimit: 0`, `restartPolicy: Never`, `activeDeadlineSeconds: 600`, no SA token, uid/gid 902, restricted security context, tiny resources. **Guards.** New `tests/zot-image-warm.sh` (image sets equal, digest pins, `IfNotPresent`, wave placement, pin label on exactly the two templates, pod security, `imagePullSecrets` parity). `tests/zot-binjovi-publisher.sh` requires the identity labels as a subset of the pod labels. `tests/zot-oidc-ui.sh` counts hooks inside the registrar block only. **Rollout note.** This sync adds the pin label, so the first hook run is not pinned (the label reaches the live pod in wave 0). Both images are already cached on the zot node, so the one-time roll needs no pull. Pinning applies from the next sync on. **Docs.** `docs/REGISTRY-AVAILABILITY.md` gains "Image warm-up before a rollout": the failure chain, wave and content rules, the one-sync label gap, the cold-rebuild limit (stale node PATs), and how to read a failed hook. https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean added 1 commit 2026-09-02 13:05:12 +00:00

feat(registry): warm every StatefulSet image on the zot node before a rollout ... 13857fd88d

Add Job/zot-image-warm, an Argo CD Sync hook in wave -1. It pulls every
image of the zot-v2 pod template onto the node that runs zot-v2-0 before
wave 0 updates the StatefulSet. A bad or unpullable image fails the sync,
and the old pod keeps serving.

Why: on 2026-09-02 an image change rolled zot-v2-0 to an image the node
could not pull. The node pulls codeberg.org images through zot itself,
and the direct fallback failed with 401 on a stale node PAT. The revert
could not roll the Pending pod (OrderedReady). The registry was down for
40 minutes.

The hook pod follows zot through the label registry.sean.farm/image-warm
and a required podAffinity on kubernetes.io/hostname. Namespace/registry
moves to wave -1 so the hook has a namespace on a cold rebuild. The zot
container passes only --version to the image entrypoint, because the
upstream image and the seanfarm fork keep the binary at different paths.

tests/zot-image-warm.sh guards the contract: image sets equal, digest
pins, IfNotPresent, wave placement, pin label on exactly two templates,
pod security, tiny resources, and no imagePullSecrets drift.
tests/zot-binjovi-publisher.sh now requires the identity labels as a
subset of the pod labels. tests/zot-oidc-ui.sh counts hooks inside the
registrar block only. docs/REGISTRY-AVAILABILITY.md records the chain,
the rules, the one-sync label gap, and the cold-rebuild limit.

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

binjovi-bot merged commit 13857fd88d into trunk 2026-09-02 13:05:47 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/zot!23

No description provided.