sean/zot
Watch
1
0
Fork
You've already forked zot
 0
Code Issues Pull requests Projects Releases 43 Packages Wiki Activity

feat(registry): run the seanfarm fork of zot with a bounded S3 retry (second landing) #25

Merged
binjovi-bot merged 1 commit from feat/zot-seanfarm-fork-2 into trunk 2026-09-02 13:29:08 +00:00
Conversation 0 Commits 1 Files changed 3 +87 -12
sean commented 2026-09-02 13:28:04 +00:00
Owner
Copy link

Re-lands d210701 (v0.1.8, reverted in v0.1.9) on top of the image warm-up hook (v0.1.11).

Why the first landing failed

The StatefulSet rolled to codeberg.org/someara/zot:v2.1.15-seanfarm.1 on a node that could not pull it: the containerd mirror for codeberg.org is zot itself (down during the swap) and the node's direct codeberg.org credential is stale (401). OrderedReady then held the revert behind the Pending pod.

What is different now

  • Job/zot-image-warm (sync wave -1) pulls every zot-v2 image onto the node of the running pod while the old pod still serves. This PR changes the hook's zot container to the fork image in the same commit as the StatefulSet, as the hook rules require (tests/zot-image-warm.sh enforces equal image sets).
  • Precondition from the hook docs checked live: zot-v2-0 carries registry.sean.farm/image-warm=zot-v2 on seanfarm-worker-a40389, so the hook pod lands on that node.
  • The fork image resolves to digest 8963f24d… both through registry.sean.farm/someara/zot (mirror, sync-on-demand) and anonymously on codeberg.org — the hook pull does not need the node credential while zot serves.
  • Argo CD now terminates a sync after 7200 s (seanfarm PR 192) and the Binjovi observer v7 preempts only proven-stale operations, so a hung sync can no longer block a revert.

Change

  • manifests/registry/manifest.yaml: StatefulSet zot image → fork digest; hook zot container → same reference; comments.
  • tests/zot-binjovi-publisher.sh: pins the fork digest; rejects ghcr.io/project-zot/zot and a self-pull through registry.sean.farm.
  • docs/REGISTRY-AVAILABILITY.md: "S3 transient errors" section (chain, predicate, log line to count, rebuild path incl. the hook image rule).

bash tests/check.sh green.

Verify after deploy

Job/zot-image-warm Succeeded on a40389 before the StatefulSet rolled; zot-v2-0 Running with the fork digest; registry-write-read gate passes; /v2/ 200. Then 3–7 days of Loki: transient s3 error, retrying present, unexpected error, removing .uploads/ files == 0.

https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

Re-lands d210701 (v0.1.8, reverted in v0.1.9) on top of the image warm-up hook (v0.1.11). ## Why the first landing failed The StatefulSet rolled to `codeberg.org/someara/zot:v2.1.15-seanfarm.1` on a node that could not pull it: the containerd mirror for codeberg.org is zot itself (down during the swap) and the node's direct codeberg.org credential is stale (401). OrderedReady then held the revert behind the Pending pod. ## What is different now - `Job/zot-image-warm` (sync wave -1) pulls every `zot-v2` image onto the node of the running pod while the old pod still serves. This PR changes the hook's `zot` container to the fork image in the same commit as the StatefulSet, as the hook rules require (`tests/zot-image-warm.sh` enforces equal image sets). - Precondition from the hook docs checked live: `zot-v2-0` carries `registry.sean.farm/image-warm=zot-v2` on `seanfarm-worker-a40389`, so the hook pod lands on that node. - The fork image resolves to digest `8963f24d…` both through `registry.sean.farm/someara/zot` (mirror, sync-on-demand) and anonymously on `codeberg.org` — the hook pull does not need the node credential while zot serves. - Argo CD now terminates a sync after 7200 s (seanfarm PR 192) and the Binjovi observer v7 preempts only proven-stale operations, so a hung sync can no longer block a revert. ## Change - `manifests/registry/manifest.yaml`: StatefulSet `zot` image → fork digest; hook `zot` container → same reference; comments. - `tests/zot-binjovi-publisher.sh`: pins the fork digest; rejects `ghcr.io/project-zot/zot` and a self-pull through `registry.sean.farm`. - `docs/REGISTRY-AVAILABILITY.md`: "S3 transient errors" section (chain, predicate, log line to count, rebuild path incl. the hook image rule). `bash tests/check.sh` green. ## Verify after deploy `Job/zot-image-warm` Succeeded on a40389 before the StatefulSet rolled; `zot-v2-0` Running with the fork digest; `registry-write-read` gate passes; `/v2/` 200. Then 3–7 days of Loki: `transient s3 error, retrying` present, `unexpected error, removing .uploads/ files` == 0. https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5
feat(registry): run the seanfarm fork of zot with a bounded S3 retry
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
4ca0947f9c
 
In-cluster pushes fail a few times a day with `blob upload unknown`. MinIO
answers a request with HTTP 499 / ClientDisconnected while the request
context is still alive; aws-sdk-go v1 makes zero retries for that class, and
zot v2.1.15 then removes the whole upload session.

The registry now runs codeberg.org/someara/zot:v2.1.15-seanfarm.1, pinned by
digest. It is upstream v2.1.15 plus a bounded retry (3 attempts, about 0.5 s
worst case) around the idempotent s3 driver calls, retried only for code
ClientDisconnected or status 499. Source: codeberg someara/zot-src, branch
seanfarm, tag v2.1.15-seanfarm.1 (commit 97d35c01). Built in-cluster by the
seanfarm Shipwright Build `zot`; its Dockerfile runs the retry tests in a
build stage.

Second landing. The first (v0.1.8) rolled the StatefulSet to an image the
node could not pull, because the mirror is zot itself and the node's
codeberg credential was stale; v0.1.9 reverted it. Job/zot-image-warm
(v0.1.11) now pulls every zot-v2 image onto the node of the running pod in
sync wave -1, while the old pod still serves. This commit changes the hook
zot container to the fork image in the same commit as the StatefulSet, as
the hook rules require, and the live pod already carries the pin label
(checked: zot-v2-0 on seanfarm-worker-a40389). The fork image resolves to
the same digest through registry.sean.farm and anonymously on codeberg.org.

The base stays v2.1.15 because of project-zot/zot#4336. The image is pulled
from codeberg, never from registry.sean.farm: the registry cannot pull
itself. tests/zot-binjovi-publisher.sh pins the digest and rejects the
upstream image and a self-pull; tests/zot-image-warm.sh rejects a hook image
set that differs from the StatefulSet. docs/REGISTRY-AVAILABILITY.md gets a
section on the chain, the predicate, the log line to count, and the rebuild
path.

(cherry picked from commit d210701be3)

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/zot!25
No description provided.