fix(registry): give storage readiness a measured timeout #2121

Merged

pipeline-bot merged 1 commit from fix/zot-readiness-margin into trunk

2026-08-31 15:41:44 +00:00

sean commented

2026-08-31 14:10:38 +00:00

Owner

Follow-up to today's zot-v2 storage migration. Zot recovered on its own and is healthy now — this PR fixes the two durable defects the outage exposed. No imperative cluster changes were made.

What happened (verified, not inferred)

The migration was hard-down 13:43:10–13:50:31 UTC and flapping until ~14:00. Two causes:

The availability gate raced itself. PruneLast=true and the zot → zot-v2 rename merged in one commit. ArgoCD history shows the parent (pipelines-argocd) and child (registry) apps both started syncing 554f5cef at 13:42:52Z — the same second. The child cut its sync operation from its pre-PruneLast spec, pruned StatefulSet/zot at 13:43:08, and zot-v2 (empty PVC → full S3 index rebuild, 7m21s) could not serve until 13:50:31. The doc's stated safety property — "the old pod continues to serve traffic" — never applied.
The 5s readiness timeout is inside observed S3 latency. Eight settled-pod samples of /v2/_catalog?n=1 measured 0.94–5.35s. With failureThreshold: 3, three unlucky reads pull the endpoint of a perfectly alive registry. This flapped during the rebuild and will recur under load.

The fix

timeoutSeconds: 5 → 10 on the catalog readiness probe. The catalog path stays — /readyz proves only the process, and the migration gate needs the storage-backed signal. The 5s value was the same-day reviewed number; this supersedes it with measured data, and the test contract's assertion carries the measurements.
docs/REGISTRY-AVAILABILITY.md gains a Gate ordering rule section with the timeline evidence and the durable lesson: merge an Application-spec gate one step before the bundle change it gates, confirm it on the live Application, then merge the change. A same-commit gate races the child's automated sync in a two-level app-of-apps. The ordering spans two merges, so it is documented rather than statically asserted.

Deploy note

The probe change edits the pod template, so deploying this restarts the singleton once — one bounded index-rescan window (~5 min live, within the 900s startup budget). Pick the moment; do not deploy mid-kernel-roll. If #2119 (zot UI OIDC) merges around the same time, batching them costs one restart instead of two.

Testing

tests/zot-binjovi-publisher.sh contract updated first (red against the unmodified manifest, green here); full tests/check.sh 142/142, 0 skipped.

https://claude.ai/code/session_019JVStQiFwbSLGHArU2sA4X

Follow-up to today's zot-v2 storage migration. Zot recovered on its own and is healthy now — this PR fixes the two durable defects the outage exposed. No imperative cluster changes were made. ### What happened (verified, not inferred) The migration was hard-down 13:43:10–13:50:31 UTC and flapping until ~14:00. Two causes: 1. **The availability gate raced itself.** `PruneLast=true` and the `zot` → `zot-v2` rename merged in one commit. ArgoCD history shows the parent (`pipelines-argocd`) and child (`registry`) apps both started syncing `554f5cef` at **13:42:52Z — the same second**. The child cut its sync operation from its pre-`PruneLast` spec, pruned `StatefulSet/zot` at 13:43:08, and `zot-v2` (empty PVC → full S3 index rebuild, 7m21s) could not serve until 13:50:31. The doc's stated safety property — "the old pod continues to serve traffic" — never applied. 2. **The 5s readiness timeout is inside observed S3 latency.** Eight settled-pod samples of `/v2/_catalog?n=1` measured 0.94–5.35s. With `failureThreshold: 3`, three unlucky reads pull the endpoint of a perfectly alive registry. This flapped during the rebuild and will recur under load. ### The fix - `timeoutSeconds: 5` → `10` on the catalog readiness probe. The catalog path stays — `/readyz` proves only the process, and the migration gate needs the storage-backed signal. The 5s value was the same-day reviewed number; this supersedes it with measured data, and the test contract's assertion carries the measurements. - `docs/REGISTRY-AVAILABILITY.md` gains a **Gate ordering rule** section with the timeline evidence and the durable lesson: merge an Application-spec gate one step *before* the bundle change it gates, confirm it on the live Application, then merge the change. A same-commit gate races the child's automated sync in a two-level app-of-apps. The ordering spans two merges, so it is documented rather than statically asserted. ### Deploy note The probe change edits the pod template, so deploying this restarts the singleton once — one bounded index-rescan window (~5 min live, within the 900s startup budget). Pick the moment; do not deploy mid-kernel-roll. If #2119 (zot UI OIDC) merges around the same time, batching them costs one restart instead of two. ### Testing - `tests/zot-binjovi-publisher.sh` contract updated first (red against the unmodified manifest, green here); full `tests/check.sh` 142/142, 0 skipped. https://claude.ai/code/session_019JVStQiFwbSLGHArU2sA4X

sean added 1 commit

2026-08-31 14:10:38 +00:00

fix(registry): give storage readiness a measured timeout

pipeline/ci CI green @ aeb36f799bbc

Details

aeb36f799b

The catalog readiness probe reads S3. Settled-pod samples on 2026-08-31
measured 0.94s to 5.35s against the 5s timeout, so three slow reads in a
row (failureThreshold 3) pulled the endpoint of a live registry during the
zot-v2 index rebuild. Raise the timeout to 10s. The observed p-high clears
it with margin, and a sustained stall above 10s still un-readies the pod,
which at that point is the correct signal. The catalog path stays: /readyz
proves only the process, and the migration gate needs the storage path.

Also record the gate post-mortem in docs/REGISTRY-AVAILABILITY.md. The
PruneLast option and the StatefulSet rename merged in one commit, the
parent and child applications both started syncing 554f5cef at 13:42:52Z,
and the child cut its operation from the pre-PruneLast spec. It pruned
StatefulSet/zot at 13:43:08, seven minutes before zot-v2 could serve. The
rule that follows: merge an Application-spec gate one step before the
bundle change it gates, and confirm it on the live Application first. That
ordering spans two merges, so it is documented, not statically asserted.

The probe change edits the pod template, so deploying this restarts the
singleton once and costs one bounded index-rescan window.

Claude-Session: https://claude.ai/code/session_019JVStQiFwbSLGHArU2sA4X

sean referenced this pull request

2026-08-31 14:12:26 +00:00

feat(registry): accept the Ory session in the zot UI #2119

sean referenced this pull request from a commit

2026-08-31 14:41:35 +00:00

feat(registry): repoint the registry Application at sean/zot

sean referenced this pull request

2026-08-31 14:46:08 +00:00

feat(registry): repoint the registry Application at sean/zot #2122

~~sean referenced this pull request 2026-08-31 14:46:12 +00:00~~

chore: retire the registry bundle from Pipelines #2123

sean changed target branch from main to trunk