feat(registry): let a rollout replace a zot-v2 pod that is not Ready #27

Merged

binjovi-bot merged 2 commits from feat/zot-parallel-pod-management into trunk

2026-09-02 13:55:00 +00:00

sean commented

2026-09-02 13:41:32 +00:00

Owner

Why

Link 5 of the 2026-09-02 outage chain: the GitOps revert could not roll the Pending zot-v2-0 pod. With the default podManagementPolicy: OrderedReady the StatefulSet controller returns from processReplica before the update loop when a pod exists but is not Running and Ready (stateful_set_control.go: if !isRunningAndReady(replicas[i]) && monotonic { return true, nil }). A human had to delete the pod.

What

StatefulSet/zot-v2: podManagementPolicy: Parallel. The update loop then runs and deletes the pod whose revision differs from the update revision. One replica has no other pod to order.
tests/zot-pod-management.sh: pins the policy and the replica count (red before the manifest change, green after).
docs/REGISTRY-AVAILABILITY.md: new section "Pod management policy" with the controller code path and the one-time migration procedure.

Migration (one-time, break glass)

The field is immutable. The first sync attempt of this commit fails on the StatefulSet apply and the automated operation retries (limit 5). While that operation runs, kubectl -n registry delete statefulset zot-v2 --cascade=orphan removes only the StatefulSet object; the pod, the ControllerRevisions and PersistentVolumeClaim/data-zot-v2-0 lose their owner reference and stay. The next attempt creates the StatefulSet, which adopts the pod (same ControllerRevision hash, the hash covers the pod template only) and restores the PVC owner reference. No pod restart. --cascade=orphan is mandatory: whenDeleted: Delete puts an owner reference on the PVC, so a cascading delete would remove the local index.

Proof

bash tests/check.sh green on 5601928.

https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

## Why Link 5 of the 2026-09-02 outage chain: the GitOps revert could not roll the Pending `zot-v2-0` pod. With the default `podManagementPolicy: OrderedReady` the StatefulSet controller returns from `processReplica` before the update loop when a pod exists but is not Running and Ready (`stateful_set_control.go`: `if !isRunningAndReady(replicas[i]) && monotonic { return true, nil }`). A human had to delete the pod. ## What - `StatefulSet/zot-v2`: `podManagementPolicy: Parallel`. The update loop then runs and deletes the pod whose revision differs from the update revision. One replica has no other pod to order. - `tests/zot-pod-management.sh`: pins the policy and the replica count (red before the manifest change, green after). - `docs/REGISTRY-AVAILABILITY.md`: new section "Pod management policy" with the controller code path and the one-time migration procedure. ## Migration (one-time, break glass) The field is immutable. The first sync attempt of this commit fails on the StatefulSet apply and the automated operation retries (limit 5). While that operation runs, `kubectl -n registry delete statefulset zot-v2 --cascade=orphan` removes only the StatefulSet object; the pod, the ControllerRevisions and `PersistentVolumeClaim/data-zot-v2-0` lose their owner reference and stay. The next attempt creates the StatefulSet, which adopts the pod (same ControllerRevision hash, the hash covers the pod template only) and restores the PVC owner reference. No pod restart. `--cascade=orphan` is mandatory: `whenDeleted: Delete` puts an owner reference on the PVC, so a cascading delete would remove the local index. ## Proof `bash tests/check.sh` green on 5601928. https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean added 1 commit

2026-09-02 13:41:32 +00:00

feat(registry): let a rollout replace a zot-v2 pod that is not Ready

binjovi/ci Binjovi completed the frozen plan

Details

560192863b

Set podManagementPolicy: Parallel on StatefulSet/zot-v2. With the default
OrderedReady, the controller waits for the existing pod to be Running and
Ready before it looks at the update revision (stateful_set_control.go,
processReplica). A GitOps revert of a bad image then never rolls a Pending
pod. That was link 5 of the 2026-09-02 outage; a human had to delete the
pod. With Parallel the update loop runs and deletes the pod whose revision
differs from the update revision. One replica has no other pod to order.

The field is immutable. The docs record the one-time migration: delete the
live StatefulSet with --cascade=orphan while the automated sync of this
commit runs, so the next attempt creates it around the running pod and the
retained PVC. tests/zot-pod-management.sh pins the policy and the replica
count.

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean added 1 commit

2026-09-02 13:51:58 +00:00

fix(registry): keep the adopted pod during the policy migration

binjovi/ci Binjovi completed the frozen plan

Details

3e79ca5a81

A refutation pass found that the recreated StatefulSet does not hash to the
revision of the running pod. The live template carries two fields that are
not in Git: the restartedAt annotation of a 2026-08-31 rollout restart and
the STAKATER_ZOT_CONFIG_SECRET env that Reloader injects. Partition 0 would
roll the pod at once. Set updateStrategy.rollingUpdate.partition: 1 for the
migration; the next template change sets it back to 0 and rolls the pod
anyway.

The docs now record the real failure mode of a late delete: Binjovi does not
revert Argo CD targets, Argo CD does not retry a failed revision, and the
recovery is a new commit on main. The guard accepts the transient partition
and prints a reminder.

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5