fix(repartition): the reclaim must delete the pod, not just the claim #2161

Merged

pipeline-bot merged 1 commit from fix/repartition-reclaim-pod into trunk 2026-09-04 14:00:18 +00:00

Conversation 0 Commits 1 Files changed 2 +48

sean commented 2026-09-04 13:59:34 +00:00

Owner

Copy link

Every node in the first fleet run stalled on this and needed clearing by hand.

Deleting the PVC, PV and ZFSVolume is not enough. The controller's pod outlives the claim, stays bound to a PV that no longer exists, and the scheduler reports:

N node(s) didn't match PersistentVolume's node affinity

for ever. CNPG never re-provisions the instance, the cluster holds at 4/5, and the green gate — correctly, because cnpg-not-ready is a real predicate — burns its budget waiting. That cost 40m48s on the first node before the pod was deleted by hand, after which the operator built a fresh instance in ~100 seconds (postgres-6 returned as postgres-12).

scripts/k3s-remove-node.sh Phase 3 force-deleted the bound pod. That step was dropped in the port. This restores it.

Two details that matter

Order is load-bearing. pvc-protection holds a PVC in Terminating while any pod still references it, so the claim delete stays --wait=false and the pod delete follows it. Reversing them deadlocks the claim instead.

Pods are resolved by the claim they actually mount (spec.volumes[].persistentVolumeClaim.claimName), not by name coincidence — CNPG happens to name the pod after the claim, MinIO does not.

Verified by mutation

mutation	result
remove the force-delete	the reclaim never force-deletes the pod bound to a reclaimed claim
move it above the claim delete	the pod delete must FOLLOW the --wait=false claim delete (pvc=390 pod=389)

My first attempt at those mutations silently passed because the test strings had the wrong indentation and never applied. They're now done by line number, so a no-op edit can't masquerade as a passing guard.

tests/check.sh — 147/147.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

**Every node in the first fleet run stalled on this** and needed clearing by hand. Deleting the PVC, PV and ZFSVolume is not enough. The controller's pod **outlives the claim**, stays bound to a PV that no longer exists, and the scheduler reports: ``` N node(s) didn't match PersistentVolume's node affinity ``` for ever. CNPG never re-provisions the instance, the cluster holds at 4/5, and the green gate — correctly, because `cnpg-not-ready` is a real predicate — burns its budget waiting. That cost **40m48s on the first node** before the pod was deleted by hand, after which the operator built a fresh instance in ~100 seconds (`postgres-6` returned as `postgres-12`). `scripts/k3s-remove-node.sh` Phase 3 force-deleted the bound pod. **That step was dropped in the port.** This restores it. ## Two details that matter **Order is load-bearing.** `pvc-protection` holds a PVC in `Terminating` while any pod still references it, so the claim delete stays `--wait=false` and the pod delete *follows* it. Reversing them deadlocks the claim instead. **Pods are resolved by the claim they actually mount** (`spec.volumes[].persistentVolumeClaim.claimName`), not by name coincidence — CNPG happens to name the pod after the claim, MinIO does not. ## Verified by mutation | mutation | result | | --- | --- | | remove the force-delete | `the reclaim never force-deletes the pod bound to a reclaimed claim` | | move it above the claim delete | `the pod delete must FOLLOW the --wait=false claim delete (pvc=390 pod=389)` | My first attempt at those mutations **silently passed** because the test strings had the wrong indentation and never applied. They're now done by line number, so a no-op edit can't masquerade as a passing guard. `tests/check.sh` — 147/147. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit 2026-09-04 13:59:35 +00:00

fix(repartition): the reclaim must delete the pod, not just the claim ... fd40de6a72

Every node in the first fleet run stalled on this and needed clearing by
hand. Deleting the PVC, PV and ZFSVolume is not enough: the controller's
pod OUTLIVES the claim, stays bound to a PV that no longer exists, and the
scheduler reports

  N node(s) didn't match PersistentVolume's node affinity

for ever. CNPG never re-provisions the instance, the cluster holds at 4/5,
and the green gate — correctly, because cnpg-not-ready is a real predicate
— burns its budget waiting. That cost 40m48s on the first node before the
pod was deleted by hand, after which the operator built a fresh instance
immediately (postgres-6 returned as postgres-12 in ~100 seconds).

scripts/k3s-remove-node.sh Phase 3 force-deleted the bound pod. That step
was dropped in the port. This restores it.

Order is load-bearing. pvc-protection holds a PVC in Terminating while any
pod still references it, so the claim delete stays --wait=false and the pod
delete FOLLOWS it. Reversing them would deadlock the claim instead.

Pods are resolved by the claim they actually mount
(spec.volumes[].persistentVolumeClaim.claimName), not by name coincidence —
CNPG happens to name the pod after the claim, MinIO does not.

The guard now pins the force-delete AND its position relative to the claim
delete. Verified by mutation: removing the line fails with "the reclaim
never force-deletes the pod bound to a reclaimed claim", and moving it above
the claim delete fails with "the pod delete must FOLLOW the --wait=false
claim delete (pvc=390 pod=389)".

The first mutation attempt silently passed because the test strings had the
wrong indentation and never applied — the mutations are now done by line
number so a no-op edit cannot look like a passing guard.

tests/check.sh: 147/147.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

pipeline-bot merged commit fd40de6a72 into trunk 2026-09-04 14:00:18 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/pipelines!2161

No description provided.