fix(rustfs): disarm whenDeleted: Delete now the resize is done #449

Merged

binjovi-bot merged 1 commit from fix/rustfs-retain-drives into trunk

2026-09-08 09:01:21 +00:00

sean commented

2026-09-08 08:54:55 +00:00

Owner

whenDeleted was Delete for exactly one release. volumeClaimTemplates are immutable, so growing the drives from 55Gi to 110Gi was a delete-and-recreate, and Delete is what let the recreate provision new claims instead of rebinding the old undersized ones (98d8c77a armed it, 34900385 used it).

That resize has landed and verified — four drive-rustfs-N claims at 110Gi, 4/4 Ready, erasure ratio measured at 2.004x — and it was the only reason to arm it.

Why it must not stay armed

This store is taking over from MinIO and will hold 144 GiB including the zot bucket: every OCI blob, the kernel image spine.

With Delete, anything that removes this StatefulSet destroys all four drives with it, and the paths that remove it are ordinary — a prune after an editing mistake, a bad force recreate, a rename. The cost of Retain is the opposite and much cheaper: an orphaned set of claims that someone deletes on purpose.

Both policies are now Retain. The field is mutable, so this applies in place — confirmed by server-side dry run, no recreate:

statefulset.apps/rustfs serverside-applied (server dry run)

`force: true` stays

With both policies Retain, a forced recreate now preserves the drives and rebinds them, so it costs an outage rather than the data.

The trap this leaves behind, and how it is marked

Re-arming Delete is the obvious wrong move for a future resize. Both the manifest and the guard say what to do instead: zfs-nvme-128k sets allowVolumeExpansion: true and the CSI driver supports it, so expand the four PVCs in place and then bring volumeClaimTemplates up to match. Recreating this StatefulSet once it holds data is a data-loss event, not a resize.

tests/rustfs.sh now requires Retain and fails loudly if Delete is re-armed, rather than only asserting the value it wants:

whenDeleted re-armed to Delete                 detected
scale-down allowed to destroy a drive          detected
resize reverted to 55Gi                        detected
half-done rename (template only)               detected

bash tests/check.sh exits 0.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

`whenDeleted` was `Delete` for **exactly one release**. `volumeClaimTemplates` are immutable, so growing the drives from 55Gi to 110Gi was a delete-and-recreate, and `Delete` is what let the recreate provision new claims instead of rebinding the old undersized ones (`98d8c77a` armed it, `34900385` used it). That resize has landed and verified — four `drive-rustfs-N` claims at 110Gi, 4/4 Ready, erasure ratio measured at 2.004x — and it was the only reason to arm it. ## Why it must not stay armed This store is taking over from MinIO and will hold **144 GiB including the `zot` bucket**: every OCI blob, the kernel image spine. With `Delete`, anything that removes this StatefulSet destroys all four drives with it, and the paths that remove it are ordinary — a prune after an editing mistake, a bad `force` recreate, a rename. The cost of `Retain` is the opposite and much cheaper: an orphaned set of claims that someone deletes on purpose. Both policies are now `Retain`. The field is mutable, so this applies **in place** — confirmed by server-side dry run, no recreate: ``` statefulset.apps/rustfs serverside-applied (server dry run) ``` ## `force: true` stays With both policies `Retain`, a forced recreate now **preserves** the drives and rebinds them, so it costs an outage rather than the data. ## The trap this leaves behind, and how it is marked Re-arming `Delete` is the obvious wrong move for a future resize. Both the manifest and the guard say what to do instead: `zfs-nvme-128k` sets `allowVolumeExpansion: true` and the CSI driver supports it, so expand the four PVCs in place and then bring `volumeClaimTemplates` up to match. **Recreating this StatefulSet once it holds data is a data-loss event, not a resize.** `tests/rustfs.sh` now requires `Retain` *and* fails loudly if `Delete` is re-armed, rather than only asserting the value it wants: ``` whenDeleted re-armed to Delete detected scale-down allowed to destroy a drive detected resize reverted to 55Gi detected half-done rename (template only) detected ``` `bash tests/check.sh` exits 0. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit

2026-09-08 08:54:55 +00:00

fix(rustfs): disarm whenDeleted: Delete now the resize is done

binjovi/ci Binjovi completed the frozen plan

Details

39278e894f

whenDeleted was Delete for exactly one release. volumeClaimTemplates are
immutable, so growing the drives from 55Gi to 110Gi was a delete-and-recreate,
and Delete is what let the recreate provision new claims instead of rebinding
the old undersized ones (98d8c77a armed it, 34900385 used it). That resize has
landed and verified -- four drive-rustfs-N claims at 110Gi, 4/4 Ready -- and it
was the only reason to arm it.

Leaving it armed is a standing hazard, and a growing one. This store is taking
over from MinIO and will hold 144 GiB including the zot bucket: every OCI blob,
the kernel image spine. With Delete, anything that removes this StatefulSet
destroys all four drives with it, and the paths that remove it are ordinary --
a prune after an editing mistake, a bad force recreate, a rename. The cost of
Retain is the opposite and much cheaper: an orphaned set of claims that someone
deletes on purpose.

Both policies are now Retain. The field is mutable, so this applies in place --
confirmed by server-side dry run, no recreate.

force: true stays on the Kustomization. With both policies Retain a forced
recreate now PRESERVES the drives and rebinds them, so it costs an outage rather
than the data.

The manifest and the guard both record what to do instead next time, because
re-arming Delete is the obvious wrong move for a future resize: zfs-nvme-128k
sets allowVolumeExpansion: true and the CSI driver supports it, so expand the
four PVCs in place and then bring volumeClaimTemplates up to match. Recreating
this StatefulSet once it holds data is a data-loss event, not a resize.

tests/rustfs.sh now requires Retain and fails loudly if Delete is re-armed,
rather than only asserting the value it wants. Four negative controls, all
detecting.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 39278e894f into trunk

2026-09-08 09:01:21 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!449

No description provided.

Rows
Columns