feat(rustfs): grow the erasure set to 4 x 110Gi, deterministically #445

Merged

binjovi-bot merged 1 commit from feat/rustfs-resize into trunk 2026-09-08 08:28:27 +00:00

Conversation 0 Commits 1 Files changed 2 +84 -17

sean commented 2026-09-08 08:22:28 +00:00

Owner

Copy link

RustFS was provisioned at 4 x 55Gi -- about 110 GiB usable at 2+2 -- to match MinIO's per-drive size so the two were comparable. It has to hold the 144 GiB now on MinIO, and 136 GiB of that is the zot bucket alone, so at 55Gi the migration would have failed at its last and largest step. 4 x 110Gi is about 220 GiB usable.

This is Phase 0-B of the MinIO to RustFS migration. Phase 0-A (98d8c77a) is live: persistentVolumeClaimRetentionPolicy.whenDeleted: Delete plus force: true, with ownerReferences confirmed present on all four claims.

Why the claim template is renamed

data becomes drive, and the rename is what makes the resize deterministic rather than a coin flip.

volumeClaimTemplates are immutable. Confirmed against the live API server:

The StatefulSet "rustfs" is invalid: spec: Forbidden: updates to statefulset spec
for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy',
'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden

So this is a delete-and-recreate, performed by force: true. Phase 0-A makes the delete garbage-collect the drives. That is necessary but not sufficient.

A claim is named <template>-<statefulset>-<ordinal>. Resizing alone keeps the name data-rustfs-N, and then:

1. Flux deletes the StatefulSet and re-applies it in the same reconcile, background propagation
2. the old claims stay pinned by kubernetes.io/pvc-protection until their pods finish terminating — tens of seconds
3. if the recreated pods reach scheduling first, they bind the surviving 55Gi claims

Nothing errors. The store just stays small — the exact failure this line of work exists to prevent.

Renaming removes the race instead of hoping to win it. The new pods ask for drive-rustfs-N, which cannot already exist, so they always get fresh claims at 110Gi. The old claims are owned by the deleted StatefulSet's UID and are collected on their own schedule; the ordering cannot change the outcome. Their PVs are reclaimPolicy: Delete, so the ZFS datasets go too.

This is safe exactly once

RustFS holds 384K. The manifest records what to do instead next time: zfs-nvme-128k sets allowVolumeExpansion: true and the CSI driver supports it, so a later resize expands the four PVCs in place and then brings the template up to match. Recreating this StatefulSet once it holds data is a data-loss event, not a resize.

Pre-flight

tank free on all five workers, against the ~72 GiB per drive the full migration needs:

seanfarm-worker-25ce78   313G      seanfarm-worker-42afba   345G
seanfarm-worker-b6ceba   348G      seanfarm-worker-07b866   345G
seanfarm-worker-a40389   328G

The 2+2 ratio is still an assumption and still unmeasured. It is called out in the manifest and must be measured (write 1 GiB, sum the four /data deltas) before the zot move relies on 220 GiB.

Guard

tests/rustfs.sh pins 110Gi and structurally couples the claim template name to the volumeMount that consumes it — a half-done rename produces a StatefulSet the API server accepts and pods that never mount a drive.

resize reverted to 55Gi                        detected
half-done rename (template only)               detected
half-done rename (mount only)                  detected
retention weakened to Retain                   detected
scale-down allowed to destroy a drive          detected

It also anchors the two retention assertions to their YAML lines. They were plain grep -F, and the comments added here quote both fields verbatim, so the whenDeleted control read NOT DETECTED until it was anchored — the third time this session a guard has matched its own explanation.

bash tests/check.sh exits 0.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

RustFS was provisioned at 4 x 55Gi -- about 110 GiB usable at 2+2 -- to match MinIO's per-drive size so the two were comparable. It has to hold the **144 GiB** now on MinIO, and **136 GiB of that is the zot bucket alone**, so at 55Gi the migration would have failed at its last and largest step. 4 x 110Gi is about 220 GiB usable. This is Phase 0-B of the MinIO to RustFS migration. Phase 0-A (`98d8c77a`) is live: `persistentVolumeClaimRetentionPolicy.whenDeleted: Delete` plus `force: true`, with ownerReferences confirmed present on all four claims. ## Why the claim template is renamed `data` becomes `drive`, and **the rename is what makes the resize deterministic** rather than a coin flip. `volumeClaimTemplates` are immutable. Confirmed against the live API server: ``` The StatefulSet "rustfs" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden ``` So this is a delete-and-recreate, performed by `force: true`. Phase 0-A makes the delete garbage-collect the drives. **That is necessary but not sufficient.** A claim is named `<template>-<statefulset>-<ordinal>`. Resizing alone keeps the name `data-rustfs-N`, and then: 1. Flux deletes the StatefulSet and re-applies it **in the same reconcile**, background propagation 2. the old claims stay pinned by `kubernetes.io/pvc-protection` until their pods finish terminating — tens of seconds 3. if the recreated pods reach scheduling first, they bind the surviving **55Gi** claims Nothing errors. The store just stays small — the exact failure this line of work exists to prevent. Renaming removes the race instead of hoping to win it. The new pods ask for `drive-rustfs-N`, which cannot already exist, so they always get fresh claims at 110Gi. The old claims are owned by the deleted StatefulSet's UID and are collected on their own schedule; the ordering cannot change the outcome. Their PVs are `reclaimPolicy: Delete`, so the ZFS datasets go too. ## This is safe exactly once RustFS holds 384K. The manifest records what to do **instead** next time: `zfs-nvme-128k` sets `allowVolumeExpansion: true` and the CSI driver supports it, so a later resize expands the four PVCs in place and then brings the template up to match. **Recreating this StatefulSet once it holds data is a data-loss event, not a resize.** ## Pre-flight `tank` free on all five workers, against the ~72 GiB per drive the full migration needs: ``` seanfarm-worker-25ce78 313G seanfarm-worker-42afba 345G seanfarm-worker-b6ceba 348G seanfarm-worker-07b866 345G seanfarm-worker-a40389 328G ``` The 2+2 ratio is **still an assumption and still unmeasured**. It is called out in the manifest and must be measured (write 1 GiB, sum the four `/data` deltas) before the zot move relies on 220 GiB. ## Guard `tests/rustfs.sh` pins 110Gi and **structurally couples** the claim template name to the volumeMount that consumes it — a half-done rename produces a StatefulSet the API server accepts and pods that never mount a drive. ``` resize reverted to 55Gi detected half-done rename (template only) detected half-done rename (mount only) detected retention weakened to Retain detected scale-down allowed to destroy a drive detected ``` It also **anchors the two retention assertions** to their YAML lines. They were plain `grep -F`, and the comments added here quote both fields verbatim, so the `whenDeleted` control read NOT DETECTED until it was anchored — the third time this session a guard has matched its own explanation. `bash tests/check.sh` exits 0. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit 2026-09-08 08:22:28 +00:00

feat(rustfs): grow the erasure set to 4 x 110Gi, deterministically ... 34900385fa

RustFS was provisioned at 4 x 55Gi -- about 110 GiB usable at 2+2 -- to match
MinIO's per-drive size so the two were comparable. It has to hold the 144 GiB
now on MinIO, and 136 GiB of that is the zot bucket alone, so at 55Gi the
migration would have failed at its last and largest step. 4 x 110Gi is about
220 GiB usable.

WHY THE CLAIM TEMPLATE IS RENAMED. `data` becomes `drive`, and the rename is
what makes the resize deterministic rather than a coin flip.

volumeClaimTemplates are immutable -- confirmed against the live API server:

  The StatefulSet "rustfs" is invalid: spec: Forbidden: updates to statefulset
  spec for fields other than 'replicas', 'ordinals', 'template',
  'updateStrategy', 'revisionHistoryLimit',
  'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

so this is a delete-and-recreate, performed by force: true on the Kustomization.
98d8c77a added persistentVolumeClaimRetentionPolicy.whenDeleted: Delete so the
delete garbage-collects the drives, and the ownerReferences are confirmed live
on all four claims. That is necessary but NOT sufficient.

A claim is named <template>-<statefulset>-<ordinal>. Resizing alone keeps the
name data-rustfs-N, and then Flux deletes the StatefulSet and re-applies it in
the SAME reconcile with background propagation, while the old claims stay pinned
by kubernetes.io/pvc-protection until their pods finish terminating -- tens of
seconds. If the recreated pods reach scheduling first they bind the surviving
55Gi claims and the resize silently does nothing. Nothing errors. The store just
stays small, which is the failure this whole line of work exists to prevent.

Renaming removes the race instead of hoping to win it: the new pods ask for
drive-rustfs-N, which cannot already exist, so they always get fresh claims at
110Gi. The old claims are owned by the deleted StatefulSet's UID and are
collected on their own schedule; the ordering cannot change the outcome. Their
PVs are reclaimPolicy: Delete, so the ZFS datasets go with them.

This is safe to do exactly once. RustFS holds 384K. The manifest records what to
do instead next time: zfs-nvme-128k sets allowVolumeExpansion: true and the CSI
driver supports it, so a later resize expands the four PVCs in place and then
brings the template up to match. Recreating this StatefulSet once it holds data
is a data-loss event, not a resize.

Pre-flight: tank has 313-348 GiB free on all five workers, against the ~72 GiB
per drive the full migration needs.

The 2+2 ratio is still an assumption and is still unmeasured. It is called out
in the manifest and must be measured before the zot move relies on 220 GiB.

Guard: tests/rustfs.sh pins 110Gi and structurally couples the claim template
name to the volumeMount that consumes it -- a half-done rename produces a
StatefulSet the API server accepts and pods that never mount a drive. Five
negative controls, all detecting.

It also ANCHORS the two retention assertions to their YAML lines. They were
plain grep -F, and the comments added here quote both fields verbatim, so the
whenDeleted control read NOT DETECTED until it was anchored -- the third time in
this session a guard has matched its own explanation.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 34900385fa into trunk 2026-09-08 08:28:27 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!445

No description provided.