feat(rustfs): install a parallel S3 to evaluate a move off MinIO #411

Merged

binjovi-bot merged 1 commit from feat/rustfs into trunk

2026-09-07 14:06:23 +00:00

sean commented

2026-09-07 14:02:51 +00:00

Owner

Raw manifests, no Helm. Four pods, one 55Gi drive each on zfs-nvme-128k — a 4-drive erasure set, 2 data + 2 parity, one per worker. 55Gi matches MinIO-s per-drive size so the two are comparable.

Nothing migrates. MinIO keeps every live bucket: zot-s OCI blobs, Forgejo storage, Loki, Tempo, pkg, Binjovi logs. This is a parallel install so the replacement can be measured before it is trusted, on a release candidate (1.0.0-rc.5) — right maturity for a dev cluster, wrong maturity for the only copy of anything.

The blocker, recorded where the next reader will find it

RustFS implements the S3 API fully but its MinIO admin API is incomplete: mc admin user list and mc admin policy list fail against it. Every scoped identity here depends on that API — the tenantbucket converge Jobs (mc admin policy create / user add / policy attach / ilm rule import, plus the negative bucket-isolation checks from H29/H30), the minio.sean.farm CRDs, and the Operator Tenant. See rustfs#1571. If scoped credentials cannot be provisioned declaratively, RustFS cannot replace MinIO here regardless of how a benchmark reads.

zfs-nvme-128k, not zfs-buildkit-128k: that family is named for its consumer and is not the right home for object data.

The part worth reviewing: the start-up deadlock

The readiness probe passes only once storage quorum forms, and quorum needs the pods to resolve each other first. publishNotReadyAddresses on the headless Service plus podManagementPolicy: Parallel is what breaks it. Get either wrong and you have four pods that never become Ready with nothing in any log saying why — and neither is visible in a render or a dry run.

tests/rustfs.sh pins both, plus the digest pin, replica/drive-list agreement, per-node anti-affinity, the restricted pod posture, and that no credential is ever written into a manifest. Verified against six mutations, each detected.

No ingress — ClusterIP and headless only, console by port-forward. Publishing it needs a Gateway listener and an LE certificate, which is a deliberate decision and not a side effect of an evaluation, so this cannot touch the LE-prod rate limit. The guard fails if an HTTPRoute, Gateway or Certificate appears in the directory.

Exempted from kernel-roll gating: a release-candidate object store that nothing in the kernel path reads must never hold a roll.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Raw manifests, no Helm. Four pods, one 55Gi drive each on `zfs-nvme-128k` — a 4-drive erasure set, 2 data + 2 parity, one per worker. 55Gi matches MinIO-s per-drive size so the two are comparable. **Nothing migrates.** MinIO keeps every live bucket: zot-s OCI blobs, Forgejo storage, Loki, Tempo, `pkg`, Binjovi logs. This is a parallel install so the replacement can be measured before it is trusted, on a **release candidate** (1.0.0-rc.5) — right maturity for a dev cluster, wrong maturity for the only copy of anything. ### The blocker, recorded where the next reader will find it RustFS implements the S3 API fully but its **MinIO admin API is incomplete**: `mc admin user list` and `mc admin policy list` fail against it. Every scoped identity here depends on that API — the `tenantbucket` converge Jobs (`mc admin policy create` / `user add` / `policy attach` / `ilm rule import`, plus the negative bucket-isolation checks from H29/H30), the `minio.sean.farm` CRDs, and the Operator `Tenant`. See [rustfs#1571](https://github.com/rustfs/rustfs/issues/1571). **If scoped credentials cannot be provisioned declaratively, RustFS cannot replace MinIO here regardless of how a benchmark reads.** `zfs-nvme-128k`, not `zfs-buildkit-128k`: that family is named for its consumer and is not the right home for object data. ### The part worth reviewing: the start-up deadlock The readiness probe passes only once storage quorum forms, and quorum needs the pods to resolve each other first. `publishNotReadyAddresses` on the headless Service **plus** `podManagementPolicy: Parallel` is what breaks it. Get either wrong and you have four pods that never become Ready with nothing in any log saying why — and neither is visible in a render or a dry run. `tests/rustfs.sh` pins both, plus the digest pin, replica/drive-list agreement, per-node anti-affinity, the restricted pod posture, and that no credential is ever written into a manifest. Verified against six mutations, each detected. **No ingress** — ClusterIP and headless only, console by port-forward. Publishing it needs a Gateway listener and an LE certificate, which is a deliberate decision and not a side effect of an evaluation, so this cannot touch the LE-prod rate limit. The guard fails if an HTTPRoute, Gateway or Certificate appears in the directory. Exempted from kernel-roll gating: a release-candidate object store that nothing in the kernel path reads must never hold a roll. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit

2026-09-07 14:02:52 +00:00

feat(rustfs): install a parallel S3 to evaluate a move off MinIO

binjovi/ci Binjovi completed the frozen plan

Details

084ee93b6a

Raw manifests, no Helm. Four pods, one 55Gi drive each on zfs-nvme-128k: a
4-drive erasure set, 2 data + 2 parity, one per worker. 55Gi matches MinIO's
per-drive size so the two are comparable.

NOTHING MIGRATES. MinIO keeps every live bucket -- zot's OCI blobs, Forgejo
storage, Loki, Tempo, pkg, Binjovi logs. This is a parallel install so the
replacement can be measured before it is trusted, on a release candidate
(1.0.0-rc.5) that is the right maturity for a dev cluster and the wrong
maturity for the only copy of anything.

THE BLOCKER, recorded where the next reader will find it. RustFS implements the
S3 API fully but its MinIO ADMIN API is incomplete: `mc admin user list` and
`mc admin policy list` fail. Every scoped identity here depends on that API --
the tenantbucket converge Jobs that run `mc admin policy create` / `user add` /
`policy attach` / `ilm rule import` and then negatively verify bucket isolation
(H29/H30), the minio.sean.farm CRDs, and the Operator Tenant. See
rustfs/rustfs#1571. If scoped credentials cannot be provisioned declaratively,
RustFS cannot replace MinIO here regardless of how a benchmark reads.

zfs-nvme-128k, not zfs-buildkit-128k: that family is named for its consumer and
is not the right home for object data. If recordsize turns out to matter here
too it gets its own matched family.

THE START-UP DEADLOCK is the part worth reviewing. The readiness probe passes
only once storage quorum forms, and quorum needs the pods to resolve each other
first. publishNotReadyAddresses on the headless Service plus podManagementPolicy
Parallel is what breaks it; getting either wrong yields four pods that never
become Ready with nothing in any log saying why, and neither is visible in a
render or a dry run. tests/rustfs.sh pins both, along with the digest pin, the
replica/drive-list agreement, per-node anti-affinity, the restricted pod
posture, and that no credential is ever written into a manifest. Verified
against six mutations, each detected.

No ingress: ClusterIP and headless only, console by port-forward. Publishing it
needs a Gateway listener and an LE certificate, which is a deliberate decision
and not a side effect of an evaluation -- so this cannot touch the LE-prod rate
limit. The guard fails if an HTTPRoute, Gateway or Certificate appears here.

Exempted from kernel-roll gating: a release-candidate object store that nothing
in the kernel path reads must never hold a roll.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76