feat(rustfs): scoped per-consumer identities, starting with tempo #451
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/rustfs-provisioning"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Groundwork for moving buckets off MinIO.
rustfs-identities/is the RustFS counterpart oftenantbucket/: it provisions a bucket, a policy and a key for one consumer, then proves the key can reach nothing else. The two directories run side by side for the length of the migration;tenantbucket/retires with MinIO.Tempo is first because it is the smallest and most survivable bucket (8.8 MiB, 268 objects).
One identity per consumer
On MinIO this bucket is reached by
olly-platform, a single key whose policy grantsplatform-*— a wildcard over bucket names — plusworkflows,workflows/*andpkg, carried by five Secrets: loki, tempo, the Argo artifact repository, sccache and package reads.docs/object-store-identities.mdrecords it as the widest credential in the tenant and the largest security improvement available in this migration.Splitting it costs one extra user at the moment the key has to be created on RustFS anyway, so it is done now rather than deferred.
The endpoint is the ClusterIP Service
MinIO's consumers use
minio-hl, andsean/zotcarries a guard forbidding exactly that. The headless Service hands out individual server addresses, so an upload that selects a server during a node roll loses it and fails with HTTP 499 — that happened twice during the 2026-09-02 node roll. RustFS'srustfsService is a stable ClusterIP, so the migration does not carry the latent bug forward.Measured, not assumed
Whether this pattern was reusable at all turned on whether RustFS implements the MinIO admin API the Jobs depend on. Every step was run against the live rc.5 cluster before being written down:
The isolation check uses a dedicated
isolation-canarybucket rather than pairing identities against each other the way the MinIO Jobs do (forgejo against zot, binjovi-logs against forgejo). That pairing rots as buckets come and go during a migration, and a negative check against a bucket that does not exist passes for the wrong reason.The guard is written as a class, not a case
tests/rustfs-identities.shiterates the directory instead of naming consumers, so the six identities still to come are covered the moment their file lands. It pins the ClusterIP endpoint, both halves of the isolation proof, namespace conditions on every ClusterSecretStore, the absence of wildcard bucket grants, and unconditionaluser add.Two things it learned the hard way, both fixed:
Housekeeping
upstream-inventory.txt6 → 7 for the newminio/mcpin (the reverse check fails on an exact count).rustfs-identities, with a note that it must lose the exemption when the zot bucket moves — the kernel image spine then cannot pull unless RustFS can hand out zot's key.bash tests/check.shexits 0. Server-side dry run applies all 8 objects clean.https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
Phase 2/3 groundwork for the MinIO migration. rustfs-identities/ is the RustFS counterpart of tenantbucket/: it provisions a bucket, a policy and a key for one consumer, then PROVES the key can reach nothing else. The two directories run side by side for the length of the migration; tenantbucket/ retires with MinIO. Tempo is first because it is the smallest and most survivable bucket (8.8 MiB, 268 objects). ONE IDENTITY PER CONSUMER. On MinIO this bucket is reached by `olly-platform`, a single key whose policy grants `platform-*` -- a WILDCARD over bucket names -- plus workflows, workflows/* and pkg, carried by five Secrets: loki, tempo, the Argo artifact repository, sccache and package reads. docs/object-store-identities.md records it as the widest credential in the tenant and the largest security improvement available here. Splitting it costs one extra user at the moment the key has to be created on RustFS anyway, so it is done now rather than deferred. THE ENDPOINT IS THE CLUSTERIP SERVICE. MinIO's consumers use minio-hl, and sean/zot carries a guard forbidding exactly that: the headless Service hands out individual server addresses, so an upload that selects a server during a node roll loses it and fails with HTTP 499. That happened twice during the 2026-09-02 node roll. RustFS's `rustfs` Service is a stable ClusterIP, so the migration does not carry the latent bug forward. MEASURED, NOT ASSUMED. The reusability of this whole pattern turned on whether RustFS implements the MinIO admin API the Jobs depend on. Every step was run against the live rc.5 cluster before being written down: mc ready, mb, admin policy create, admin user add, admin policy attach, and both halves of the isolation check. All pass, including the negative on read AND on write. The isolation check uses a dedicated `isolation-canary` bucket rather than pairing identities against each other the way the MinIO Jobs do (forgejo against zot, binjovi-logs against forgejo). That pairing rots as buckets come and go during a migration, and a negative check against a bucket that does not exist passes for the wrong reason. tests/rustfs-identities.sh iterates the directory instead of naming consumers, so the six identities still to come are covered the moment their file lands. It pins the ClusterIP endpoint, both halves of the isolation proof, namespace conditions on every ClusterSecretStore, the absence of wildcard bucket grants, and unconditional `user add` (an ancestor guarded that behind `user info ||`, silently skipped a rotation, exited 0, and Forgejo CrashLooped days later). Two things the guard learned the hard way, both fixed: * it first matched the manifest's own explanatory comments, which quote the wrong forms verbatim, so executable-content checks now read a comment-stripped copy. That is the fourth time in this work a guard has read its own explanation and passed. * its negative controls re-invoke the script, so a control that failed to trip would have recursed without bound. The inner run now stops before that section: a hang is a worse failure than a wrong answer, because it looks like a slow CI job. upstream-inventory.txt goes 6 -> 7 for the new minio/mc pin, and the roll-gate allow-list gains rustfs-identities with a note that it MUST lose the exemption when the zot bucket moves -- the kernel image spine then cannot pull unless RustFS can hand out zot's key. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76