feat(traces): move Tempo's storage from MinIO to RustFS #63
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/tempo-rustfs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
First consumer off MinIO.
platform-tempo-tracesis 8.8 MiB and 268 objects — the smallest and most survivable bucket, which is why it goes first.The data is already there and proven.
sean/seanfarm'srustfs-mirrorJob copied the bucket and gated onmc diffbeing empty:The MinIO bucket is left untouched and is the rollback until MinIO retires.
What changed
rustfs.rustfs.svc.cluster.local:9000/minio/health/readywith 200 (verified live), so only the host movesrustfs-tempo-s3-credentials, which admits namespaceollyonlySecret key names stay
minio_access_key/minio_secret_key, so this is an endpoint change and not also a key rename. The init container is renamedwait-for-rustfs, because one calledwait-for-miniothat waits for RustFS is a trap for the next reader.The endpoint is the stable ClusterIP Service
Tempo was on
minio-hl.sean/zotcarries a guard forbidding exactly that shape: the headless Service hands out individual server addresses, so a transfer that selects a server during a node roll loses it and fails with HTTP 499 — twice during the 2026-09-02 node roll. That latent bug is not carried forward.Credential scope narrows
Tempo used
olly-platform: one key whose policy grantsplatform-*— a wildcard over bucket names — plusworkflows,workflows/*andpkg, shared by five Secrets. It now uses a key that can reachplatform-tempo-tracesand nothing else, proved by a negative isolation check at provisioning time.create-bucketsis removed, and it had to berustfs-identitiesnow creates the bucket declaratively, and the scoped key deliberately has nos3:CreateBucket. Verified against the live store rather than assumed:--ignore-existingdoes not save it — mc still asks, and the answer is no. Left in place, this container would have put Tempo into CrashLoopBackOff on the repoint.Verification after merge
Tempo pod Ready, no auth errors in its log, and a trace queried through Grafana. ArgoCD auto-syncs this path from
mainwithpruneandselfHeal, so the merge is the deploy.https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
First consumer off MinIO. platform-tempo-traces is 8.8 MiB and 268 objects, the smallest and most survivable bucket, which is why it goes first. The data is already there and proven: sean/seanfarm's rustfs-mirror Job copied the bucket and gated on `mc diff` being empty -- 6.83 MiB transferred, then ==> platform-tempo-traces is object-for-object identical on RustFS The MinIO bucket is left untouched and is the rollback until MinIO retires. WHAT CHANGED * storage endpoint -> rustfs.rustfs.svc.cluster.local:9000 * the wait probe -> the same host. RustFS answers /minio/health/ready with 200 (verified against the live cluster), so only the host moves. The container is renamed wait-for-rustfs, because a container called wait-for-minio that waits for RustFS is a trap for the next reader. * credentials -> ClusterSecretStore rustfs-tempo-s3-credentials, which admits namespace olly only. The Secret key names stay minio_access_key / minio_secret_key, so this is an endpoint change and not also a key rename. THE ENDPOINT IS THE STABLE ClusterIP SERVICE, not a headless one. Tempo was on minio-hl. sean/zot carries a guard forbidding exactly that shape because the headless Service hands out individual server addresses: a transfer that selects a server during a node roll loses it and fails with HTTP 499, twice during the 2026-09-02 node roll. That latent bug is not carried forward. CREDENTIAL SCOPE NARROWS. Tempo used `olly-platform`, one key whose policy grants platform-* -- a wildcard over bucket names -- plus workflows, workflows/* and pkg, shared by five Secrets. It now uses a key that can reach platform-tempo-traces and nothing else, proved by a negative isolation check at provisioning time. create-buckets IS REMOVED, and it had to be. rustfs-identities now creates the bucket declaratively, and the scoped key deliberately has no s3:CreateBucket. Verified against the live store rather than assumed: mc mb --ignore-existing r/platform-tempo-traces mc: <ERROR> Unable to make bucket. Access Denied `--ignore-existing` does not save it: mc still asks, and the answer is no. Left in place this container would have put Tempo in CrashLoopBackOff on the repoint. VERIFY AFTER MERGE: tempo pod Ready, no auth errors in its log, and a trace queried through Grafana. ArgoCD auto-syncs this path from main with prune and selfHeal, so the merge is the deploy. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76