feat(rustfs): mirror buckets off MinIO, starting with tempo #455

Merged

binjovi-bot merged 1 commit from feat/rustfs-mirror into trunk 2026-09-08 11:05:23 +00:00

Conversation 0 Commits 1 Files changed 9 +456 -1

sean commented 2026-09-08 10:59:29 +00:00

Owner

Copy link

Bulk object movement, one Job per bucket, plus the shared read-only source identity. platform-tempo-traces first: 8.8 MiB, 268 objects — the smallest and most survivable bucket. The MinIO copy is left untouched and stays the rollback until MinIO retires.

Neither side uses root

The obvious implementation points mc at MinIO root on one side and RustFS root on the other. A mirror needs exactly two rights — read the source, write the destination — so this supplies both at that grade:

side	credential	scope
source	new MinIO migration-reader	GetObject + ListBucket on the 8 migrating buckets, nothing else
destination	tempo's own scoped RustFS key	write to platform-tempo-traces only

Root on either side would put a credential able to delete every bucket in the cluster into a Job whose only purpose is bulk copying — the exact credential a mistake in this directory would leak.

The reader proves it is read-only rather than asserting it: it lists a source bucket, then fails if a write succeeds, and fails if it can reach cnpg-backups, which is not migrating.

Both endpoints are the stable ClusterIP Services

A bulk transfer is precisely the case the headless Service breaks: it hands out individual server addresses, and a transfer that loses its server mid-flight fails with HTTP 499. That happened twice during the 2026-09-02 node roll, and sean/zot carries a guard forbidding it. MinIO's own consumers still use minio-hl; this does not.

The parity gate is the output, not the exit status

mc diff exits 0 whether or not the buckets differ. A Job that trusted the exit code would report a half-copied bucket as a success. The gate is [ -s /tmp/diff.out ].

One race, named rather than hidden

Tempo keeps writing to MinIO until it is repointed, so an object written between the mirror and the diff shows as missing and fails the Job — which then retries and picks it up. At 268 objects that converges immediately. The same pattern on binjovi-logs (207k objects) will need the consumer paused or a final incremental pass after the repoint, and that is called out in the manifest for whoever does it.

Guard

tests/rustfs-mirror.sh iterates the directory so later buckets are covered when their file lands. It pins: no root credential on either side, ClusterIP on both, and a diff gate that actually gates.

root credential on the destination   detected
headless endpoint                    detected
diff gate removed                    detected
parity check removed                 detected

upstream-inventory.txt 7 → 9 for the two new minio/mc pins. bash tests/check.sh exits 0. Server-side dry run applies all 10 objects clean.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Bulk object movement, one Job per bucket, plus the shared read-only source identity. `platform-tempo-traces` first: 8.8 MiB, 268 objects — the smallest and most survivable bucket. **The MinIO copy is left untouched** and stays the rollback until MinIO retires. ## Neither side uses root The obvious implementation points `mc` at MinIO root on one side and RustFS root on the other. A mirror needs exactly two rights — read the source, write the destination — so this supplies both at that grade: | side | credential | scope | |---|---|---| | source | new MinIO `migration-reader` | `GetObject` + `ListBucket` on the 8 migrating buckets, nothing else | | destination | tempo's own scoped RustFS key | write to `platform-tempo-traces` only | Root on either side would put a credential able to delete every bucket in the cluster into a Job whose only purpose is bulk copying — the exact credential a mistake in this directory would leak. The reader **proves** it is read-only rather than asserting it: it lists a source bucket, then fails if a write succeeds, and fails if it can reach `cnpg-backups`, which is not migrating. ## Both endpoints are the stable ClusterIP Services A bulk transfer is precisely the case the headless Service breaks: it hands out individual server addresses, and a transfer that loses its server mid-flight fails with **HTTP 499**. That happened twice during the 2026-09-02 node roll, and `sean/zot` carries a guard forbidding it. MinIO's own consumers still use `minio-hl`; this does not. ## The parity gate is the output, not the exit status `mc diff` **exits 0 whether or not the buckets differ**. A Job that trusted the exit code would report a half-copied bucket as a success. The gate is `[ -s /tmp/diff.out ]`. ## One race, named rather than hidden Tempo keeps writing to MinIO until it is repointed, so an object written between the mirror and the diff shows as missing and fails the Job — which then retries and picks it up. At 268 objects that converges immediately. **The same pattern on `binjovi-logs` (207k objects) will need the consumer paused or a final incremental pass after the repoint**, and that is called out in the manifest for whoever does it. ## Guard `tests/rustfs-mirror.sh` iterates the directory so later buckets are covered when their file lands. It pins: no root credential on either side, ClusterIP on both, and a diff gate that actually gates. ``` root credential on the destination detected headless endpoint detected diff gate removed detected parity check removed detected ``` `upstream-inventory.txt` 7 → 9 for the two new `minio/mc` pins. `bash tests/check.sh` exits 0. Server-side dry run applies all 10 objects clean. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit 2026-09-08 10:59:30 +00:00

feat(rustfs): mirror buckets off MinIO, starting with tempo ... 1f1d43832b

Bulk object movement, one Job per bucket, plus the shared read-only source
identity. platform-tempo-traces first: 8.8 MiB and 268 objects, the smallest and
most survivable bucket. The MinIO copy is left untouched and stays the rollback
until MinIO retires.

NEITHER SIDE USES ROOT. The obvious implementation points mc at MinIO root on
one side and RustFS root on the other. A mirror needs exactly two rights -- read
the source, write the destination -- so this supplies both at that grade: a new
read-only MinIO `migration-reader` scoped to the eight migrating buckets, and, on
the destination, the bucket's own scoped RustFS key from rustfs-identities.
Root on either side would put a credential able to delete every bucket in the
cluster into a Job whose only purpose is bulk copying.

The reader PROVES it is read-only rather than asserting it: it lists a source
bucket, then fails if a write succeeds, and fails if it can reach cnpg-backups,
which is not migrating.

BOTH ENDPOINTS ARE THE STABLE ClusterIP SERVICES. A bulk transfer is precisely
the case the headless Service breaks -- it hands out individual server addresses,
and a transfer that loses its server mid-flight fails with HTTP 499. That
happened twice during the 2026-09-02 node roll and sean/zot carries a guard
forbidding it. MinIO's own consumers still use minio-hl; this does not.

The parity gate is the OUTPUT of `mc diff`, not its exit status. mc diff exits 0
whether or not the buckets differ, so a Job that trusted the exit code would
report a half-copied bucket as a success.

One race is named rather than hidden: Tempo keeps writing to MinIO until it is
repointed, so an object written between the mirror and the diff shows as missing
and fails the Job, which then retries and picks it up. At 268 objects that
converges immediately. The same pattern on binjovi-logs (207k objects) will need
the consumer paused or a final incremental pass after the repoint.

tests/rustfs-mirror.sh iterates the directory so later buckets are covered when
their file lands, and pins: no root credential on either side, ClusterIP on both,
and a diff gate that actually gates. Four negative controls, all detecting.

upstream-inventory.txt 7 -> 9 for the two new minio/mc pins.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 1f1d43832b into trunk 2026-09-08 11:05:22 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!455

No description provided.