sean/olly
Watch
1
0
Fork
You've already forked olly
 0
Code Issues Pull requests Projects Releases 60 Packages Wiki Activity

chore: move loki, tempo and victoriametrics to hcloud-volumes #41

Merged
binjovi-bot merged 1 commit from chore/hcloud-volumes into trunk 2026-09-03 20:15:12 +00:00
Conversation 0 Commits 1 Files changed 3 +28 -11
sean commented 2026-09-03 20:14:38 +00:00
Owner
Copy link

Part of emptying Mayastor so its 25.5 cores and 14.8 GiB come back to the fleet. This is the last of the workload repos; only Forgejo remains after it.

A volumeClaimTemplate is immutable, so each StatefulSet has to be deleted with --cascade=orphan and rebuilt by Argo CD before its claim can be replaced. Loki and Tempo therefore also move whenDeleted from Delete to Retain: with Delete the claim carries an ownerReference and a plain delete would take it with the StatefulSet.

Loki keeps its chunks in platform-loki-chunks and Tempo its traces in platform-tempo-traces, so a fresh volume costs only unflushed recent data.

VictoriaMetrics is the one with something to lose. Each replica holds its own 14 days, vmagent dual-writes by pod DNS name, and vmsingle has no peer repair, so a replaced replica starts empty and only fills forward. The plan is to swap one replica, copy the window across with vmctl vm-native, verify a 13-day-old sample, then swap the other. Both replicas hold the same window today, so either can seed the other. Doing them one at a time is what keeps a complete copy in existence throughout.

The repl1 reasoning on VictoriaMetrics is kept rather than discarded: redundancy lives in the application, so storage-level replication only duplicates it. hcloud-volumes is a single network-attached copy, the same shape as repl1, and it follows its pod across nodes.

bash tests/check.sh passes.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Part of emptying Mayastor so its 25.5 cores and 14.8 GiB come back to the fleet. This is the last of the workload repos; only Forgejo remains after it. A `volumeClaimTemplate` is immutable, so each StatefulSet has to be deleted with `--cascade=orphan` and rebuilt by Argo CD before its claim can be replaced. Loki and Tempo therefore also move `whenDeleted` from `Delete` to `Retain`: with `Delete` the claim carries an ownerReference and a plain delete would take it with the StatefulSet. Loki keeps its chunks in `platform-loki-chunks` and Tempo its traces in `platform-tempo-traces`, so a fresh volume costs only unflushed recent data. **VictoriaMetrics is the one with something to lose.** Each replica holds its own 14 days, vmagent dual-writes by pod DNS name, and vmsingle has no peer repair, so a replaced replica starts empty and only fills forward. The plan is to swap one replica, copy the window across with `vmctl vm-native`, verify a 13-day-old sample, then swap the other. Both replicas hold the same window today, so either can seed the other. Doing them one at a time is what keeps a complete copy in existence throughout. The repl1 reasoning on VictoriaMetrics is kept rather than discarded: redundancy lives in the application, so storage-level replication only duplicates it. `hcloud-volumes` is a single network-attached copy, the same shape as repl1, and it follows its pod across nodes. `bash tests/check.sh` passes. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
chore: move loki, tempo and victoriametrics to hcloud-volumes
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
b512d271af
 
Part of emptying Mayastor so its 25.5 cores and 14.8 GiB come back to
the fleet.

A volumeClaimTemplate is immutable, so each StatefulSet must be deleted
with --cascade=orphan and rebuilt by Argo CD before its claim can be
replaced. Loki and Tempo therefore also move whenDeleted from Delete to
Retain: with Delete the claim carries an ownerReference and a plain
delete would take it. VictoriaMetrics already retains.

Loki keeps its chunks in the MinIO platform-loki-chunks bucket and Tempo
its traces in platform-tempo-traces, so a fresh volume costs only
unflushed recent data.

VictoriaMetrics is different. Each replica holds its own 14 days,
vmagent dual-writes by pod DNS name, and vmsingle has no peer repair, so
a replaced replica starts empty and only fills forward. Swap one replica,
copy the window across with vmctl, verify, then swap the other. Both
replicas hold the same window today, so either can seed the other.

The repl1 reasoning on VictoriaMetrics is kept, not discarded: redundancy
lives in the application, so storage-level replication only duplicates
it. hcloud-volumes is a single network-attached copy, the same shape as
repl1, and it follows its pod across nodes.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/olly!41
No description provided.