sean/olly
Watch
1
0
Fork
You've already forked olly
 0
Code Issues Pull requests Projects Releases 60 Packages Wiki Activity

fix(metrics): seed an empty replica from the offsite backup #43

Merged
binjovi-bot merged 1 commit from chore/vm-restore into trunk 2026-09-03 20:53:09 +00:00
Conversation 0 Commits 1 Files changed 1 +60
sean commented 2026-09-03 20:52:38 +00:00
Owner
Copy link

victoriametrics-1 moved to hcloud-volumes and started empty, as designed. Each replica keeps its own 14 days, vmagent dual-writes by pod DNS name, and vmsingle has no peer repair, so a replaced replica only fills forward.

Why not vmctl. Copying the window with vmctl vm-native works but is far too slow: 8619 export requests against a server running --search.maxConcurrentRequests=2, measured at about five hours. Raising --vm-concurrency above 1 immediately 429s, because the copy and ordinary Grafana queries share those same two slots. Raising the servers limit would mean two more releases and would degrade queries while it ran.

vmrestore instead. It pulls the same window as one bulk transfer from Hetzner Object Storage in minutes, and never queries the serving replica. The backup already exists: the sidecar wrote 18.8 GB for victoriametrics-0 at 16:48 today.

The initContainer runs only when /storage holds no data, so it is inert on every normal restart and cannot touch a replica that already has data.

The source is victoriametrics-0 by name rather than ${POD_NAME}, because the point is to seed a replica whose own backup is gone. vmbackup makes S3 match local every 6h, so an emptied replicas own backup is emptied with it. That already happened to victoriametrics-1 at 20:20 today, deleting 13.9 GB. Worth knowing before anyone swaps replica 0.

Temporary. It should be removed once both replicas are on hcloud-volumes and populated, or replica 0s backup becomes a silent single point of truth.

bash tests/check.sh passes.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

`victoriametrics-1` moved to `hcloud-volumes` and started empty, as designed. Each replica keeps its own 14 days, vmagent dual-writes by pod DNS name, and vmsingle has no peer repair, so a replaced replica only fills forward. **Why not vmctl.** Copying the window with `vmctl vm-native` works but is far too slow: 8619 export requests against a server running `--search.maxConcurrentRequests=2`, measured at about **five hours**. Raising `--vm-concurrency` above 1 immediately 429s, because the copy and ordinary Grafana queries share those same two slots. Raising the servers limit would mean two more releases and would degrade queries while it ran. **vmrestore instead.** It pulls the same window as one bulk transfer from Hetzner Object Storage in minutes, and never queries the serving replica. The backup already exists: the sidecar wrote 18.8 GB for `victoriametrics-0` at 16:48 today. The initContainer runs **only when `/storage` holds no data**, so it is inert on every normal restart and cannot touch a replica that already has data. The source is `victoriametrics-0` by name rather than `${POD_NAME}`, because the point is to seed a replica whose *own* backup is gone. vmbackup makes S3 match local every 6h, so an emptied replicas own backup is emptied with it. That already happened to `victoriametrics-1` at 20:20 today, deleting 13.9 GB. Worth knowing before anyone swaps replica 0. **Temporary.** It should be removed once both replicas are on `hcloud-volumes` and populated, or replica 0s backup becomes a silent single point of truth. `bash tests/check.sh` passes. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
fix(metrics): seed an empty replica from the offsite backup
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
6c29ac576d
 
victoriametrics-1 moved to hcloud-volumes and started empty, as designed:
each replica keeps its own 14 days, vmagent dual-writes by pod DNS name,
and vmsingle has no peer repair, so a replaced replica only fills
forward.

Copying the window across with vmctl vm-native works but is far too slow.
It is 8619 export requests against a server running
--search.maxConcurrentRequests=2. Measured at about five hours, and it
429s immediately if vm-concurrency goes above 1, because the copy and
ordinary Grafana queries share those two slots.

vmrestore pulls the same window as one bulk transfer from Hetzner Object
Storage in minutes, and never queries the serving replica. The backup
already exists: the vmbackup sidecar wrote 18.8 GB for victoriametrics-0
at 16:48 today.

The initContainer runs only when /storage holds no data, so it is inert
on every normal restart and cannot touch a replica that has data.

The source is victoriametrics-0 by name, not POD_NAME. The whole point is
to seed a replica whose own backup is gone: vmbackup makes S3 match local
every 6h, so an emptied replica's own backup is emptied with it. That
already happened to victoriametrics-1 at 20:20 today, deleting 13.9 GB.

TEMPORARY. Remove it once both replicas are on hcloud-volumes and
populated, or replica 0's backup becomes a silent single point of truth.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/olly!43
No description provided.