feat(metrics): back VictoriaMetrics up to Hetzner Object Storage #22

Merged

sean merged 1 commit from feat/vmbackup into trunk

2026-08-10 11:33:29 +00:00

sean commented

2026-08-10 11:32:41 +00:00

Owner

VM had no off-cluster backup. Its data lived only on a node-local mayastor volume, so a rebuild or lost volume took every metric with it. In-cluster replication is not a backup.

Adds a vmbackup sidecar writing to s3://seanfarm/vm-backups/ - the same bucket barman uses, separate prefix. The credential is placed into ns olly by seanfarm PR #988 (already merged and verified live).

Three choices worth recording

Sidecar, not CronJob. vmbackup must READ the VM data directory, so it mounts the same RWO volume. In the same pod that is free; a CronJob would have to be pinned to whichever node holds the volume.
Shell loop. OSS vmbackup is ONE-SHOT with no scheduler (that is vmbackupmanager, enterprise). The image ships busybox, so a loop is the supported OSS shape. 6h interval matches CNPG.
-snapshot.createURL. VM takes its own consistent snapshot and deletes it after. No volume-level snapshot is needed - which is exactly why moving the volume to ZFS to get snapshots was not necessary.

-dst is keyed by POD_NAME so a second replica cannot overwrite the first one's backup, and pointing -dst at the same path each cycle is what makes the upload incremental.

Next: two vmsingle copies on mayastor-repl1 with vmagent dual remote-write, so redundancy moves into VM and off repl5.

VM had **no off-cluster backup**. Its data lived only on a node-local mayastor volume, so a rebuild or lost volume took every metric with it. In-cluster replication is not a backup. Adds a `vmbackup` sidecar writing to `s3://seanfarm/vm-backups/` - the same bucket barman uses, separate prefix. The credential is placed into ns `olly` by seanfarm PR #988 (already merged and verified live). ### Three choices worth recording - **Sidecar, not CronJob.** vmbackup must READ the VM data directory, so it mounts the same RWO volume. In the same pod that is free; a CronJob would have to be pinned to whichever node holds the volume. - **Shell loop.** OSS vmbackup is ONE-SHOT with no scheduler (that is `vmbackupmanager`, enterprise). The image ships busybox, so a loop is the supported OSS shape. 6h interval matches CNPG. - **`-snapshot.createURL`.** VM takes its own consistent snapshot and deletes it after. **No volume-level snapshot is needed - which is exactly why moving the volume to ZFS to get snapshots was not necessary.** `-dst` is keyed by `POD_NAME` so a second replica cannot overwrite the first one's backup, and pointing `-dst` at the same path each cycle is what makes the upload incremental. Next: two vmsingle copies on mayastor-repl1 with vmagent dual remote-write, so redundancy moves into VM and off repl5.

sean added 1 commit

2026-08-10 11:32:43 +00:00

feat(metrics): back VictoriaMetrics up to Hetzner Object Storage

pipeline/ci CI green @ 309e25c27e52

Details

309e25c27e

VM had NO off-cluster backup. Its data lived only on a node-local
mayastor volume, so a rebuild or a lost volume took every metric with
it. In-cluster replication is not a backup.

Adds a vmbackup sidecar writing to s3://seanfarm/vm-backups/, the same
bucket barman uses, under a separate prefix. The credential is placed
into ns olly by the seanfarm Flux Kustomization vm-offsite-s3.

Three choices worth recording:

A sidecar, not a CronJob. vmbackup must READ the VM data directory, so
it mounts the same RWO volume. In the same pod that is free; a CronJob
would have to be pinned to whichever node holds the volume.

A shell loop, because OSS vmbackup is ONE-SHOT and has no scheduler.
The scheduler is vmbackupmanager, which is enterprise. The image ships
busybox, so a loop is the supported OSS shape. Interval 6h matches the
CNPG schedule.

-snapshot.createURL, so VM takes its own consistent snapshot and deletes
it after. No volume-level snapshot is needed. This is exactly why moving
the volume to ZFS to get snapshots was NOT necessary.

-dst is keyed by POD_NAME so a second replica cannot overwrite the
first one's backup. Pointing -dst at the same path each cycle is what
makes the upload incremental.

sean merged commit 309e25c27e into trunk

2026-08-10 11:33:29 +00:00

sean referenced this pull request

2026-08-10 11:45:55 +00:00

fix(metrics): stop vmbackup being OOMKilled at 256Mi #23