feat(metrics): move VM redundancy into the app, off mayastor-repl5 #24

Merged

sean merged 1 commit from feat/vm-ha into trunk

2026-08-10 11:56:54 +00:00

sean commented

2026-08-10 11:55:06 +00:00

Owner

Redundancy for metrics lived entirely in the storage layer: one vmsingle on a repl5 volume. That is 5 storage replicas of a single point of failure - lose the pod or its data and there is no second VM.

Changes

2 replicas with REQUIRED podAntiAffinity on kubernetes.io/hostname. Two copies on one node is not redundancy, it is two copies that die together. 5 workers, 2 replicas, always satisfiable.
Headless service. A StatefulSet only gets stable per-pod DNS when its governing service is headless. serviceName is immutable, so this needs the STS recreated - which the repl1 change needs anyway.
vmagent dual write to both pods by name, so each vmsingle holds a COMPLETE copy. Do not collapse these back to the ClusterIP service: it load-balances, so samples would scatter and neither copy would be whole. The ClusterIP service stays, for reads.
repl5 -> repl1. With 2 app-level copies, 5 storage replicas duplicate the redundancy, and repl5 cost 200Gi of pool for 40Gi of data. The bigger reason: a repl1 volume has nothing to rebuild, so it cannot hit the ENOMEM defect currently blocking kernel rolls.

Known and deliberate

victoriametrics-0 keeps its existing repl5 PVC - a StatefulSet reuses a PVC whose name already matches. Only victoriametrics-1 gets a repl1 volume. Moving replica 0 off repl5 means dropping its PVC and restoring from the S3 backup; separate step, and safe now that vmbackup is verified.

victoriametrics-1 starts empty and fills forward. vmsingle has no peer repair, so it will not backfill history from replica 0.

Rollout note

The STS must be deleted with --cascade=orphan to change the immutable fields. That is safe now: PR #21 set whenDeleted: Retain, so the PVC is not garbage-collected with it.

Redundancy for metrics lived entirely in the storage layer: **one** vmsingle on a repl5 volume. That is 5 storage replicas of a single point of failure - lose the pod or its data and there is no second VM. ### Changes - **2 replicas with REQUIRED podAntiAffinity** on `kubernetes.io/hostname`. Two copies on one node is not redundancy, it is two copies that die together. 5 workers, 2 replicas, always satisfiable. - **Headless service.** A StatefulSet only gets stable per-pod DNS when its governing service is headless. `serviceName` is immutable, so this needs the STS recreated - which the repl1 change needs anyway. - **vmagent dual write** to both pods by name, so each vmsingle holds a COMPLETE copy. Do not collapse these back to the ClusterIP service: it load-balances, so samples would scatter and neither copy would be whole. The ClusterIP service stays, for reads. - **repl5 -> repl1.** With 2 app-level copies, 5 storage replicas duplicate the redundancy, and repl5 cost **200Gi of pool for 40Gi of data**. The bigger reason: **a repl1 volume has nothing to rebuild, so it cannot hit the ENOMEM defect currently blocking kernel rolls.** ### Known and deliberate `victoriametrics-0` **keeps its existing repl5 PVC** - a StatefulSet reuses a PVC whose name already matches. Only `victoriametrics-1` gets a repl1 volume. Moving replica 0 off repl5 means dropping its PVC and restoring from the S3 backup; separate step, and safe now that vmbackup is verified. `victoriametrics-1` starts **empty** and fills forward. vmsingle has no peer repair, so it will not backfill history from replica 0. ### Rollout note The STS must be deleted with `--cascade=orphan` to change the immutable fields. That is safe now: PR #21 set `whenDeleted: Retain`, so the PVC is not garbage-collected with it.

sean added 1 commit

2026-08-10 11:55:07 +00:00

feat(metrics): move VM redundancy into the app, off mayastor-repl5

pipeline/ci CI green @ 297599364819

Details

2975993648

Redundancy for metrics lived entirely in the storage layer: one vmsingle
on a repl5 volume. That is 5 storage replicas of a single point of
failure -- if the pod or its data is lost, there is no second VM.

This moves redundancy up into VictoriaMetrics:

- 2 vmsingle replicas, with REQUIRED podAntiAffinity on hostname. Two
  copies on one node is not redundancy, it is two copies that die
  together. 5 workers and 2 replicas, so it is always satisfiable.
- A HEADLESS service, because a StatefulSet only gets stable per-pod DNS
  when its governing service is headless. serviceName is immutable, so
  this needs the StatefulSet recreated -- which the repl1 change needs
  anyway.
- vmagent writes every sample to BOTH pods by name. Each vmsingle holds
  a COMPLETE copy. Do not collapse these back to the ClusterIP service:
  that load-balances, so samples would scatter and neither copy would be
  whole. The ClusterIP service stays, for READS.
- storageClassName repl5 -> repl1. With 2 app-level copies, 5 storage
  replicas duplicate the redundancy, and repl5 cost 200Gi of pool for
  40Gi of data. The bigger reason: a repl1 volume has NOTHING to
  rebuild, so it cannot hit the ENOMEM rebuild defect that currently
  blocks kernel rolls.

KNOWN AND DELIBERATE: victoriametrics-0 keeps its EXISTING repl5 PVC,
because a StatefulSet reuses a PVC whose name already matches. Only
victoriametrics-1 gets a repl1 volume. Moving replica 0 off repl5 means
dropping its PVC and restoring from the S3 backup, which is a separate
step and is safe to do now that vmbackup is verified.

victoriametrics-1 starts EMPTY and fills forward from vmagent. vmsingle
has no peer repair, so it will not backfill history from replica 0.

sean merged commit 2975993648 into trunk

2026-08-10 11:56:54 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/olly!24

No description provided.

Rows
Columns