fix(metrics): grow VM to 40Gi, cut retention to 14d, retain the PVC #21

Merged

sean merged 1 commit from fix/vm-storage-and-retention into trunk

2026-08-10 10:31:48 +00:00

sean commented

2026-08-10 10:29:53 +00:00

Owner

VictoriaMetrics filled its 20Gi disk and went READ-ONLY, so the cluster ingested NO metrics for ~34h (vmagent got 503 the storage is in read-only mode and retried the same blocks 2040 times). Every Grafana panel showed No data.

Nothing stored was lost. Verified by querying past timestamps: 36h/48h/72h/168h ago all return data; 6h/24h ago return none. It is a GAP in ingestion, not a deletion.

Disk was 19.9G of 19.9G with 4.9M free. The 2026_07 partition alone was 12.0G (5.7G data + 6.3G index).

Changes

storage 20Gi -> 40Gi. mayastor-repl5 has allowVolumeExpansion=true and every pool has 138-178 GiB free. Already expanded online: the volume went to 40 GiB and stayed Online at 5 replicas, no rebuild triggered.
retentionPeriod 30d -> 14d. VM drops whole MONTHLY partitions, so retention must be short enough to age out the previous month. At 30d, 2026_07 was pinned until Aug 30 and the disk could never drain.
whenDeleted Delete -> Retain. Not cosmetic. Delete puts an ownerReference with blockOwnerDeletion: true on the PVC - and changing volumeClaimTemplates REQUIRES deleting the StatefulSet, which would have garbage-collected the PVC and every metric in it. Retain makes that edit safe. (PV was already Retain.)

Known and NOT fixed here

The index is larger than the samples (10.1G vs 9.3G). That is a cardinality problem - likely per-pod labels on short-lived Argo workflow pods. Worth a separate look.

**VictoriaMetrics filled its 20Gi disk and went READ-ONLY**, so the cluster ingested NO metrics for ~34h (vmagent got 503 `the storage is in read-only mode` and retried the same blocks 2040 times). Every Grafana panel showed No data. **Nothing stored was lost.** Verified by querying past timestamps: 36h/48h/72h/168h ago all return data; 6h/24h ago return none. It is a GAP in ingestion, not a deletion. Disk was 19.9G of 19.9G with 4.9M free. The `2026_07` partition alone was 12.0G (5.7G data + 6.3G index). ### Changes 1. **storage 20Gi -> 40Gi.** mayastor-repl5 has `allowVolumeExpansion=true` and every pool has 138-178 GiB free. Already expanded online: the volume went to 40 GiB and stayed Online at 5 replicas, **no rebuild triggered**. 2. **retentionPeriod 30d -> 14d.** VM drops whole MONTHLY partitions, so retention must be short enough to age out the previous month. At 30d, `2026_07` was pinned until Aug 30 and the disk could never drain. 3. **whenDeleted Delete -> Retain.** Not cosmetic. `Delete` puts an ownerReference with `blockOwnerDeletion: true` on the PVC - and changing `volumeClaimTemplates` REQUIRES deleting the StatefulSet, which would have garbage-collected the PVC and every metric in it. `Retain` makes that edit safe. (PV was already Retain.) ### Known and NOT fixed here The index is larger than the samples (10.1G vs 9.3G). That is a cardinality problem - likely per-pod labels on short-lived Argo workflow pods. Worth a separate look.

sean added 1 commit

2026-08-10 10:29:53 +00:00

fix(metrics): grow VM to 40Gi, cut retention to 14d, retain the PVC

pipeline/ci CI green @ fa15c3473385

Details

fa15c34733

VictoriaMetrics filled its 20Gi disk and put storage into READ-ONLY mode.
vmagent got 503 "the storage is in read-only mode" and retried the same
blocks 2040 times, about 34 hours. So the cluster ingested NO metrics
from 2026-08-09 02:00 UTC until this change.

Nothing stored was lost. Data at 36h, 48h, 72h and 168h ago is intact,
and data at 6h and 24h ago is absent. It is a GAP in ingestion, not a
deletion.

Disk at the time: 19.9G of 19.9G, 4.9M free. The 2026_07 partition alone
was 12.0G (5.7G data + 6.3G index).

Three changes:

1. storage 20Gi -> 40Gi. mayastor-repl5 allows expansion and every pool
   has 138-178 GiB free, so this is affordable. The volume expanded
   online to 40 GiB and stayed Online at 5 replicas, with no rebuild.

2. retentionPeriod 30d -> 14d. VM drops whole MONTHLY partitions, so
   retention must be short enough to age out the previous month. At
   30d, 2026_07 was held until Aug 30 and the disk could not drain.

3. whenDeleted Delete -> Retain. This one is not cosmetic. Delete puts
   an ownerReference with blockOwnerDeletion on the PVC, so deleting
   this StatefulSet -- which is REQUIRED, because volumeClaimTemplates
   is immutable -- would garbage collect the PVC and every metric in
   it. Retain makes that edit safe.

Note the index is larger than the data (10.1G index vs 9.3G samples).
That is a cardinality problem and is NOT addressed here.

sean merged commit fa15c34733 into trunk

2026-08-10 10:31:48 +00:00

sean referenced this pull request

2026-08-10 11:55:07 +00:00

feat(metrics): move VM redundancy into the app, off mayastor-repl5 #24