fix(alerting): correct the inert starter set and add an OOM rule #147

Merged

binjovi-bot merged 2 commits from feat/alert-rules into trunk

2026-09-01 20:01:33 +00:00

sean commented

2026-09-01 16:05:14 +00:00

Owner

Investigating what it would take to alert on last night's incidents turned up two errors in this directory that would have broken flip-on, plus one rule worth writing now.

CORRECTIONS (both verified against the live cluster):

The observability namespace is olly. Every PrometheusRule object here named olly-platform, which does not exist — 12 references across the README and rules.yaml.
vmalert does NOT read PrometheusRule objects. It reads rule FILES via -rule. The component that watches those custom resources is the VictoriaMetrics operator, and it is not installed. The flip-on procedure said otherwise, so following it would not have loaded a single rule. The README now states the two real options: install the VM operator and convert to VMRule, or render these groups into a ConfigMap and mount them.

Credit where due: the rest of the README was already honest. Its "what is MISSING" table correctly said no evaluator, no Alertmanager, and no receiver exist. That is still true and this change does not alter it.

NEW RULE: a workload-health group alerting on containers killed for memory use. Three Binjovi replicas OOM-crash-looped for over an hour tonight with nothing observing it. kube_pod_container_status_last_terminated_reason is present in the store and the expression was replayed across the incident window. The restart term keeps the alert quiet after recovery, since the termination reason otherwise stays set for the life of the container.

Deliberately NOT a CrashLoopBackOff rule: replaying kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} over the same window returns ZERO series, because the 30 second scrape never observed the waiting state. That rule would have read as coverage and caught nothing.

TWO SIGNALS DELIBERATELY LEFT UNWRITTEN, and the README says why:

OpenBao credential-sync staleness. This is the one that actually mattered: the operator stopped reconciling for four days, a rotation landed in PostgreSQL without reaching its Secret, and the workload crash-looped on authentication. But no metric exposes status.lastRenewalTime — it needs a kube-state-metrics CustomResourceStateMetrics entry and read RBAC first. A rule now would assert against a series nobody scrapes.
Repeated availability-gate refusals. Binjovi started logging these tonight, so the signal exists in Loki, but Loki's ruler has an empty alertmanager_url and no rule groups.

The set remains inert and no contact point is configured; that decision is still open and is recorded as open rather than quietly defaulted.

https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

Investigating what it would take to alert on last night's incidents turned up two errors in this directory that would have broken flip-on, plus one rule worth writing now. CORRECTIONS (both verified against the live cluster): 1. The observability namespace is `olly`. Every PrometheusRule object here named `olly-platform`, which does not exist — 12 references across the README and rules.yaml. 2. vmalert does NOT read PrometheusRule objects. It reads rule FILES via -rule. The component that watches those custom resources is the VictoriaMetrics operator, and it is not installed. The flip-on procedure said otherwise, so following it would not have loaded a single rule. The README now states the two real options: install the VM operator and convert to VMRule, or render these groups into a ConfigMap and mount them. Credit where due: the rest of the README was already honest. Its "what is MISSING" table correctly said no evaluator, no Alertmanager, and no receiver exist. That is still true and this change does not alter it. NEW RULE: a workload-health group alerting on containers killed for memory use. Three Binjovi replicas OOM-crash-looped for over an hour tonight with nothing observing it. kube_pod_container_status_last_terminated_reason is present in the store and the expression was replayed across the incident window. The restart term keeps the alert quiet after recovery, since the termination reason otherwise stays set for the life of the container. Deliberately NOT a CrashLoopBackOff rule: replaying kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} over the same window returns ZERO series, because the 30 second scrape never observed the waiting state. That rule would have read as coverage and caught nothing. TWO SIGNALS DELIBERATELY LEFT UNWRITTEN, and the README says why: - OpenBao credential-sync staleness. This is the one that actually mattered: the operator stopped reconciling for four days, a rotation landed in PostgreSQL without reaching its Secret, and the workload crash-looped on authentication. But no metric exposes status.lastRenewalTime — it needs a kube-state-metrics CustomResourceStateMetrics entry and read RBAC first. A rule now would assert against a series nobody scrapes. - Repeated availability-gate refusals. Binjovi started logging these tonight, so the signal exists in Loki, but Loki's ruler has an empty alertmanager_url and no rule groups. The set remains inert and no contact point is configured; that decision is still open and is recorded as open rather than quietly defaulted. https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean added 1 commit

2026-09-01 16:05:15 +00:00

fix(alerting): correct the inert starter set and add an OOM rule

binjovi/ci Binjovi completed the frozen plan

Details

f328fd8372

Two errors would have broken flip-on. The observability namespace is
olly, not olly-platform, so every rule object named a namespace that
does not exist. And vmalert reads rule files, not PrometheusRule
objects: the component that watches those custom resources is the
VictoriaMetrics operator, which is not installed.

A new workload-health group alerts on a container killed for memory
use. A production incident ran for an hour without observation, and
this expression was replayed against that window. The group
deliberately does not use CrashLoopBackOff: the same replay shows the
scrape interval never observed the waiting state, so that rule would
have looked like coverage and caught nothing.

The set stays inert. The README now also records which incident
signals still have no metric, and why a rule must not be written for
them yet.

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean force-pushed feat/alert-rules from f328fd8372

binjovi/ci Binjovi completed the frozen plan

Details

to fb74d162d0

binjovi/ci Binjovi failed the frozen plan

Details

2026-09-01 18:08:59 +00:00

Compare

sean force-pushed feat/alert-rules from fb74d162d0

binjovi/ci Binjovi failed the frozen plan

Details

to c893811974