feat(metrics): say what woke each reconcile #2002
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/wake-cause-metric"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase M of the trigger-based track.
pipelines_operator_wake_total{controller,cause}counts each reconcile as a watch wake or a backstop fire; backstops also log witharmed_for.Reconcile is wrapped: classify on entry, arm from
res.RequeueAfteron exit. That single point covers all 17 pollBackstop sites (13 sit in helpers with noreqin scope) plus the four other timed requeues, with no call-site changes.Two design-doc claims corrected by measurement: operator metrics are already collected (all 7 controllers report to VictoriaMetrics via the existing kubernetes-pods job), and the operator listens on :8080 plain HTTP, not :8443 (refused). M1 is cancelled — the olly branch would have scraped the dead port.
3 mutations proved the tests load-bearing. 127/127 guards, 14/14 packages incl. 110 envtest specs.
Phase M of RELEASE-TRAIN-AND-EVENT-TRIGGERS.md. defaultPollBackstop claims a requeue "only covers a dropped event", and the comment beside it admits nobody could check: "A test cannot tell 'the watch fired' from 'the backstop fired'." Neither could an operator. Every timed requeue was invisible in production, so "the watches carry the load" was a belief with no instrument behind it. pipelines_operator_wake_total{controller,cause} now counts each reconcile as a watch wake or a backstop fire, and a backstop also logs with armed_for. Reconcile is a wrapper that classifies on entry and arms from res.RequeueAfter on exit. Arming where every result already returns through covers all seventeen pollBackstop sites -- thirteen of them in helpers with no req in scope -- plus providerOpenPollInterval, manualApprovalPollInterval, projectedTwinPollInterval and evidenceRetryDelay, with no change at any call site. The bodies are unchanged, renamed to reconcile. Honest about its limits: a Result carries only a duration, so the counter cannot say WHICH timer came due -- armed_for separates them by inspection. A timer firing at the same instant as an event is ambiguous; the 15s tolerance bounds it and resolves that case to backstop, so the net is never flattered. Eight tests drive a fixed clock. Three mutations proved they are load-bearing: treating early arrival as a timer, not consuming the record, and dropping the increment each fail their own test. No new module dependency: the counter is read back through the registry a scrape reads, without naming the gathered types. Also corrects two claims the design document made about this cluster, both measured today. Operator metrics are NOT unscraped: all seven controllers already report controller_runtime_reconcile_total to VictoriaMetrics through the existing kubernetes-pods job. And the operator listens on :8080 plain HTTP (--metrics-secure=false), not :8443 -- 8443 is connection refused. M1 is therefore cancelled: merging the olly branch would have added a job keeping only the dead 8443 endpoint, so every target would have been down while looking configured.