fix(probe): give the availability gate a failure budget #157

Merged

binjovi-bot merged 1 commit from fix/availability-failure-budget into trunk 2026-09-01 16:52:36 +00:00

Conversation 0 Commits 1 Files changed 4 +384 -23

sean commented 2026-09-01 16:04:35 +00:00

Owner

Copy link

Problem

The release availability gate failed on one bad sample. Across the 21 retained
runs in workflows, 11 of 21 failed. Ten were noise:

run           check                          fail/n   rate%  maxconsec
1546c96d8ad4  public-dns                     25/25   100.0   25   <- real outage
1c5d24fc7228  production-dashboard-activity   2/61     3.3    2
445d425a9422  production-dashboard-activity   1/74     1.4    1
62b219bdaeaf  public-dns                      1/79     1.3    1
878eac0d725c  production-dashboard-activity   2/37     5.4    2
a90190469ad6  production-dashboard-activity   3/33     9.1    1
bec7e97bf91a  production-service-health       1/64     1.6    1
ce28978fb382  production-service-health       1/78     1.3    1
d262bdb76128  production-public-ipv4          1/65     1.5    1
staging-a2    public-dns                      2/79     2.5    1
staging-a3    public-dns                      1/80     1.2    1

The separation is clean. Noise reached 2 consecutive failures and a 9.1 percent
rate. The one true outage reached 25 consecutive and 100 percent, and it never
passed preflight (preflight_successes: 0).

A gate that fails half the time stops being read as a signal.

Change

A check fails the gate only when either holds:

• 3 or more consecutive failed samples, or
• 3 or more failures and a failure rate above 10 percent.

Both thresholds sit inside the empty gap above. At the measured cycle time of
2.5 to 4.5 seconds, 3 consecutive failures is 8 to 13 seconds of sustained
outage.

The rule applies to every check. The two flakiest checks
(production-dashboard-activity, production-service-health) are cluster-local
and account for 9 of the 15 flaky results, so a carve-out for in-cluster checks
would keep half the false failures. Their errors are connection refused and
connection reset by peer, which is pod churn during rollout.

Preflight is unchanged. A sustained outage still never enters the measurement
window.

Second defect fixed

The measurement loop read:

for c.now().Before(deadline) || (summary.Failures == 0 && summary.Samples < minimumSamples) {

One blip both failed the gate and stopped the run extending to its sample
floor. Recorded failing runs show 25 to 37 samples against 75 for healthy runs.
Since the controller rejects any run under 60 samples, tolerating a blip without
this fix would have converted a false failure into
:availability_insufficient_samples.

Compatibility

failures keeps its meaning: every failed check result. Added
tolerated_failures, max_consecutive_failures, and per-check check_stats.

This change is inert on its own. availability_gate.ex:131 still rejects a
run with evidence["failures"] != 0, so a tolerated run is still refused. That
gate moves in a follow-up binjovi PR. Ordering is fail-closed: this ships first
and changes nothing, then the controller change activates it.

Tests

Red first. New cases cover a single blip, a streak of two, a streak of three, a
rate above budget, a short run below the absolute floor, and three sequences
recorded from real runs (445d425a9422, 1c5d24fc7228, 1546c96d8ad4).

scripts/check-availability-probe.sh passes; VERSION 7 to 8 with the script
pin and new symbol assertions.

https://claude.ai/code/session_016qGissozRbgkyNM5DnTQJ5

## Problem The release availability gate failed on one bad sample. Across the 21 retained runs in `workflows`, **11 of 21 failed**. Ten were noise: ``` run check fail/n rate% maxconsec 1546c96d8ad4 public-dns 25/25 100.0 25 <- real outage 1c5d24fc7228 production-dashboard-activity 2/61 3.3 2 445d425a9422 production-dashboard-activity 1/74 1.4 1 62b219bdaeaf public-dns 1/79 1.3 1 878eac0d725c production-dashboard-activity 2/37 5.4 2 a90190469ad6 production-dashboard-activity 3/33 9.1 1 bec7e97bf91a production-service-health 1/64 1.6 1 ce28978fb382 production-service-health 1/78 1.3 1 d262bdb76128 production-public-ipv4 1/65 1.5 1 staging-a2 public-dns 2/79 2.5 1 staging-a3 public-dns 1/80 1.2 1 ``` The separation is clean. Noise reached 2 consecutive failures and a 9.1 percent rate. The one true outage reached 25 consecutive and 100 percent, and it never passed preflight (`preflight_successes: 0`). A gate that fails half the time stops being read as a signal. ## Change A check fails the gate only when either holds: - 3 or more consecutive failed samples, or - 3 or more failures **and** a failure rate above 10 percent. Both thresholds sit inside the empty gap above. At the measured cycle time of 2.5 to 4.5 seconds, 3 consecutive failures is 8 to 13 seconds of sustained outage. The rule applies to **every** check. The two flakiest checks (`production-dashboard-activity`, `production-service-health`) are cluster-local and account for 9 of the 15 flaky results, so a carve-out for in-cluster checks would keep half the false failures. Their errors are `connection refused` and `connection reset by peer`, which is pod churn during rollout. Preflight is unchanged. A sustained outage still never enters the measurement window. ## Second defect fixed The measurement loop read: ```go for c.now().Before(deadline) || (summary.Failures == 0 && summary.Samples < minimumSamples) { ``` One blip both failed the gate **and** stopped the run extending to its sample floor. Recorded failing runs show 25 to 37 samples against 75 for healthy runs. Since the controller rejects any run under 60 samples, tolerating a blip without this fix would have converted a false failure into `:availability_insufficient_samples`. ## Compatibility `failures` keeps its meaning: every failed check result. Added `tolerated_failures`, `max_consecutive_failures`, and per-check `check_stats`. **This change is inert on its own.** `availability_gate.ex:131` still rejects a run with `evidence["failures"] != 0`, so a tolerated run is still refused. That gate moves in a follow-up binjovi PR. Ordering is fail-closed: this ships first and changes nothing, then the controller change activates it. ## Tests Red first. New cases cover a single blip, a streak of two, a streak of three, a rate above budget, a short run below the absolute floor, and three sequences recorded from real runs (`445d425a9422`, `1c5d24fc7228`, `1546c96d8ad4`). `scripts/check-availability-probe.sh` passes; `VERSION` 7 to 8 with the script pin and new symbol assertions. https://claude.ai/code/session_016qGissozRbgkyNM5DnTQJ5

sean added 1 commit 2026-09-01 16:04:35 +00:00

fix(probe): give the availability gate a failure budget ... 06594006be

The gate failed a release on one bad sample. 11 of 21 recorded runs
failed. Ten of them were 1 to 3 scattered failures out of 33 to 80
samples. One was a true outage.

A check now fails the gate only on 3 or more consecutive failures, or on
a failure rate above 10 percent with at least 3 failures. Recorded noise
reached 2 consecutive failures and a 9.1 percent rate. The one true
outage reached 25 consecutive failures and 100 percent, and it never
passed preflight. The thresholds sit in that gap.

The rule applies to every check. The two flakiest checks are in-cluster,
so a carve-out for cluster-local checks would keep half the false
failures.

A tolerated failure no longer cuts the run short. The measurement loop
kept sampling only while failures were zero, so one blip both failed the
gate and truncated the run below the sample floor the controller needs.

Summary keeps the meaning of failures. It adds tolerated_failures,
max_consecutive_failures, and per-check stats so a verdict is
diagnosable without the per-sample log. The controller still rejects a
run with failures above zero, so this change is inert until that gate
moves as well.

Claude-Session: https://claude.ai/code/session_016qGissozRbgkyNM5DnTQJ5

sean changed target branch from main to trunk 2026-09-01 16:16:29 +00:00

binjovi-bot merged commit 06594006be into trunk 2026-09-01 16:52:36 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/builder-images!157

No description provided.