kernel-roll: proven green, runtime probe, pre-stage, infra-only gating, zot-aware ordering #2044
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/kernel-roll-observability"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Rounds 1+2 of the roll speed/observability work, all measured against the 2026-08-14 seanfarm32 roll (85m37s):
Safety envelope extended with negative-control-verified assertions for the stage-only/cordon ordering, the label partition, the forced probe, and zot ordering. 130/130 guards green.
Measured on the 2026-08-14 seanfarm32 roll (85m37s total): 29m canary green, 14m human gate, 26m across five serialized roll-nodes, 13m of inter-node green. PROVE THE RUNTIME (safety). green-gate only ever READ cluster state and exits the instant everything is green — there is no dwell. The failure that matters most is invisible that way: on 2026-08-14 a worker booted seanfarm31, passed its on-node verify, was uncordoned and went green, then ~40 minutes later its containerd stopped creating containers. kubelet still reported Ready and the node kept its existing pods, so every passive signal stayed green while 34 pods sat unable to start. A human noticing was the only thing that caught it. green-gate now takes probeNode (the node just rolled) and PROVES it: passive, every cycle — pods on that node stuck in CreateContainerError or ContainerStatusUnknown are the wedge's fingerprint, one list call active, every 120s — schedule a throwaway pod ON that node and require it to run. This is the test that settled the seanfarm32 canary by hand; a wedged runtime fails it in seconds, an idle healthy node passes. PRE-STAGE (speed). roll-node gains stageOnly, which stops after the image stage and BEFORE the cordon. The fleet now pre-pulls in parallel with the canary roll and its gate — time the roll already spends idle — so each serialized node later starts at the cordon rather than at a fresh pull. NOT DONE, deliberately: rolling two nodes at once. roll-one holds the kernel-deploy-node mutex and tests/kernel-deploy-safety-envelope.sh asserts it, because 5-replica quorum tolerating 2 down leaves NO margin — and today proved a node can brick mid-roll. Two down plus one bricked is a lost quorum. The ~10 minutes are not worth trading that margin away. The safety envelope now asserts the stage-only exit sits ABOVE the cordon, so a fanned-out pre-stage can never disrupt many workers at once. Negative control verified: moving the exit below the cordon fails the test.Roll timings used to be forensics — the 2026-08-14 roll's workflow CRs were TTL'd before anyone asked, and its timeline had to be dug out of Loki. And the awaitingGate/Progress status fields landed write-only: nothing rendered them. Metrics (operator, scrape path already live): pipelines_deploy_awaiting_gate{project,target,gate} 1 while blocked pipelines_deploy_gate_wait_seconds observed when a gate clears pipelines_deploy_progress_completed/_total fleet roll position Gauges are deleted at terminal so a finished roll cannot linger as a stuck-looking series; the gate-wait histogram observes on the prev->cleared edge using the AwaitingSince the operator already holds. Surfaces (Sean's chosen channel is the deploy fold, not an external push): query DeployLeg gains awaitingGate/awaitingSince/progress; the dashboard deploy row shows an amber 'waiting on canary gate 14m' chip and '3/5 · <node> (<phase>)' beside the status dot; pipectl status prints AWAITING <GATE> <N>m with the exact approve command, and the fleet position. Also: a PodDisruptionBudget for zot. The roll handles zot by ordering (its drain bypasses PDBs anyway); this refuses everything ELSE that respects budgets — a manual drain of the registry's host now says no instead of silently taking image pulls down fleet-wide.pipelines_deploy_roll_seconds{project,target,step,node}: each node's time in each roll step (roll-node, green), extracted from the finished run's node graph by argoengine.RollDurations — the same walk RollProgress uses, now with timestamps. Observed exactly once, on the non-terminal->terminal edge, so a re-reconcile of a finished deploy cannot double-count. These are the numbers the 2026-08-14 analysis had to dig out of Loki before log retention ate them; the next roll's shape will be a Grafana panel instead of an archaeology session.