perf(kernel): shave ~45s per node off the roll, and stop the repartition honouring PDBs it cannot satisfy #2162

Merged

pipeline-bot merged 2 commits from feat/faster-rolls into trunk 2026-09-04 15:30:47 +00:00

Conversation 0 Commits 2 Files changed 7 +135 -35

sean commented 2026-09-04 14:30:09 +00:00

Owner

Copy link

Measured on the seanfarm47 master roll: roll-node took 3m00s. Two of its waits are fixed costs that buy nothing.

pipeline-deploy-kernel

change	was	now	why
drainTimeout	60	30	The template already says the timeout is expected — node-local stateful pods reboot with the node, and drain is --disable-eviction so no PDB holds it; CNPG's 1800s grace period does. The wait buys a clean SIGTERM window for Postgres, whose fast shutdown is seconds. Every node paid the full 60.
systemd-run --on-active	15	5	Exists so the agent pod can report before the node goes. 5s is ample; the pod runs soft=1 anyway.
Node-object watch poll	10s	5s	Up to 10s of slack on a ~2-minute event.

pipeline-repartition-node

Gets --disable-eviction, which it should have carried from the start — the kernel roll has. Every volume on the node is about to be destroyed, so its stateful pods cannot be preserved by a PDB; honouring one only stalls. Today seanfarm-worker-a40389 sat its full 10-minute drain timeout on postgres-9 and zot-v2-0:

error when evicting pods/"postgres-9" -n "cnpg": global timeout reached: 10m0s
error when evicting pods/"zot-v2-0" -n "registry": global timeout reached: 10m0s

with cnpg/postgres and registry/zot both at disruptionsAllowed=0 — then proceeded regardless. drainTimeout drops 600 → 60 with it, and the reboot delay 15 → 5 to match.

Expected

roll-node ~3m00s → ~2m15s now, ~1m30s once sean/builder-images lands the boot-time change (cloud-init and fwupd masked, loglevel=4), which cuts the node's ~120s down-time roughly in half.

tests/check.sh — 147/147.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Measured on the seanfarm47 master roll: `roll-node` took **3m00s**. Two of its waits are fixed costs that buy nothing. ## pipeline-deploy-kernel | change | was | now | why | | --- | --- | --- | --- | | `drainTimeout` | 60 | **30** | The template already says the timeout is *expected* — node-local stateful pods reboot with the node, and drain is `--disable-eviction` so no PDB holds it; CNPG's 1800s grace period does. The wait buys a clean SIGTERM window for Postgres, whose fast shutdown is seconds. Every node paid the full 60. | | `systemd-run --on-active` | 15 | **5** | Exists so the agent pod can report before the node goes. 5s is ample; the pod runs `soft=1` anyway. | | Node-object watch poll | 10s | **5s** | Up to 10s of slack on a ~2-minute event. | ## pipeline-repartition-node Gets **`--disable-eviction`**, which it should have carried from the start — the kernel roll has. Every volume on the node is about to be destroyed, so its stateful pods *cannot* be preserved by a PDB; honouring one only stalls. Today `seanfarm-worker-a40389` sat its **full 10-minute drain timeout** on `postgres-9` and `zot-v2-0`: ``` error when evicting pods/"postgres-9" -n "cnpg": global timeout reached: 10m0s error when evicting pods/"zot-v2-0" -n "registry": global timeout reached: 10m0s ``` with `cnpg/postgres` and `registry/zot` both at `disruptionsAllowed=0` — then proceeded regardless. `drainTimeout` drops 600 → 60 with it, and the reboot delay 15 → 5 to match. ## Expected `roll-node` ~3m00s → **~2m15s** now, **~1m30s** once `sean/builder-images` lands the boot-time change (cloud-init and fwupd masked, `loglevel=4`), which cuts the node's ~120s down-time roughly in half. `tests/check.sh` — 147/147. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit 2026-09-04 14:30:10 +00:00

perf(kernel): shave ~45s per node off the roll, and stop the repartition honouring PDBs it cannot satisfy ... 35338c1c55

Measured on the seanfarm47 master roll: roll-node took 3m00s. Two of its
waits are fixed costs that buy nothing:

drainTimeout 60 -> 30. The template already documents the timeout as
EXPECTED: node-local stateful pods cannot move and reboot with the node,
and drain runs --disable-eviction so no PDB holds it — CNPG's 1800s
terminationGracePeriod does. What the wait buys is a clean SIGTERM window
for Postgres before the reboot, and its fast shutdown completes in seconds.
Every node paid the full 60 on every roll.

systemd-run --on-active 15 -> 5. The delay exists so the agent pod can
report before the node goes; 5s is ample and the pod runs soft=1 anyway.

Node-object watch poll 10 -> 5s on a ~2-minute event: up to 10s of slack
removed per node.

pipeline-repartition-node gets --disable-eviction, which it should have
carried from the start — pipeline-deploy-kernel has. Every volume on the
node is about to be destroyed, so its stateful pods cannot be preserved by
a PDB; honouring one only stalls. On 2026-09-04 seanfarm-worker-a40389 sat
its full 10-minute drain timeout on postgres-9 and zot-v2-0 with
cnpg/postgres and registry/zot both at disruptionsAllowed=0, then proceeded
regardless. drainTimeout drops 600 -> 60 with it, and its reboot delay
15 -> 5 to match.

Expected on the kernel roll: roll-node ~3m00s -> ~2m15s before the image
change in sean/builder-images lands, ~1m30s after.

tests/check.sh: 147/147.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit 2026-09-04 15:21:12 +00:00

perf(kernel-roll): bound every polling kubectl, and stop serialising the preflight ... cd2aca6b2c

Two changes, both aimed at the wall clock and at the wedge class that eats it.

BOUND THE POLLING CALLS. roll-node carries a comment saying "EVERY kubectl here
carries --request-timeout", and explains why: the wait loops bound their
ITERATIONS, not their calls, so one kubectl blocked on a dead apiserver parks the
step forever and the budget never applies. It cites 2026-08-11, when that wedged
three times in one day (kpre 76m, final-verify 3.2h, kpre again 130m) for roughly
six hours lost to a 180s budget.

The claim was false for 24 calls. The gap included the entire post-reboot watch
loop, its confirm re-read, and cordon/drain/uncordon -- and on the master leg the
apiserver being rebooted IS the endpoint those calls dial, so every iteration
during the blackout was three unbounded requests against a dead socket. green-gate
and final-verify had the same hole. All 24 now carry the flag.

tests/kernel-roll-kubectl-timeouts.sh makes the comment enforceable instead of
aspirational, since it had already rotted once. It is counted, so an extraction
that silently returns an empty body fails rather than reporting three clean
templates. Verified by mutation: stripping one flag turns it red.

There is no collision with tests/poller-hang-timeouts.sh, which BANS this flag for
the dashboard pollers because client-go also appends it as a URL parameter and
that 404s `get --raw`. Nothing in these templates uses --raw; the guard says so.

STOP SERIALISING THE PREFLIGHT. verify-deploy-intent, verify-source-backup and
verify-signature each depend only on `resolve` and on nothing from each other, and
preflight reads live cluster state and depends on nothing at all. They were four
orchestrator pod lifecycles back to back, each paying its own schedule and
kubeconfig prologue, before the roll could even plan. They are now one parallel
group. All four still have to pass before `plan`, so the gate is unchanged; only
the wall clock moves. verify-source-backup was the long pole (three Codeberg round
trips plus up to 40s of mirror-lag sleep) and now overlaps the other three.

Three guards anchored on the literal `kubectl cordon "$NODE"` to locate a position
for an ordering assertion, so adding a flag read as "the cordon disappeared". They
are now flag-tolerant: leg-tolerances normalises the client-side flag away before
matching, the safety envelope greps a pattern, and the M5 mutation accepts either
spelling. None of them assert anything different than before. 148/148 guards pass,
and the regressions contract still catches all of its mutations.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

pipeline-bot merged commit cd2aca6b2c into trunk 2026-09-04 15:30:46 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/pipelines!2162

No description provided.