fix(kernel-deploy): stop run_agent hanging for hours on a blocked kubectl #2031
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/kernel-agent-hang"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
roll-noderan for 2h10m on an agent whose budget is 180 seconds, with the agent pod alreadyCompleted130 minutes earlier. This happened three times on 2026-08-11 - kpre 76m, final-verify 3.2h, kpre again 130m - costing roughly six hours and, once, letting the canary gate expire underneath the roll and fail the deploy.Root cause
The deadline gates loop iterations, not the child process.
kubectlhad no--request-timeout, so one blocked API call parks the step forever and the budget never applies. Confirmed on the live wedged pod:pid=289 comm=kubectl state=S, sleeping since the agent finished.Fix, two layers
run_agentnow carries--request-timeout=20s, so a blocked call returns and the loop re-evaluates its own deadline. This is the actual fix.roll-nodeandfinal-verifyget a step-levelactiveDeadlineSeconds(2700s / 900s). The in-script timeouts should make this unreachable; it exists so an unforeseen hang fails in minutes rather than hours.Also: a timed-out wait now says so on stderr. Previously it fell through silently to the log dump and reported
phase=?, which is why this read as "mysteriously stuck" three separate times instead of "timed out".Deliberately not a blanket kubectl wrapper:
kubectl draindrives internal watches and a global--request-timeoutcould break it. The timeouts are scoped to the polling calls that actually hang.roll-node ran for 2h10m on an agent whose budget is 180 SECONDS, with the agent pod already Completed 130 minutes earlier. It happened three times on 2026-08-11 -- kpre 76m, final-verify 3.2h, kpre again 130m -- costing roughly six hours and, on one occasion, letting the canary gate expire underneath the roll and fail the whole deploy. ROOT CAUSE. The wait loop is deadline=$(( SECONDS + $3 )) while [ "$SECONDS" -lt "$deadline" ]; do phase="$(kubectl -n "$NS" get pod ...)" sleep 4 done The deadline gates loop ITERATIONS, not the child process. kubectl had no --request-timeout, so one blocked API call parks the step forever and the budget never applies. Confirmed on the live wedged pod: `pid=289 comm=kubectl state=S`, sleeping since the agent finished. FIX, two layers: 1. Every kubectl in run_agent now carries --request-timeout=20s, so a blocked call returns and the loop re-evaluates its own deadline. This is the actual fix. 2. roll-node and final-verify get a step-level activeDeadlineSeconds (2700s / 900s). The in-script timeouts should make this unreachable; it exists so an unforeseen hang fails in minutes instead of hours. Also: a timed-out wait now says so on stderr. Previously the loop fell through silently to the log dump and reported `phase=?`, which is why this read as "mysteriously stuck" rather than "timed out" three separate times. Deliberately NOT a blanket kubectl wrapper: `kubectl drain` drives internal watches and a global --request-timeout could break it. The timeouts are scoped to the polling calls that actually hang.