feat(kernel): canary auto-rollback — a bad kernel restores itself #2079
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/canary-rollback"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Kaizen run 4, improvement A: failed canary → bootc rollback + reboot when the bad kernel booted, uncordon-only when it never did; every path exits 1 so the failure is never masked. Guard P9 (per-path exit-1) + contract M13/M14; 13/13.
A failed canary used to leave the node on the bad kernel, cordoned, for a human to rescue. Now the canary step carries continueOn.failed and a canary-rollback step runs on '{{steps.canary.status}}' == Failed: - booted == target (the bad kernel is LIVE): bootc rollback + scheduled reboot, wait for bootID change + Ready + kernel != target, uncordon. A bad kernel now costs ~10 minutes and zero manual work. - booted != target (kstage/drain-class failure — the bad kernel never booted): deployments untouched (a staged image is inert; the auto-update timer is masked in the node image), uncordon if Ready. Rollback is damage control, never a pass: every path exits 1, so the run still reads Failed everywhere and the fleet stays untouched. Guard P9 requires exit 1 after EACH verdict path; contract M13 (rollback removed) and M14 (a path masks the failure) prove it bites — 13/13.