fix(k3s): fail the add when the new node never goes Ready #156
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/add-node-readiness-fail-closed"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
k3s-add-node.shended its 5-minute readiness poll with a warning, anecho "$NAME", and an implicit exit 0. A node that registered but never went Ready was indistinguishable from a healthy add — same zero status, same node name as the last line of stdout.k3s-roll.shreplaces a worker by adding the new one FIRST and only then draining and deleting the old one. The exit status is its only evidence the replacement came up:That
|| diecould not fire, so the roll advanced to cordon, drain andhcloud server deletethe old worker with nothing healthy replacing it.The file's own header already promised the name "on stdout on success" and told the operator to drain "AFTER the new one is Ready". The code did neither.
Scope, stated honestly
With a healthy API server this was usually survivable:
wait_green's Ready sub-check sees the NotReady node and the roll aborts on timeout. The gate itself was still fail-open, and it is the only place readiness is actually asserted — the downstream timeout is a different-scope backstop, not a handled error path.The fix
It dies. The provisioned server is left in place on purpose so
--resumecan adopt it once the cause is fixed — deleting it would throw away both the evidence and the work.The new guard runs the real Phase 6 block, extracted from the script by its own header comment, against a fake kubectl — behaviour, not the presence of a word — and the extraction fails loudly if the anchor is renamed. It covers the NotReady add, the Ready add (must still exit 0 and still print the name), and a kubectl that fails outright.
Red: 2 failures against the old tail. Green: OK. Suite 50s, shellcheck clean, registered in
tests/check.sh.Found by an adversarially-verified audit for this fail-open shape, prompted by the same defect in the CNPG archive trap (#151/#152/#154).
https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5
k3s-add-node.sh ended its 5-minute readiness poll with a warning, an `echo "$NAME"`, and an implicit exit 0. A node that registered but never went Ready was therefore indistinguishable from a healthy add: same zero status, and the same node name as the last line of stdout. k3s-roll.sh replaces a worker by adding the new one FIRST and only then draining and deleting the old one, and the exit status is its only evidence that the replacement came up: add_out=$("$SCRIPT_DIR/k3s-add-node.sh" "$arch" 2>&1) || die ... That `|| die` could not fire, so the roll advanced to cordon, drain and `hcloud server delete` the old worker with nothing healthy replacing it. The file's own header already promised the name "on stdout on success" and told the operator to drain "AFTER the new one is Ready"; the code did neither. Now it dies. The provisioned server is left in place on purpose so --resume can adopt it once the cause is fixed. The new guard runs the REAL Phase 6 block, extracted from the script by its own header comment, against a fake kubectl — behaviour, not the presence of a word, and the extraction fails loudly if the anchor is renamed. It covers the NotReady add, the Ready add, and a kubectl that fails outright. Red: 2 failures against the old tail. Green: OK. Suite 50s, shellcheck clean. Found by an adversarially-verified audit for this fail-open shape, prompted by the same defect in the CNPG archive trap. Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5