feat(deploy-kernel): roll the master through the pipeline, as its own target #2053
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/kernel-roll-master"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The master was the one node the kernel roll could not touch — updating it meant a hand-run
bootc switchover SSH, which is not a deployment pipeline. It is now a declared target:Why the existing body already works on the master
The roll never used SSH — every host action is a one-shot privileged agent pod pinned to
spec.nodeNameentering the host viansenter, and the post-reboot wait already tolerates an absent apiserver (everykubectlthere is2>/dev/null || trueinside a poll).roll-node's state machine needed no change.The one thing it needed: the orchestrator must not be the node being rebooted.
plannow computesorchNode, androll-nodecarries it as a template-levelnodeSelector— which replaces the workflow-level pin rather than merging. I measured that with a two-leg probe workflow rather than assuming it.Why the blackout is survivable (checked live)
workflow-controllerreplicas run on workerscilium-operator,hcloud-ccm,local-path-provisioner, each one replica that returns with the nodeIt is a one-node fleet, so the pipeline treats the master as the canary — the mandatory recorded canary decision lands before the only reboot.
Also
verify-deploy-intentderives the expectedtargetReffrom the declared fleet instead of hardcodingseanfarm-workers, so a master-target deploy cannot authorize a worker roll or vice versafinal-verifyacceptsfleet=masterorchNoderegistered as step plumbing in the run-evidence contract guard, with its reasonOrdering is deliberately NOT automated here
The master must roll before the workers whenever the k3s version moves (a kubelet may never be newer than the apiserver). That is a second concern; bundling it is how one bad edit takes down both halves. Two targets, two approvals, and the docs say which goes first.
Verified — both legs, dryRun against the live cluster
129/129 pipeline guards pass.
The master was the one node the kernel roll could not touch. Updating it meant a hand-run `bootc switch` over SSH, which is not a deployment pipeline. It is now a declared target: pipectl deploy linux <bare-version> seanfarm-master pipectl deploy linux <bare-version> seanfarm-workers WHY THE EXISTING BODY ALREADY WORKS ON THE MASTER The roll never used SSH. Every host action is a one-shot privileged agent pod pinned to spec.nodeName that enters the host through nsenter, and the post-reboot wait already tolerates an absent apiserver: every kubectl in that loop is `2>/dev/null || true` inside a poll. So roll-node needed no change to its state machine. The single thing it needed is that the ORCHESTRATOR is not the node being rebooted. plan now computes `orchNode` and roll-node carries it as a template-level nodeSelector. A template-level nodeSelector REPLACES the workflow-level one rather than merging with it — measured, not assumed, with a two-leg probe workflow. A workers roll drives from the master, as before; a master roll drives from a Ready worker. WHY THE BLACKOUT IS SURVIVABLE, CHECKED AGAINST THE LIVE CLUSTER - both workflow-controller replicas run on WORKERS, so the controller keeps reconciling while the apiserver restarts - a running pod does not need the apiserver to stay alive, so the orchestrator survives the gap - the master carries no stateful workload: DaemonSets, plus cilium-operator, hcloud-cloud-controller-manager and local-path-provisioner, each one replica that returns with the node - its drain therefore evicts almost nothing, and the drain timeout stays the expected outcome The master is a one-node fleet, so the pipeline treats it as the canary. The mandatory recorded canary decision lands BEFORE the only reboot, which is the gate you want in front of an API server. Idempotence is unchanged. roll-node has no retryStrategy, so a dead orchestrator pod fails the step instead of silently re-reading BOOT0 after a reboot and rebooting a node that already cycled; the staged-digest check in step 5 remains the second guard. WHAT ELSE MOVED - verify-deploy-intent no longer hardcodes seanfarm-workers. It derives the expected targetRef from the declared fleet, so a master-target PipelineDeploy cannot authorize a worker roll or the reverse. - final-verify accepts fleet=master and names the fleet it checked. - the PLAN summary line says which fleet it planned. - orchNode is registered as step plumbing in the run-evidence contract guard, with the reason. It is consumed by roll-node in the same run and the projector is right not to read it. ORDERING IS DELIBERATELY NOT AUTOMATED HERE. The master must roll before the workers whenever the k3s version moves, because a kubelet may never be newer than the apiserver. That sequencing is a second concern; making one change do both is how a bad edit takes down both halves. The two targets are separate deployments with separate approvals, and the docs say which goes first. VERIFIED, both legs, dryRun against the live cluster: fleet=master plan: orchestrator pods pin to seanfarm-worker-25ce78 enumerated seanfarm-master-b38636 only fleet=workers plan: orchestrator pods pin to seanfarm-master-b38636 enumerated all 5 workers — unchanged 129/129 pipeline guards pass.5a464b66e574e4039923