sean/pipelines
Watch
1
0
Fork
You've already forked pipelines
 0
Code Issues Pull requests Projects Releases 758 Packages Wiki Activity

fix(deploy-kernel): tolerate a node that converged before its turn #2066

Merged
pipeline-bot merged 1 commit from fix/roll-node-already-on-target into trunk 2026-08-18 19:00:16 +00:00
Conversation 0 Commits 1 Files changed 1 +16 -2
sean commented 2026-08-18 18:58:21 +00:00
Owner
Copy link

sf38 postmortem: bootc-fetch-apply-updates.timer applied a prestaged deployment mid-roll; the leg then None-crashed kreboot and false-failed a converged fleet. Leg no-ops on already-on-target; kreboot accepts staged-or-booted. Timer mask rides in builder-images.

sf38 postmortem: bootc-fetch-apply-updates.timer applied a prestaged deployment mid-roll; the leg then None-crashed kreboot and false-failed a converged fleet. Leg no-ops on already-on-target; kreboot accepts staged-or-booted. Timer mask rides in builder-images.
fix(deploy-kernel): tolerate a node that converged before its turn
All checks were successful
pipeline/ci CI green @ d45e80a1b361
Details
d45e80a1b3
 
The sf38 workers roll: bootc-fetch-apply-updates.timer (enabled in the
node image) found the deployment PRESTAGE had staged and applied it
with a reboot mid-roll -- a40389 cleanly cycled onto the target 25
minutes after staging, outside any leg. Its own serialized leg then
crashed kreboot on status.staged=null, burned the full 1500 s reboot
ceiling watching for a bootID change that could never come, and
recorded Failed over a fleet that was 5/5 converged and Ready.

Two tolerances: the leg short-circuits as a no-op when the node is
already Ready on the target kernel (before any drain), and kreboot
accepts the digest in staged OR booted, mirroring the kstage fix
(v0.1.662). The timer itself gets masked in node-bootc (builder-images
sibling change) -- the roll pipeline is the only thing that may reboot
a node.
pipeline-bot force-pushed fix/roll-node-already-on-target from d45e80a1b3
All checks were successful
pipeline/ci CI green @ d45e80a1b361
Details
to d216e5e133
All checks were successful
pipeline/ci CI green @ d216e5e13399
Details
2026-08-18 18:59:54 +00:00 Compare
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/pipelines!2066
No description provided.