fix(operator): never freeze an integration car without its Change CR #2074

Merged
pipeline-bot merged 1 commit from fix/train-change-cr-race into trunk 2026-08-19 12:38:13 +00:00
Owner

Kaizen improvement 4 (found live: sf42 poisoned by a 5-min sensor lag after the sf41 fleet roll). Queue freeze waits up to 5m for the Change CR, then Skips loudly (new missing-change-cr enum). Also catches typoed change numbers.

Kaizen improvement 4 (found live: sf42 poisoned by a 5-min sensor lag after the sf41 fleet roll). Queue freeze waits up to 5m for the Change CR, then Skips loudly (new missing-change-cr enum). Also catches typoed change numbers.
fix(operator): never freeze an integration car without its Change CR
All checks were successful
pipeline/ci CI green @ d6dcb7bb43d7
d6dcb7bb43
The change-events sensor can lag a declared intent by minutes when its
pod is rescheduled — a kernel fleet roll does exactly that. On
2026-08-19 the sf42 release froze its car ~5 minutes before
linux-change-forgejo-37 existed: the car carried an EMPTY ChangeRef,
the rebase workflow got no change-cr label, and the WorkflowProjector
refused its successful result forever ("omitted valid changeNumber").
A transient sensor lag poisoned the whole train; recovery took a
delete-and-redeclare.

The queue freeze now refuses to freeze while a wanted change's CR is
absent: within a 5-minute grace it requeues (sensor lag resolves in
seconds); past it the release goes Skipped with the new
missing-change-cr reason naming the numbers — which also turns a
typoed --change number into a 5-minute loud failure instead of a
poisoned train. SkipReason enum extended in the API type and CRD base.
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/pipelines!2074
No description provided.