sean/pipelines
Watch
1
0
Fork
You've already forked pipelines
 0
Code Issues Pull requests Projects Releases 758 Packages Wiki Activity

fix(kernel-deploy): raise greenBudget 900s -> 2400s #2029

Merged
sean merged 1 commit from fix/kernel-green-budget into trunk 2026-08-11 00:12:32 +00:00
Conversation 0 Commits 1 Files changed 1 +20 -1
sean commented 2026-08-10 17:15:12 +00:00
Owner
Copy link

A seanfarm30 fleet roll wedged 2 nodes in. wait_green FAILED after seanfarm-worker-25ce78 with three complaints: two repl5 mayastor volumes still rebuilding, three flux Kustomizations not ready, and a crashing pod. 900s was not long enough - two concurrent repl5 rebuilds measured ~10 min each - so the gate gave up and the roll advanced while the cluster was still recovering.

wait_green is fail-closed and already checks mayastor volumes, flux and crashing pods. It only needed long enough for them to converge.

⚠ CORRECTION to the first version of this PR

It claimed the rebuild caused the next node's 504 Gateway Timeout when staging its bootc image. That was a plausible correlation and it was wrong.

The real cause was MinIO - the S3 backend behind zot - at 99% FULL, returning XMinioStorageFull (507). The same shortage crashlooped Loki (could not flush chunks) and failed CI image pushes, including this PR's own first CI run. Bucket usage at the time: platform-loki-chunks 158 GiB, workflows 71 GiB, zot 40 GiB, against ~250 GiB usable.

Fixed separately: a 3-day expiry lifecycle rule on the workflows bucket plus mc admin config set scanner speed=fastest. MinIO went 99% -> 84% and Loki recovered.

So this change is worth landing on its own merit - the gate should wait for convergence - but it is NOT what unblocked the roll.

A seanfarm30 fleet roll wedged 2 nodes in. `wait_green` FAILED after `seanfarm-worker-25ce78` with three complaints: two repl5 mayastor volumes still rebuilding, three flux Kustomizations not ready, and a crashing pod. **900s was not long enough** - two concurrent repl5 rebuilds measured ~10 min each - so the gate gave up and the roll advanced while the cluster was still recovering. `wait_green` is fail-closed and already checks mayastor volumes, flux and crashing pods. It only needed long enough for them to converge. ### ⚠ CORRECTION to the first version of this PR It claimed the rebuild caused the next node's `504 Gateway Timeout` when staging its bootc image. That was a plausible correlation and it was **wrong**. The real cause was **MinIO - the S3 backend behind zot - at 99% FULL**, returning `XMinioStorageFull` (507). The same shortage crashlooped Loki (could not flush chunks) and failed CI image pushes, including this PR's own first CI run. Bucket usage at the time: `platform-loki-chunks` 158 GiB, `workflows` 71 GiB, `zot` 40 GiB, against ~250 GiB usable. Fixed separately: a 3-day expiry lifecycle rule on the `workflows` bucket plus `mc admin config set scanner speed=fastest`. MinIO went 99% -> 84% and Loki recovered. **So this change is worth landing on its own merit - the gate should wait for convergence - but it is NOT what unblocked the roll.**
The seanfarm30 fleet roll wedged 2 nodes in. The chain, measured:

  25ce78 reboots -> its mayastor replicas fault
  -> TWO repl5 volumes rebuild, one of them pvc-29e43cdd
  -> wait_green gives up at 900s and FAILS
  -> the roll advances to 42afba anyway
  -> 42afba cannot stage: 504 Gateway Timeout from registry.sean.farm
  -> because pvc-29e43cdd IS ZOT'S OWN VOLUME, still rebuilding

So the roll degraded the registry it depends on, then outran the
recovery. Two concurrent repl5 rebuilds measured about 10 minutes each,
which does not fit in 900s.

wait_green is the right place to fix this. It is fail-closed and it
already checks mayastor volumes; it simply needed long enough to let
them converge before the next node stages. 900 -> 2400.

The amplifier is recorded in the comment but deliberately NOT changed
here: zot sits on mayastor-repl5, so every worker reboot faults one of
its replicas. Moving it to repl2 would cut that exposure, but its PVC
holds build/node-bootc, which shipwright kernel-chain/4-node builds
in-cluster and which is NOT synced from codeberg. Recreating that PVC
destroys the image the roll stages. That migration needs its own plan.
sean force-pushed fix/kernel-green-budget from 7d634fb566
Some checks failed
pipeline/ci CI failed @ 7d634fb5669d
Details
to 2641237653
All checks were successful
pipeline/ci CI green @ 2641237653e1
Details
2026-08-11 00:08:43 +00:00 Compare
sean changed title from fix(kernel-deploy): greenBudget 900s is too short, and the roll ate its own registry to fix(kernel-deploy): raise greenBudget 900s -> 2400s 2026-08-11 00:08:57 +00:00
sean merged commit 2641237653 into trunk 2026-08-11 00:12:31 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/pipelines!2029
No description provided.