docs: design the node replace pipeline #228

Merged

binjovi-bot merged 1 commit from docs/node-replace-pipeline into trunk

2026-09-03 19:40:59 +00:00

sean commented

2026-09-03 19:38:00 +00:00

Owner

Records the design for moving the replace-roll off the laptop and into Pipelines. Nothing is built; the document is marked PROPOSED, NOT BUILT.

Why it is needed. Part 3 of the Mayastor exit wants an unfragmented ZFS pool. A bootc switch keeps the disk, so a kernel roll cannot repartition. And mayastor is sda4, between root and tank, so deleting it leaves a hole that nothing can grow into, because nothing follows tank.

The constraint that shaped it. There is no Hetzner headroom for a sixth server, so the laptop script strategy of add-then-remove at N+1 is unavailable. Running at N-1 is safe only after Mayastor is gone: 54.4 cores are requested today against 48 on four workers, and the io-engine holds 20 of them. After removal it is 34.4 against 48.

The recommendation. Repartition in place rather than replace the server. Both destroy tank, but the in-place path needs no spare server, no join token, no SSH, no cloud token, and no right to delete a node. It runs from the privileged agent pod the kernel roll already uses. The replace path is kept for a bricked node or a server-type change.

The hazards found while reading the laptop scripts are recorded because they were expensive to find: the seanfarm firewall blocks SSH from inside the cluster, there is no cloud-init anywhere in the add path, and the k3s join token has no in-cluster source.

Also amends KERNEL-DEPLOY-PIPELINE.md and KERNEL-ROLL.md, which both state the replace-roll stays laptop-manual, and adds the document to the README index.

bash tests/check.sh passes.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Records the design for moving the replace-roll off the laptop and into Pipelines. **Nothing is built**; the document is marked PROPOSED, NOT BUILT. **Why it is needed.** Part 3 of the Mayastor exit wants an unfragmented ZFS pool. A `bootc switch` keeps the disk, so a kernel roll cannot repartition. And mayastor is `sda4`, between root and `tank`, so deleting it leaves a hole that nothing can grow into, because nothing follows `tank`. **The constraint that shaped it.** There is no Hetzner headroom for a sixth server, so the laptop script strategy of add-then-remove at N+1 is unavailable. Running at N-1 is safe only after Mayastor is gone: 54.4 cores are requested today against 48 on four workers, and the io-engine holds 20 of them. After removal it is 34.4 against 48. **The recommendation.** Repartition in place rather than replace the server. Both destroy `tank`, but the in-place path needs no spare server, no join token, no SSH, no cloud token, and no right to delete a node. It runs from the privileged agent pod the kernel roll already uses. The replace path is kept for a bricked node or a server-type change. The hazards found while reading the laptop scripts are recorded because they were expensive to find: the `seanfarm` firewall blocks SSH from inside the cluster, there is no cloud-init anywhere in the add path, and the k3s join token has no in-cluster source. Also amends `KERNEL-DEPLOY-PIPELINE.md` and `KERNEL-ROLL.md`, which both state the replace-roll stays laptop-manual, and adds the document to the README index. `bash tests/check.sh` passes. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit

2026-09-03 19:38:01 +00:00

docs: design the node replace pipeline

binjovi/ci Binjovi completed the frozen plan

Details

642dcdcf08

Part 3 of the Mayastor exit needs an unfragmented ZFS pool. A bootc switch
keeps the disk, so it cannot repartition. Mayastor sits at sda4, between
root and tank, so its space cannot be given to tank by growing a
partition. Nothing follows tank.

The Hetzner account has no headroom for a sixth server, so the laptop
script's add-then-remove at N+1 is not available. We must spend
redundancy and run at N-1. That is safe only after Mayastor is removed:
54.4 cores are requested today against 48 on four workers, and the
io-engine holds 20 of them. After removal it is 34.4 against 48.

The document therefore recommends repartitioning in place over replacing
the server. Both destroy tank, but in place needs no spare server, no
join token, no SSH, no cloud token and no right to delete a node. It runs
from the privileged agent pod the kernel roll already uses. The replace
path is kept for a bricked node or a change of server type, and its
hazards are recorded because they were expensive to find: the firewall
blocks SSH from inside the cluster, there is no cloud-init, and the k3s
join token has no in-cluster source.

Nothing is built. This records the design and the reasoning, and amends
the two kernel documents that state the replace-roll stays laptop-manual.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

binjovi-bot merged commit 642dcdcf08 into trunk

2026-09-03 19:40:59 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!228

No description provided.

Rows
Columns