sean/pipelines
Watch
1
0
Fork
You've already forked pipelines
 0
Code Issues Pull requests Projects Releases 758 Packages Wiki Activity

feat(repartition): give the whole disk tail to ZFS, one worker at a time #2159

Merged
pipeline-bot merged 1 commit from feat/pipeline-repartition-node into trunk 2026-09-04 11:04:05 +00:00
Conversation 0 Commits 1 Files changed 7 +794
sean commented 2026-09-04 10:59:20 +00:00
Owner
Copy link

Mayastor was removed on 2026-09-04. Its 288.8G partition is still carved on every 457.8G worker and now holds nothing, while tank runs on 143G at 57-78% full and 55-66% fragmented — one node wouldn't accept a new 1Gi volume at all. A bootc switch roll can't fix this: it reboots with the disk intact and cannot rewrite a partition table.

Requires sean/seanfarm #250 to land and converge first. kernel-deploy-guard is failurePolicy: Fail, so an unlisted creator is denied at the preflight agent — before anything is cordoned.

Why the work runs at early boot

Forced, not stylistic:

  • k3s-agent.service is baked KillMode=process with no k3s-killall.sh, so stopping it leaves containerd and every shim alive holding ZFS mounts.
  • The agent pod's own rootfs is a snapshot under tank/containerd, so a live zpool destroy tank would unmount the filesystem its payload executes from.

So the agent arms a one-shot systemd unit ordered before the ZFS import units, and reboots. With bimage set it also bootc switches in the same visit — one reboot per node, not two.

The containerd trap

A hand-run repartition of seanfarm-worker-2a9194 earlier today succeeded at the disk level and left the node unable to start any container:

unable to prepare extraction snapshot: target snapshot "sha256:..." already exists

containerd's metadata database lives on sda3 and survives the pool it indexes. After zpool destroy it still listed ~200 snapshots with no datasets, so every unpack failed — and the node couldn't run the pod that would have repaired it. Recovery needed out-of-band SSH. The boot script now clears that metadata in the same unit, before k3s-agent ever starts.

Ordering that isn't obvious

Uncordon comes before the reclaim, deliberately inverting k3s-remove-node.sh, which cleans while cordoned because its node is leaving. Here the node is returning and every volume must come back to it: MinIO is maxSkew:1 + DoNotSchedule and all three ZFS classes are WaitForFirstConsumer, so the just-emptied node is the only domain that can take the replacement. Cleaning while cordoned scatters them and leaves two erasure drives or two Postgres instances sharing a host.

The reclaim deletes PVC, PV and ZFSVolume. k3s-remove-node.sh deleted only the PVC — which is why 14 orphaned ZFSVolume CRs had accumulated, the finalizer never clearing once the owner node is gone.

The gate

The four Mayastor predicates removed in v0.1.743 were the gate's only truthful storage checks. Pod liveness is fine for a reboot, where data survives, and not for an operation that destroys it. Added: MinIO online/healing drive counts from the unauthenticated cluster metrics, per-ordinal OpenBao /v1/sys/health (200 == initialized and unsealed and active-or-standby), and a no-PVC-Pending predicate that catches an incomplete reclaim. Fail-closed throughout. greenBudget defaults 5400, not 2400 — a CNPG re-clone alone measured 57m35s for three instances.

Safety

dryRun defaults true, inverted from the kernel roll. Shares the kernel-deploy mutex, holds kernel-deploy-node across the whole node including its green gate, and pins orchestrator pods to the control plane so the workflow can never drain the node it runs on.

The dedicated SA can delete a dead PV/PVC/ZFSVolume and cannot create or patch one; nodes: patch, never delete.

Verified by mutation

tests/repartition-safety-envelope.sh pins the orderings by line number. Each of these turns it red:

Mutation Caught
delete the containerd clear the boot script does not clear containerd metadata
move labelclear after sgdisk -d labelclear must precede sgdisk -d (labelclear=285 delete=283)
arm the intent before enabling the unit arm the intent file AFTER enabling the unit
grant nodes: delete a repartition must never delete a node

bash tests/check.sh — 147/147 guards pass.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Mayastor was removed on 2026-09-04. Its **288.8G** partition is still carved on every 457.8G worker and now holds nothing, while `tank` runs on **143G at 57-78% full and 55-66% fragmented** — one node wouldn't accept a new 1Gi volume at all. A `bootc switch` roll can't fix this: it reboots with the disk intact and cannot rewrite a partition table. **Requires `sean/seanfarm` #250 to land and converge first.** `kernel-deploy-guard` is `failurePolicy: Fail`, so an unlisted creator is denied at the *preflight* agent — before anything is cordoned. ## Why the work runs at early boot Forced, not stylistic: - `k3s-agent.service` is baked `KillMode=process` with no `k3s-killall.sh`, so stopping it leaves containerd and every shim alive holding ZFS mounts. - The agent pod's own rootfs is a snapshot under `tank/containerd`, so a live `zpool destroy tank` would unmount the filesystem its payload executes from. So the agent **arms** a one-shot systemd unit ordered before the ZFS import units, and reboots. With `bimage` set it also `bootc switch`es in the same visit — **one reboot per node, not two.** ## The containerd trap A hand-run repartition of `seanfarm-worker-2a9194` earlier today succeeded at the disk level and left the node unable to start **any** container: ``` unable to prepare extraction snapshot: target snapshot "sha256:..." already exists ``` containerd's metadata database lives on `sda3` and **survives the pool it indexes**. After `zpool destroy` it still listed ~200 snapshots with no datasets, so every unpack failed — and the node couldn't run the pod that would have repaired it. Recovery needed out-of-band SSH. The boot script now clears that metadata in the same unit, before `k3s-agent` ever starts. ## Ordering that isn't obvious **Uncordon comes before the reclaim**, deliberately inverting `k3s-remove-node.sh`, which cleans while cordoned because its node is *leaving*. Here the node is *returning* and every volume must come back to it: MinIO is `maxSkew:1` + `DoNotSchedule` and all three ZFS classes are `WaitForFirstConsumer`, so the just-emptied node is the only domain that can take the replacement. Cleaning while cordoned scatters them and leaves two erasure drives or two Postgres instances sharing a host. The reclaim deletes **PVC, PV and ZFSVolume**. `k3s-remove-node.sh` deleted only the PVC — which is why 14 orphaned ZFSVolume CRs had accumulated, the finalizer never clearing once the owner node is gone. ## The gate The four Mayastor predicates removed in v0.1.743 were the gate's only *truthful* storage checks. Pod liveness is fine for a reboot, where data survives, and **not** for an operation that destroys it. Added: MinIO online/healing drive counts from the unauthenticated cluster metrics, per-ordinal OpenBao `/v1/sys/health` (200 == initialized **and** unsealed **and** active-or-standby), and a no-PVC-Pending predicate that catches an incomplete reclaim. Fail-closed throughout. `greenBudget` defaults **5400**, not 2400 — a CNPG re-clone alone measured 57m35s for three instances. ## Safety `dryRun` defaults **true**, inverted from the kernel roll. Shares the `kernel-deploy` mutex, holds `kernel-deploy-node` across the whole node including its green gate, and pins orchestrator pods to the control plane so the workflow can never drain the node it runs on. The dedicated SA can **delete** a dead PV/PVC/ZFSVolume and cannot create or patch one; `nodes: patch`, never `delete`. ## Verified by mutation `tests/repartition-safety-envelope.sh` pins the orderings by line number. Each of these turns it red: | Mutation | Caught | | --- | --- | | delete the containerd clear | `the boot script does not clear containerd metadata` | | move `labelclear` after `sgdisk -d` | `labelclear must precede sgdisk -d (labelclear=285 delete=283)` | | arm the intent before enabling the unit | `arm the intent file AFTER enabling the unit` | | grant `nodes: delete` | `a repartition must never delete a node` | `bash tests/check.sh` — **147/147 guards pass.** https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
feat(repartition): give the whole disk tail to ZFS, one worker at a time
All checks were successful
pipeline/ci CI green @ ae67e4a10ae3
Details
ae67e4a10a
 
Mayastor was removed on 2026-09-04. Its 288.8G partition is still carved on
every 457.8G worker and now holds nothing, while tank runs on 143G at 57-78%
full and 55-66% fragmented. One node would not accept a new 1Gi volume at
all. A bootc switch roll cannot fix this: it reboots with the disk intact
and cannot rewrite a partition table.

WHY THE WORK RUNS AT EARLY BOOT

Forced, not stylistic. k3s-agent.service is baked KillMode=process with no
k3s-killall.sh in the image, so stopping it leaves containerd and every shim
alive holding zfs mounts. And the agent pod's own rootfs is a snapshot under
tank/containerd, so a live `zpool destroy tank` would unmount the filesystem
its payload executes from. The agent therefore ARMS a one-shot systemd unit
ordered before the zfs import units, and reboots. If bimage is set it also
bootc switches in the same visit, so the node returns on the new image AND
repartitioned — one reboot, not two.

THE CONTAINERD TRAP

A hand-run repartition of seanfarm-worker-2a9194 on 2026-09-04 succeeded at
the disk level and left the node unable to start ANY container:

  unable to prepare extraction snapshot: target snapshot "sha256:..." already exists

containerd's metadata database lives on sda3 and SURVIVES the pool it
indexes. After zpool destroy it still listed ~200 snapshots with no
datasets, so every unpack failed — and the node could not run the pod that
would have repaired it. Recovery needed out-of-band SSH. The boot script now
clears that metadata in the same unit, before k3s-agent ever starts.

ORDERING THAT IS NOT OBVIOUS

Uncordon comes BEFORE the volume reclaim, deliberately inverting
k3s-remove-node.sh, which cleans while cordoned because its node is LEAVING.
Here the node is RETURNING and every volume must come back to it: MinIO is
maxSkew:1 + DoNotSchedule and all three ZFS classes are
WaitForFirstConsumer, so the just-emptied node is the only domain that can
take the replacement. Cleaning while cordoned scatters them and leaves two
erasure drives or two Postgres instances sharing a host.

The reclaim deletes PVC, PV and ZFSVolume. k3s-remove-node.sh deleted only
the PVC, which is why 14 orphaned ZFSVolume CRs had accumulated by
2026-09-04 — the finalizer never clears once the owner node is gone.

THE GATE

The four Mayastor predicates deleted in v0.1.743 were the gate's only
TRUTHFUL storage checks. Pod liveness is adequate for a reboot, where data
survives, and NOT for an operation that destroys it. Added: MinIO online and
healing drive counts from the unauthenticated cluster metrics, per-ordinal
OpenBao /v1/sys/health (200 == initialized AND unsealed AND active-or-
standby), and a no-PVC-Pending predicate that catches an incomplete reclaim.
Fail-closed throughout. greenBudget defaults 5400, not the kernel roll's
2400: a CNPG re-clone alone measured 57m35s for three instances.

SAFETY

dryRun defaults TRUE, inverted from pipeline-deploy-kernel — a stray
`argo submit --from` on a template that wipes five disks must print a plan.
Shares the kernel-deploy mutex so neither operation runs while the other has
a node down, holds kernel-deploy-node across the whole node including its
green gate, and pins orchestrator pods to the control plane so the workflow
can never drain the node it runs on.

A dedicated ServiceAccount holds the storage verbs. It can delete a dead PV,
PVC or ZFSVolume and cannot create or patch one; it has nodes: patch and NOT
delete. Requires sean/seanfarm #250 (kernel-deploy-guard creator list) to
land FIRST — that policy is failurePolicy: Fail, so an unlisted creator is
denied at the preflight agent, before anything is drained.

tests/repartition-safety-envelope.sh pins the orderings by line number and
was verified by mutation: deleting the containerd clear, moving labelclear
after the GPT rewrite, arming the intent before enabling the unit, and
granting nodes: delete each turn it red.

tests/check.sh: 147/147 guards pass.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/pipelines!2159
No description provided.