chore(mayastor): stop delivering the Mayastor Kustomizations #242

Merged

binjovi-bot merged 1 commit from chore/remove-mayastor into trunk 2026-09-03 23:16:06 +00:00

Conversation 0 Commits 1 Files changed 3 +10 -44

sean commented 2026-09-03 23:13:10 +00:00

Owner

Copy link

Mayastor stores nothing. The storage migration finished on 2026-09-03 and every DiskPool now reports USED 0 B. What remains is the reservation: 4 CPU, 2 GiB of RAM and 2 GiB of hugepages on each of five workers, which is why the workers sit at 88-93% of requested CPU.

This PR is deliberately tiny: two resource lines and two Kustomization CR files. Nothing else.

Why it does not delete anything live, and why that is the point

cluster is prune: false (kubernetes/flux/clusters/seanfarm/infrastructure/cluster.yaml:34), so this does not remove the live openebs-mayastor / openebs-mayastor-pools objects.

prune: false stops garbage collection, not application. While those two files are on main, the cluster Kustomization re-applies them every 10 minutes, so an imperative flux delete would be undone before the next reconcile. This commit is what makes the imperative delete stick. Same mechanism and same reasoning as the four observability agents on 2026-08-07, whose comment this extends.

Why the inner directories stay

Deleting kubernetes/flux/infrastructure/openebs-mayastor{,-pools}/ here would point a still-live Kustomization at a path that no longer exists. A Ready=False Kustomization blocks the kernel roll's green gate. They go in a follow-up PR, after the live objects are gone. Please do not combine the two.

The blocker that was cleared first

The kernel roll's green gate read Mayastor in four places and three were fail-closed:

green_check() 1158   kubectl get diskpool -A ... || return 1
green_check() 1163   curl mayastor-api-rest ...  || return 1
green_check() 1168   kubectl -n openebs-mayastor get sts mayastor-etcd

After the CRD is deleted the first becomes error: the server doesn't have a resource type "diskpool" and the gate returns 1 for ever — silently removing the cluster's ability to roll a patched kernel, which AGENTS.md names as a security property. There was an earlier trap too: the gate builds its expected pool set from kubectl get node -l openebs.io/engine=mayastor, so removing the node labels alone already reds it.

Fixed and deployed first in sean/pipelines #2158, released as v0.1.743. The live ClusterWorkflowTemplate now has zero Mayastor references and the surviving CNPG, OpenBao, MinIO and crashloop predicates are intact.

Checks

bash tests/check.sh passes.

Teardown that follows this merge

flux -n flux-system delete kustomization openebs-mayastor-pools   # DiskPools die while the operator lives
kubectl get diskpool -A                                            # HARD GATE: must be empty
flux -n flux-system delete kustomization openebs-mayastor
kubectl delete crd diskpools.openebs.io

The order is not stylistic. The five DiskPools carry openebs.io/diskpool-protection, cleared only by the running mayastor-operator-diskpool. Kill the operator first and they hang in Terminating for ever and the namespace never deletes.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Mayastor stores nothing. The storage migration finished on 2026-09-03 and **every DiskPool now reports `USED 0 B`**. What remains is the reservation: 4 CPU, 2 GiB of RAM and 2 GiB of hugepages on each of five workers, which is why the workers sit at 88-93% of requested CPU. **This PR is deliberately tiny**: two resource lines and two Kustomization CR files. Nothing else. ## Why it does not delete anything live, and why that is the point `cluster` is `prune: false` (`kubernetes/flux/clusters/seanfarm/infrastructure/cluster.yaml:34`), so this does not remove the live `openebs-mayastor` / `openebs-mayastor-pools` objects. `prune: false` stops garbage collection, not application. While those two files are on `main`, the `cluster` Kustomization re-applies them every 10 minutes, so an imperative `flux delete` would be undone before the next reconcile. **This commit is what makes the imperative delete stick.** Same mechanism and same reasoning as the four observability agents on 2026-08-07, whose comment this extends. ## Why the inner directories stay Deleting `kubernetes/flux/infrastructure/openebs-mayastor{,-pools}/` here would point a still-live Kustomization at a path that no longer exists. A `Ready=False` Kustomization blocks the kernel roll's green gate. They go in a follow-up PR, **after** the live objects are gone. Please do not combine the two. ## The blocker that was cleared first The kernel roll's green gate read Mayastor in four places and three were fail-closed: ``` green_check() 1158 kubectl get diskpool -A ... || return 1 green_check() 1163 curl mayastor-api-rest ... || return 1 green_check() 1168 kubectl -n openebs-mayastor get sts mayastor-etcd ``` After the CRD is deleted the first becomes `error: the server doesn't have a resource type "diskpool"` and the gate returns 1 for ever — silently removing the cluster's ability to roll a patched kernel, which `AGENTS.md` names as a security property. There was an earlier trap too: the gate builds its expected pool set from `kubectl get node -l openebs.io/engine=mayastor`, so **removing the node labels alone already reds it**. Fixed and deployed first in `sean/pipelines` #2158, released as **v0.1.743**. The live `ClusterWorkflowTemplate` now has zero Mayastor references and the surviving CNPG, OpenBao, MinIO and crashloop predicates are intact. ## Checks `bash tests/check.sh` passes. ## Teardown that follows this merge ``` flux -n flux-system delete kustomization openebs-mayastor-pools # DiskPools die while the operator lives kubectl get diskpool -A # HARD GATE: must be empty flux -n flux-system delete kustomization openebs-mayastor kubectl delete crd diskpools.openebs.io ``` The order is not stylistic. The five DiskPools carry `openebs.io/diskpool-protection`, cleared only by the running `mayastor-operator-diskpool`. Kill the operator first and they hang in `Terminating` for ever and the namespace never deletes. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit 2026-09-03 23:13:10 +00:00

chore(mayastor): stop delivering the Mayastor Kustomizations ... 56fd6ae8b4

Mayastor stores nothing. The storage migration moved all eight workloads
to Hetzner Cloud volumes on 2026-09-03, and as of today every DiskPool
reports USED 0 B. What remains is the reservation: 4 CPU, 2 GiB of RAM
and 2 GiB of hugepages on each of five workers, which is why the workers
sit at 88-93% of requested CPU.

This commit is deliberately small. It removes the two resource lines and
the two Kustomization CR files, and nothing else.

The `cluster` Kustomization is prune:false, so this does NOT delete the
live openebs-mayastor and openebs-mayastor-pools objects. That is the
point. prune:false stops garbage collection, not application: while those
two files are on main, `cluster` re-applies them every 10 minutes, so an
imperative `flux delete` would be undone before the next reconcile. This
commit is what makes the delete stick. Same mechanism and same reasoning
as the observability agents on 2026-08-07, whose comment this extends.

The inner manifest directories stay in git on purpose. Deleting them here
would point a still-live Kustomization at a path that no longer exists,
and a Ready=False Kustomization blocks the kernel roll's green gate. They
go in a follow-up, after the live objects are gone.

The green gate itself was unblocked first, in sean/pipelines v0.1.743:
it read Mayastor in four places and three were fail-closed, so deleting
the CRD would have left `kubectl get diskpool -A || return 1` returning 1
for ever. The live ClusterWorkflowTemplate now has zero Mayastor
references.

tests/check.sh passes.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

binjovi-bot merged commit 56fd6ae8b4 into trunk 2026-09-03 23:16:06 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!242

No description provided.