fix(kernel): drop the Mayastor predicates from the green gate #2158

Merged

pipeline-bot merged 1 commit from chore/green-gate-drop-mayastor into trunk 2026-09-03 23:08:02 +00:00

Conversation 0 Commits 1 Files changed 3 +35 -46

sean commented 2026-09-03 23:07:08 +00:00

Owner

Copy link

Mayastor is being removed from the cluster. This must land and deploy before any teardown starts, because the kernel roll's green gate reads Mayastor in four places and three of them are fail-closed.

The blocker

green_check() in pipeline-deploy-kernel.yaml:

Line	Read	On removal
1158	kubectl get diskpool -A ... || return 1	error: the server doesn't have a resource type "diskpool" -> returns 1 for ever
1163	curl mayastor-api-rest... || return 1	Service gone -> hard fail
1168	kubectl -n openebs-mayastor get sts mayastor-etcd	namespace gone -> hard fail

AGENTS.md names the ability to roll a patched kernel as a security property. Deleting Mayastor first would take that away with no warning, and the failure only shows up at the next roll.

There is an earlier trap in the same block. Line 1157 builds the expected pool set from kubectl get node -l openebs.io/engine=mayastor, and 1159 reports (stale) for any pool whose node has lost the label. Removing the node labels alone already reds the gate, before a single object is deleted.

What changed

The excised range 1153-1175 sits cleanly between the CNPG and OpenBao predicates. local expected has no reader outside it and both j and x are reassigned immediately after, so nothing dangles. Verified against the live ClusterWorkflowTemplate, not only the file.

The safety envelope needed more than a deletion. Every storage assertion in tests/kernel-deploy-safety-envelope.sh was a Mayastor assertion, so removing them would have left the "storage has fully rejoined" half of that guard vacuous: the gate could lose all of its storage predicates and CI would stay green. They are re-pointed at the replicated state that actually remains on a rebooted node, CNPG plus the OpenBao raft plus the MinIO erasure sets, all on ZFS, and extended with an explicit fail-closed check on each of those reads. That last property is the one roll-common.sh got wrong by swallowing kubectl errors with 2>/dev/null.

RBAC loses openebs.io/diskpools and keeps postgresql.cnpg.io/clusters, which the new assertion now pins.

greenBudget stays at 2400. The 2026-08-10 incident that set it is kept as the record of why, with a note that its replica-rebuild half can no longer happen.

Checks

bash tests/check.sh — 146/146 guards pass, 0 skipped, 0 failed. All five kernel contract tests green individually.

After merge

Confirm the live object before touching the cluster:

kubectl get clusterworkflowtemplate pipeline-deploy-kernel -o yaml | grep -ci mayastor   # want 0

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Mayastor is being removed from the cluster. **This must land and deploy before any teardown starts**, because the kernel roll's green gate reads Mayastor in four places and three of them are fail-closed. ## The blocker `green_check()` in `pipeline-deploy-kernel.yaml`: | Line | Read | On removal | | --- | --- | --- | | 1158 | `kubectl get diskpool -A ... \|\| return 1` | `error: the server doesn't have a resource type "diskpool"` -> **returns 1 for ever** | | 1163 | `curl mayastor-api-rest... \|\| return 1` | Service gone -> hard fail | | 1168 | `kubectl -n openebs-mayastor get sts mayastor-etcd` | namespace gone -> hard fail | `AGENTS.md` names the ability to roll a patched kernel as a security property. Deleting Mayastor first would take that away with no warning, and the failure only shows up at the next roll. There is an earlier trap in the same block. Line 1157 builds the expected pool set from `kubectl get node -l openebs.io/engine=mayastor`, and 1159 reports `(stale)` for any pool whose node has lost the label. **Removing the node labels alone already reds the gate**, before a single object is deleted. ## What changed The excised range 1153-1175 sits cleanly between the CNPG and OpenBao predicates. `local expected` has no reader outside it and both `j` and `x` are reassigned immediately after, so nothing dangles. Verified against the live `ClusterWorkflowTemplate`, not only the file. **The safety envelope needed more than a deletion.** Every storage assertion in `tests/kernel-deploy-safety-envelope.sh` was a Mayastor assertion, so removing them would have left the "storage has fully rejoined" half of that guard vacuous: the gate could lose all of its storage predicates and CI would stay green. They are re-pointed at the replicated state that actually remains on a rebooted node, CNPG plus the OpenBao raft plus the MinIO erasure sets, all on ZFS, and extended with an explicit fail-closed check on each of those reads. That last property is the one `roll-common.sh` got wrong by swallowing kubectl errors with `2>/dev/null`. RBAC loses `openebs.io/diskpools` and keeps `postgresql.cnpg.io/clusters`, which the new assertion now pins. `greenBudget` stays at 2400. The 2026-08-10 incident that set it is kept as the record of why, with a note that its replica-rebuild half can no longer happen. ## Checks `bash tests/check.sh` — **146/146 guards pass**, 0 skipped, 0 failed. All five kernel contract tests green individually. ## After merge Confirm the live object before touching the cluster: ``` kubectl get clusterworkflowtemplate pipeline-deploy-kernel -o yaml | grep -ci mayastor # want 0 ``` https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit 2026-09-03 23:07:08 +00:00

fix(kernel): drop the Mayastor predicates from the green gate ... 18487cfc9c

Mayastor is being removed from the cluster. The kernel roll's green gate
reads it in four places and three of them are FAIL-CLOSED, so the moment
the objects go the gate can never return green again:

  green_check() line 1158  kubectl get diskpool -A ... || return 1
  green_check() line 1163  curl mayastor-api-rest ...  || return 1
  green_check() line 1168  kubectl -n openebs-mayastor get sts mayastor-etcd

After the CRD is deleted the first is `error: the server doesn't have a
resource type "diskpool"`, and the gate returns 1 for ever. AGENTS.md
names the ability to roll a patched kernel as a security property, so
this change has to land and deploy BEFORE the cluster teardown starts,
not after.

There is an earlier trap in the same block: line 1157 builds the expected
pool set from `kubectl get node -l openebs.io/engine=mayastor`, and line
1159 reports `(stale)` for a pool whose node lost the label. So removing
the node labels alone already reds the gate, before a single object is
deleted.

The excised range sits cleanly between the CNPG and OpenBao predicates.
`local expected` has no reader outside it, and both `j` and `x` are
reassigned immediately after, so nothing dangles.

The safety envelope needed more than a deletion. EVERY storage assertion
in it was a Mayastor assertion, so removing them would have left the
"storage has fully rejoined" half of that guard vacuous: the gate could
lose all of its storage predicates and CI would stay green. The
assertions are re-pointed at the replicated state that actually remains
on a rebooted node -- CNPG, the OpenBao raft, and the MinIO erasure sets,
all on ZFS -- plus an explicit check that each of those reads is
fail-closed, which is the property the laptop original got wrong.

The RBAC loses openebs.io/diskpools and keeps postgresql.cnpg.io/clusters,
which the new assertion now pins.

greenBudget stays at 2400. The 2026-08-10 incident that set it is kept as
the record of why, with a note that its replica-rebuild half can no longer
happen. It is a ceiling, not a target, and CNPG, OpenBao and MinIO
recovery still needs the room.

tests/check.sh: 146/146 guards pass.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

pipeline-bot merged commit 18487cfc9c into trunk 2026-09-03 23:08:02 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/pipelines!2158

No description provided.