fix(tests): make 55 refutations across ten guards able to fail #465

Merged
binjovi-bot merged 1 commit from sean/inert-refutations into trunk 2026-09-08 21:53:24 +00:00 AGit
Owner

'! grep ...' written as a bare statement NEVER fails: bash documents that set -e does not exit if a return value is inverted with '!'. Fifty-five assertions were written that way, including no forgejo-admin-token in three execution manifests, no toEntities: [world] in two policies, no BEGIN OPENSSH PRIVATE KEY in the deploy secret, and no wildcard in the database grant. Before converting anything, all 55 were mechanically rewritten to hard-fail and run: exactly one fired, and it was mis-anchored rather than wrong. That is what makes the conversion safe in one pass. The correct form, '! grep ... || fail ...', consumes the status and is left alone (ten remain). The helper lives in tests/refute.sh rather than tests/lib.sh, which is the cluster test library with its own fail. An unreadable input is treated as a failure, not an absence. tests/refutations-are-effective.sh exercises the helper three ways and refuses any guard that goes back to a bare '! grep', with a non-vacuous scan and three negative controls. Stacks on #464. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

'! grep ...' written as a bare statement NEVER fails: bash documents that set -e does not exit if a return value is inverted with '!'. Fifty-five assertions were written that way, including no forgejo-admin-token in three execution manifests, no toEntities: [world] in two policies, no BEGIN OPENSSH PRIVATE KEY in the deploy secret, and no wildcard in the database grant. Before converting anything, all 55 were mechanically rewritten to hard-fail and run: exactly one fired, and it was mis-anchored rather than wrong. That is what makes the conversion safe in one pass. The correct form, '! grep ... || fail ...', consumes the status and is left alone (ten remain). The helper lives in tests/refute.sh rather than tests/lib.sh, which is the cluster test library with its own fail. An unreadable input is treated as a failure, not an absence. tests/refutations-are-effective.sh exercises the helper three ways and refuses any guard that goes back to a bare '! grep', with a non-vacuous scan and three negative controls. Stacks on #464. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
Argo Events reconciles nothing. `kubectl get sensors,eventsources,eventbus -A`
returns no resources anywhere in the cluster: the Pipelines EventSource that
used to trigger builds went with Pipelines, and Binjovi never used Argo Events.
One `controller-manager` Deployment has been idling in ns `argo-events` since.

Removing the wrapper is not enough on its own. The `cluster` Flux Kustomization
that delivers it is `prune: false`, so deleting the file stops Flux MANAGING
the live `argo-events` Kustomization without deleting it, and the controller
would keep running with nothing in the tree declaring it. That is exactly what
happened to the four observability agents in 2026-08, whose Kustomization
objects had to be deleted separately afterwards.

So this is two halves that must land together: the tree stops declaring
argo-events, and a tombstone Job deletes the live Kustomization object. That
one delete is enough, because the `argo-events` Kustomization is itself
`prune: true` -- Flux garbage-collects the controller, its RBAC and the
namespace around them. The Job's namespace grant is READ ONLY on purpose: Flux
performs the delete and the Job only waits for the namespace to disappear, so a
prune that never ran cannot hide behind a namespace the Job removed itself.

The CRDs stay for a later release. Deleting a CRD while its controller still
runs leaves a finalizer wedge, so `crds/argo-events-*.yaml` goes only after
this Job has run and the controller is confirmed gone.

`tests/argo-events-retired.sh` pins both halves and gives every assertion a
negative control: a widened RBAC grant, a missing prune proof and a namespace
delete grant each turn it red on their own.

The upstream inventory loses the `argo-events` row and the `natsio/` allow
line, whose only subject was the vendored manifest this deletes; no image under
that prefix is left in the tree.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
ArgoCD Application `argo-workflows` syncs the engine into namespace
`workflows`: two `workflows-argo-workflows-server` replicas, two
`workflows-argo-workflows-workflow-controller` replicas, the
`argo-otlp-exporter`, the workflows.sean.farm HTTPRoutes and twelve
Pipelines-era ExternalSecrets -- 70 objects.

Nothing creates a Workflow any more. `builder-images` went native, the
`zot-src` shadow entry never created a Workflow object at all, and the
`argo-workflows` project left `sean/binjovi-plans` in the release before this
one, so Binjovi has already stopped tending the Application. In ArgoCD's own
terms it was orphaned long ago: its tracking-id names `pipelines-argocd`, an
app-of-apps deleted with Pipelines.

THE FINALIZER IS THE POINT. The Application carries no
`resources-finalizer.argocd.argoproj.io` and runs `syncPolicy.automated` with
`selfHeal`, so deleting it as it stands orphans all 70 objects and leaves them
running, and deleting them first just makes ArgoCD put them back. Adding the
finalizer before the delete makes ArgoCD cascade, which also means the
Application's own resource list decides what goes -- safer than this Job
enumerating seventy names and getting one wrong, and it keeps the RBAC to a
single object.

THE NAMESPACE SURVIVES. `workflows` also holds the five live Binjovi agents
and the registry-package-prune CronJob, none of them owned by this
Application. `CreateNamespace=true` made the namespace without tracking it, so
the cascade cannot reach it -- verified against the live Application, whose
resource list names no Namespace at all. The Job asserts that before it patches
anything and refuses if it ever stops being true, asserts the five agents are
running before, and asserts they are still running after. Its grant in ns
`workflows` is read-only: a botched run must not be able to remove the very
agents the post-condition protects. Retiring Pipelines broke a ServiceAccount
and two Forgejo tokens in this same namespace, which is the failure these
pre- and post-conditions exist to catch.

The CRDs stay for a later release. A CRD deleted while its controller still
runs leaves a finalizer wedge, so `crds/argo-workflows-*.yaml` and the 64
finished Workflow records go only after this Job has run.

`tests/argo-workflows-retired.sh` pins the cascade, the read-only namespace
grant, the survivor list and the before-and-after check, parses the inline
script under `sh`, and gives each of them a negative control: a lost finalizer,
a delete grant in ns `workflows` and a dropped survivor each turn it red alone.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
chore(pipelines): archive sean/pipelines
All checks were successful
binjovi/ci Binjovi completed the frozen plan
d6639f8a35
Nothing deploys from it. It sources 0 ArgoCD Applications and 0 Flux
GitRepositories, and every target its Binjovi entry named is already gone --
all five Applications (pipeline-base, pipelines, pipelines-operator,
pipelines-project-registry, workflows-infra-bare) and all four image
Deployments (pipelines-operator-controller-manager, pipelines,
pipelines-release-api, pipelines-mcp), verified NotFound against the live
cluster. Its ~25 pipeline templates, ~17 golden fixtures and 6 guards that name
MinIO are dead code; archiving the repo removes every one of those MinIO
references at once, with no endpoint edit.

BACKUP CONFIRMED FIRST, per gitmirrors/README.md. Forgejo and codeberg agree
exactly on main and trunk (3f02028f) and NO ref exists only on Forgejo. 261
refs exist only on codeberg: the mirror pushes main and trunk with no delete
refspec, so codeberg keeps branches Forgejo has since removed. The README's
`diff ... || STOP` therefore reports a difference for this repo no matter what
-- read the DIRECTION, not the exit code. A codeberg superset is the safe one;
a ref only on Forgejo is the failure that check exists for.

`gitmirrors` is prune: false on purpose -- a bad render under prune: true would
delete every backup mirror in the fleet and stop the codeberg backups silently
-- so removing the file only stops Flux managing the two CRs. A tombstone Job
releases them, push mirror first, because that is the one still writing to
codeberg while the repository CR exists.

THE CODE IS NOT DELETED. Both CRs carry deletionPolicy: Retain, so releasing
them is a no-op on the Forgejo side. The Job re-reads that from the LIVE object
and refuses if either says Delete, rather than trusting the manifest that used
to declare it: before forgejo-operator v0.6.34 the ForgejoRepository finalizer
called DeleteRepo() and destroyed all code, refs and history. What does change
is a cold rebuild -- with no ForgejoRepository there is no importSource, so
`make rebuild` will not re-seed sean/pipelines. That is what archiving it
means, and codeberg keeps the code.

`tests/pipelines-archived.sh` pins the Retain check, the live read, the pinned
RBAC and the release order, parses the inline script under `sh`, and turns red
on a removed policy check or a widened grant.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
chore(argo): delete the Argo CRDs and the RBAC that only Workflows used
Some checks failed
binjovi/ci Binjovi failed the frozen plan
316e18d1f5
RELEASE THIS AFTER the controllers are gone, not with them. Both tombstones
here refuse until they are, but the ordering is real: a CRD deleted while its
controller still watches it wedges on a finalizer.

ELEVEN CRDs. Three from Argo Events, eight from Argo Workflows. The three
ArgoCD CRDs share the argoproj.io group and stay -- 28 live Applications need
them. The `crds` Flux Kustomization is prune: false and says why in its own
comment ("CRITICAL: Never prune CRDs to prevent data loss"), so removing the
files stops Flux re-applying them and nothing more; a tombstone Job does the
delete, with the eleven names pinned in `resourceNames` so RBAC is the fence
whatever the script says. Removing workflows.argoproj.io garbage-collects the
64 finished Workflow records and their pods, on purpose.

FOURTEEN RBAC OBJECTS. Three ServiceAccounts with no running pod, four
`*-executor` RoleBindings bound to a Role the Argo cascade deletes,
Role/RoleBinding binjovi-deploy-flux in ns flux-system -- a standing write
grant on Flux objects held by an account nothing uses, while the native flux
agents have their own -- and five binjovi-argo-* CiliumNetworkPolicies that
select isolation labels NO RUNNING POD carries.

THE ASYMMETRY IS THE RISK. binjovi-deploy-argocd was created for Argo like all
of them and shares their naming, but the live native binjovi-argocd-agent pod
runs as it. It stays; its `-executor` RoleBinding does not. The tombstone
cannot reach it -- it is absent from every resourceNames list -- and the Job
asserts it is present before and after. `binjovi-control-workflows` keeps its
pod and log reads and loses its `argoproj.io/workflows` rule, which is standing
privilege on a kind that no longer exists.

The file is renamed to what it now is: native-deploy-rbac.yaml.

GUARDS. `tests/binjovi-argo-execution.sh` was two guards in one. Its second
half pins the live Kyverno credential policy that protects the package and RPG
lanes, so it survives as tests/binjovi-untrusted-credential-guard.sh; the first
half is replaced by assertions that the retired names cannot return.

While rewriting it: every `! grep` refutation in that guard was INERT. Bash
documents that `set -e` does not exit "if the command's return value is being
inverted with !", so a dozen refutations reported nothing and passed whatever
the file said. They now go through a `refute` helper that exits. It caught a
mistake in this very commit on its first run -- an over-broad refutation of
`apiGroups: [argoproj.io]`, which ArgoCD Applications also need.

tests/argo-crds-retired.sh and tests/binjovi-argo-rbac-retired.sh pin the rest,
each with negative controls: an ArgoCD CRD inside the delete grant, a removed
controller pre-check, a deleted ArgoCD CRD file, a grant widened to the live
agent's ServiceAccount, an unpinned rule, and a dropped post-condition each
turn a guard red on its own.

`tests/argo-workflows-crds/run-all.sh` pinned the v4.0.8 CRD schemas by digest
and goes with them, along with its entries in run-all.sh and
every-guard-has-a-caller.sh.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
fix(tests): make 55 refutations across ten guards able to fail
All checks were successful
binjovi/ci Binjovi completed the frozen plan
feadb696dc
`! grep ...` written as a bare statement NEVER fails. Bash documents that
`set -e` does not exit "if the command's return value is being inverted with
!", so the refutation runs, finds the very thing it refutes, and the script
carries on to print PASS:

  set -euo pipefail
  ! grep -q root /etc/passwd   # grep succeeds, ! yields 1
  echo "still here"            # prints, and the script exits 0

Fifty-five assertions were written that way. Among the things they claim to
prove: no `forgejo-admin-token` in the zelnet, zerotea and zdns execution
manifests; no `toEntities: [world]` in the package-test policy or the SDK-fetch
policy; no `BEGIN OPENSSH PRIVATE KEY` in the deploy secret; no `"*"` in the
database grant; no retired `binjovi-forgejo-integration-v21` generation; no
`pipeline-bot` in the olly and tetragon mirrors. None of them could fail.

NONE OF THEM WAS HIDING A LIVE PROBLEM. Before converting anything, all 55 were
mechanically rewritten to a hard-failing form and run: exactly one fired,
`tests/binjovi.sh:182`, and it was mis-anchored rather than wrong -- fixed in
the commit before this one. That is what makes this conversion safe to do in
one pass rather than one guard at a time.

`! grep ... || fail ...` is CORRECT and is left alone. There the status is
consumed by the `||`, so the inversion is control flow rather than an
assertion. Ten of those remain, in k3s-registries-contract.sh,
host-route-mtu-contract.sh and binjovi-ibmi-rpg-native.sh.

The helper lives in tests/refute.sh, not tests/lib.sh: lib.sh is the CLUSTER
test library -- colours, counters, vault_* and kubectl helpers, and a `fail` of
its own that several of these guards would then be sitting underneath. A
hermetic guard that only reads files should inherit none of that.

An unreadable input is a FAILURE, not an absence. grep exits 1 for "no match"
and 2 or more for an error, and treating an error as absence is the same
vacuous pass in a different costume, so `refute` separates them.

tests/refutations-are-effective.sh keeps both halves fixed: it exercises the
helper on a present pattern, an absent one and an unreadable path, and it
refuses any guard that goes back to a bare `! grep`. Its scan is non-vacuous --
finding no `! grep ... ||` at all fails too, so a pattern that stops matching
cannot pass silently. Three negative controls: a `refute` that returns instead
of exiting, one that accepts an unreadable input, and a reintroduced bare
refutation each turn it red on its own.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
sean force-pushed sean/inert-refutations from feadb696dc
All checks were successful
binjovi/ci Binjovi completed the frozen plan
to b634d1e448
All checks were successful
binjovi/ci Binjovi completed the frozen plan
2026-09-08 21:32:49 +00:00
Compare
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/seanfarm!465
No description provided.