fix(kernel-chain): declare the sign wave's push credential, and stop serving revoked tokens #475

Merged

binjovi-bot merged 1 commit from sean/object-store-identities into trunk 2026-09-09 11:03:07 +00:00 AGit

Conversation 0 Commits 1 Files changed 7 +255 -3

sean commented 2026-09-09 10:53:38 +00:00

Owner

Copy link

Two failures found while releasing, both of the same shape as the missing
pipeline-cosign-signer ServiceAccount, and one of them was taking the whole
release train down once a day.

THE SIGN WAVE'S PUSH CREDENTIAL WAS NEVER DECLARED EITHER.

With its ServiceAccount restored, the cosign Job got as far as
MountVolume.SetUp failed ... secret "registry-credentials" not found and sat
in Init:0/1. The retired Pipelines stack created that Secret in namespace
workflows; the sign wave only ever referenced it. It is now declared beside the
Job that mounts it, from the same source as registry-sean-farm-push in
namespace shipwright-build -- the self-minted Zot admin credential through the
singleton registry store. cosign needs it to push the .sig layer.

tests/kernel-sign-dependencies.sh holds the roll-critical wave to declaring
every identity and credential it mounts. Three negative controls, each
confirmed red on its own: the push credential undeclared, the signer identity
undeclared, and the Job falling back to the default account.

A ROTATING TOKEN WAS BEING DELIVERED SLOWER THAN IT WAS REVOKED.

binjovictl release refused every project with
release_candidates_unavailable, and binjovictl authority showed every
provider-guarded project blocked · denied while binjovi -- the one
authoritative project -- was fine. Forgejo's own log had the answer: 401 at
apiAuthentication on branch_protections, from the three control-plane pods.

The binjovi-reader ForgejoToken rotated at 10:11:43Z and Forgejo revoked the
previous token. The ExternalSecret delivering it refreshes hourly and had last
synced at 09:39:39Z, so for the next 28 minutes the control plane presented a
dead credential. The last successful branch-protection read was at 10:11:20 --
twenty-three seconds before the rotation.

This is a daily outage, not an incident. It self-heals, which makes it worse:
it looks transient and returns every day. Three more deliveries of the
release-assets token had the same 1h interval and the same latent failure --
the package lanes would fail an asset upload for up to an hour after each
rotation. All three now match the writer's long-standing 1m.

tests/rotating-token-delivery.sh refuses any ExternalSecret that carries a
ForgejoToken output at anything but 1m. Its negative control was confirmed red.

The fourth delivery, binjovi-forgejo-reader, is declared in sean/binjovi
manifests/credentials.yaml -- the one that actually broke -- and is fixed
there.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Two failures found while releasing, both of the same shape as the missing pipeline-cosign-signer ServiceAccount, and one of them was taking the whole release train down once a day. THE SIGN WAVE'S PUSH CREDENTIAL WAS NEVER DECLARED EITHER. With its ServiceAccount restored, the cosign Job got as far as `MountVolume.SetUp failed ... secret "registry-credentials" not found` and sat in Init:0/1. The retired Pipelines stack created that Secret in namespace workflows; the sign wave only ever referenced it. It is now declared beside the Job that mounts it, from the same source as `registry-sean-farm-push` in namespace shipwright-build -- the self-minted Zot admin credential through the singleton `registry` store. cosign needs it to push the .sig layer. tests/kernel-sign-dependencies.sh holds the roll-critical wave to declaring every identity and credential it mounts. Three negative controls, each confirmed red on its own: the push credential undeclared, the signer identity undeclared, and the Job falling back to the default account. A ROTATING TOKEN WAS BEING DELIVERED SLOWER THAN IT WAS REVOKED. `binjovictl release` refused every project with `release_candidates_unavailable`, and `binjovictl authority` showed every provider-guarded project `blocked · denied` while binjovi -- the one `authoritative` project -- was fine. Forgejo's own log had the answer: 401 at apiAuthentication on `branch_protections`, from the three control-plane pods. The binjovi-reader ForgejoToken rotated at 10:11:43Z and Forgejo revoked the previous token. The ExternalSecret delivering it refreshes hourly and had last synced at 09:39:39Z, so for the next 28 minutes the control plane presented a dead credential. The last successful branch-protection read was at 10:11:20 -- twenty-three seconds before the rotation. This is a daily outage, not an incident. It self-heals, which makes it worse: it looks transient and returns every day. Three more deliveries of the release-assets token had the same 1h interval and the same latent failure -- the package lanes would fail an asset upload for up to an hour after each rotation. All three now match the writer's long-standing 1m. tests/rotating-token-delivery.sh refuses any ExternalSecret that carries a ForgejoToken output at anything but 1m. Its negative control was confirmed red. The fourth delivery, binjovi-forgejo-reader, is declared in sean/binjovi manifests/credentials.yaml -- the one that actually broke -- and is fixed there. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit 2026-09-09 10:53:38 +00:00

fix(kernel-chain): declare the sign wave's push credential, and stop serving revoked tokens ... ca9708010f

Two failures found while releasing, both of the same shape as the missing
pipeline-cosign-signer ServiceAccount, and one of them was taking the whole
release train down once a day.

THE SIGN WAVE'S PUSH CREDENTIAL WAS NEVER DECLARED EITHER.

With its ServiceAccount restored, the cosign Job got as far as
`MountVolume.SetUp failed ... secret "registry-credentials" not found` and sat
in Init:0/1. The retired Pipelines stack created that Secret in namespace
workflows; the sign wave only ever referenced it. It is now declared beside the
Job that mounts it, from the same source as `registry-sean-farm-push` in
namespace shipwright-build -- the self-minted Zot admin credential through the
singleton `registry` store. cosign needs it to push the .sig layer.

tests/kernel-sign-dependencies.sh holds the roll-critical wave to declaring
every identity and credential it mounts. Three negative controls, each
confirmed red on its own: the push credential undeclared, the signer identity
undeclared, and the Job falling back to the default account.

A ROTATING TOKEN WAS BEING DELIVERED SLOWER THAN IT WAS REVOKED.

`binjovictl release` refused every project with
`release_candidates_unavailable`, and `binjovictl authority` showed every
provider-guarded project `blocked · denied` while binjovi -- the one
`authoritative` project -- was fine. Forgejo's own log had the answer: 401 at
apiAuthentication on `branch_protections`, from the three control-plane pods.

The binjovi-reader ForgejoToken rotated at 10:11:43Z and Forgejo revoked the
previous token. The ExternalSecret delivering it refreshes hourly and had last
synced at 09:39:39Z, so for the next 28 minutes the control plane presented a
dead credential. The last successful branch-protection read was at 10:11:20 --
twenty-three seconds before the rotation.

This is a daily outage, not an incident. It self-heals, which makes it worse:
it looks transient and returns every day. Three more deliveries of the
release-assets token had the same 1h interval and the same latent failure --
the package lanes would fail an asset upload for up to an hour after each
rotation. All three now match the writer's long-standing 1m.

tests/rotating-token-delivery.sh refuses any ExternalSecret that carries a
ForgejoToken output at anything but 1m. Its negative control was confirmed red.

The fourth delivery, binjovi-forgejo-reader, is declared in sean/binjovi
manifests/credentials.yaml -- the one that actually broke -- and is fixed
there.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit c5f75d2a4e into trunk 2026-09-09 11:03:07 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!475

No description provided.