fix(release): delete a retryably-Failed provenance signer so the retry can actually happen

sean commented

2026-08-14 19:27:09 +00:00

Owner

The bug behind today's four-release stall being un-self-healable: on a retryable signer failure the operator cleared the run ref and said retrying, but the Failed workflow kept its deterministic name, so every resubmission adopted the same Failed workflow. attempts stayed frozen for hours; the unwedge was a manual kubectl delete of the four signers.

Fix: record the stall, then after signerRetryBackoff (5m since finishedAt, read via argoengine.Times — engine-boundary clean) delete the Failed workflow and release the name; the next pass submits fresh. Crash-safe via the existing signer-is-gone branch.

envtest: asserts the Failed signer is GONE after a retryable failure (the old test hand-flipped the same workflow to Succeeded — live Argo never does that, which is how the adoption bug hid), plus a pacing episode: fresh finishedAt stays latched, aged finishedAt is deleted and resubmitted. 130/130 guards, full controller suite green.

The bug behind today's four-release stall being un-self-healable: on a retryable signer failure the operator cleared the run ref and said retrying, but the Failed workflow kept its deterministic name, so every resubmission adopted the same Failed workflow. attempts stayed frozen for hours; the unwedge was a manual kubectl delete of the four signers. Fix: record the stall, then after signerRetryBackoff (5m since finishedAt, read via argoengine.Times — engine-boundary clean) delete the Failed workflow and release the name; the next pass submits fresh. Crash-safe via the existing signer-is-gone branch. envtest: asserts the Failed signer is GONE after a retryable failure (the old test hand-flipped the same workflow to Succeeded — live Argo never does that, which is how the adoption bug hid), plus a pacing episode: fresh finishedAt stays latched, aged finishedAt is deleted and resubmitted. 130/130 guards, full controller suite green.

sean added 2 commits

2026-08-14 19:27:09 +00:00

fix(evidence): classify codeberg quota 413 as transport — defer behind the durable copy, do not stall

pipeline/ci CI green @ 39100b51fae4

Details

39100b51fa

The 25 GiB codeberg packages quota filled on 2026-08-14. Every package
PUT returned HTTP 413, which fell into the unclassified arm of put_once,
so all four release trains hard-stalled at provenance — even though the
durable in-cluster evidence copy (MinIO pkg/evidence) plus the deferred
evidence-export executor exist for exactly this failure shape.

413 joins 403/429/000/5xx as the transport class in all seven arms
across the four codeberg publishers (provenance, evidence-export,
deployment-audit, promote-kernel). With DURABLE=1 the provenance signer
now defers the codeberg export and the release proceeds Deployable;
the exporter converges the offsite copy when the quota clears.

tests/codeberg-registry-retry.sh anchors the new class string.

fix(release): a retryably-Failed provenance signer must be deleted, or the retry can never happen

pipeline/ci CI green @ 8b9677d2f3ea

Details

8b9677d2f3

The retry path cleared the run reference and reported "retrying", but the
Failed workflow still occupied its deterministic name — the resubmission
adopted the same Failed workflow and re-latched it. The stall held forever
with zero fresh attempts. On 2026-08-14 four releases sat "retrying" for
hours until the Failed signers were deleted by hand.

Now the failure records the stall first, then — once signerRetryBackoff
(5 m since the workflow finished, via argoengine.Times) has passed —
deletes the Failed workflow and releases the name, so the next pass
genuinely submits a fresh signer. The backoff is the retry pace; the
archive keeps the deleted run's logs. A crash between delete and clearRun
converges through the existing signer-is-gone branch.

The envtest now proves the workflow is GONE after a retryable failure
(the old test hand-flipped the same workflow to Succeeded, which live
Argo never does, masking the adoption bug), and proves the pacing: a
freshly-finished failure stays latched, an aged one is deleted.

pipeline-bot merged commit 8b9677d2f3 into trunk

2026-08-14 19:30:37 +00:00

sean referenced this pull request

2026-08-14 19:41:17 +00:00

fix(release): delete a retryably-Failed evidence exporter so the deferred export can converge #2047

Rows
Columns

fix(release): delete a retryably-Failed provenance signer so the retry can actually happen #2046