sean/seanfarm
Watch
1
0
Fork
You've already forked seanfarm
 0
Code Issues Pull requests Projects Releases 400 Packages Wiki Activity

test(jobs): require a finish timeout on repeatable Jobs #399

Merged
binjovi-bot merged 1 commit from feat/job-ttl-contract into trunk 2026-09-07 03:04:13 +00:00
Conversation 0 Commits 1 Files changed 10 +221
sean commented 2026-09-07 01:56:57 +00:00
Owner
Copy link

The problem

22 finished Jobs set no ttlSecondsAfterFinished. Their pods stay until a
person deletes the Job. On 2026-09-07 the cluster held 104 finished pods, 90
Completed and 14 Error. 88 of those pods belonged to those Jobs.

17 other Job manifests already set ttlSecondsAfterFinished: 86400. The
absence is therefore a gap and not a convention.

Read this before you add a finish timeout

The Flux Kustomizations that carry these Jobs use prune: false. See
kubernetes/flux/clusters/seanfarm/infrastructure/binjovi.yaml, and the same
field in forgejo-users.yaml, tenantbucket.yaml and forgejo-app.yaml.

Flux applies every object that its manifest declares, so Flux creates again any
object that is absent from the cluster. The TTL controller deletes the Job
86400 seconds after the Job finishes. Flux then finds the Job absent and
creates it at the next reconcile. The Job runs again, about one time each
day, for as long as its manifest stays in git.

The cluster shows the cycle. On 2026-09-07 all 29 Jobs that set a finish
timeout were younger than their timeout, and each one was Complete. Example
rows, with the age in hours:

minio   minio-zot-v2-identity                     ttl=86400 age=7.1h  Complete
forgejo binjovi-forgejo-integration-v23           ttl=86400 age=8.9h  Complete
binjovi binjovi-release-cli-oauth-v3              ttl=86400 age=17.3h Complete
forgejo forgejo-admin-token-issuer                ttl=86400 age=23.6h Complete

A finish timeout is therefore correct ONLY for a Job that converges. A second
run must reach the same state, and it must not fail.

The new guard

tests/job-ttl-contract.sh reads every Job document under
kubernetes/flux/infrastructure with yq, and requires
ttlSecondsAfterFinished: 86400. tests/check.sh calls the guard, so the
guard gates a merge. tests/every-guard-has-a-caller.sh passes.

The guard is defensive:

  • Zero Job manifests is a failure, not a pass. A broken search cannot look
    like success.
  • An allowlist entry that names a file that does not exist is a failure.
  • An allowlist entry that names a file with no Job document is a failure.
  • An allowlist entry that no longer describes the manifest is a failure, so
    the lists shrink instead of rot.
  • A file cannot be in both lists.

Six mutation checks confirm that each of those paths goes red.

Jobs that gained ttlSecondsAfterFinished: 86400

Each Job was read first. Each comment in the manifest records why the repeat
run converges.

Manifest Job
binjovi/availability-agent-bootstrap.yaml binjovi-availability-agent-bootstrap-v6
binjovi/buildkit-cache-v2-tombstone.yaml binjovi-buildkit-admission-cache-v2-retire-v2
binjovi/buildkit-cache-v2-tombstone.yaml binjovi-buildkit-release-cache-v2-retire-v2
binjovi/release-cli-oauth-v1-tombstone.yaml binjovi-release-oauth-retire-v1
binjovi/release-cli-oauth-v2-tombstone.yaml binjovi-release-oauth-retire-v2
binjovi/olly-protection-tombstone.yaml olly-protection-retire-v1
binjovi/ibmi-rpg-storage.yaml binjovi-rpg-storage-v1
tenantbucket/binjovi-logs.yaml minio-binjovi-logs
shipwright/zot-base-seed.yaml zot-base-seed

Why each repeat run converges:

  • The four tombstones. Each delete accepts HTTP 404, and each absence wait
    accepts HTTP 404. The target objects are already gone, so a second run
    changes nothing. olly-protection-retire-v1 also requires the three Binjovi
    protections to be Ready first, which is a read.
  • minio-binjovi-logs. The manifest note already records the measurement:
    mc ilm rule import replaces the whole lifecycle configuration, so the
    result is exactly one rule with a fixed ID, however many times the Job runs.
    The bucket, policy and user steps accept the state that is already in place.
    The key comes from a Password generator with refreshInterval: "0", so the
    key does not rotate.
  • binjovi-rpg-storage-v1. Same shape. It uses a bare
    mc admin policy attach, so I checked the cluster: minio-zot-v2-identity
    uses the same bare form, holds a 86400 timeout, and its 2026-09-06 pod log
    shows Attached Policies: [zot-policy] To User: zot-v2 on a MinIO whose
    StatefulSets date from 2026-07-22. A repeat attach is accepted in this mc
    release. The probe objects are written to the same paths each time.
  • zot-base-seed. Its own note says that the Job is rebuild-durable and
    idempotent. It copies the same two amd64 manifests to the same two tags.
  • binjovi-availability-agent-bootstrap-v6. It reads the Deployment
    first, and it patches only when the observed spec differs from the declared
    one. It then waits for one ready replica.

ttlSecondsAfterFinished is a mutable field on a Job, so Flux can patch the
live Jobs in place. The change does not need a delete and recreate.

Jobs in the allowlist, and why

NO_FINISH_TIMEOUT — these Jobs must keep no timeout.

  • forgejo-users/repository-bootstrap.yaml (binjovi-repository-bootstrap-v1)
    — the 40 indexed completions are the import evidence for the 40
    repositories. A timeout deletes that record, and Flux then imports again.
    tests/binjovi-repository-bootstrap.sh line 13 asserts the absence.
  • forgejo-users/management-grants-bootstrap.yaml
    (binjovi-management-grants-bootstrap-v2) — this Job is the evidence that
    the management grants were applied one time to the 41 repositories.
    tests/binjovi-collaborator-bootstrap.sh line 12 asserts the absence.

OTHER_FINISH_TIMEOUT — these Jobs use a different value on purpose. The
guard asserts the exact value, so a silent change is still a failure.

  • argocd/manifest.yaml (argocd-redis-secret-init) = 60 — rendered output
    of the upstream Argo CD chart. The value comes from the chart, and the Job
    is a Helm hook.
  • shipwright-builds/kernel-chain/5-sign/manifest.yaml
    (cosign-sign-kernel-${KERNEL_TAG}) = 5400 — the Job must live longer than
    the 1h reconcile interval of its Kustomization. The manifest holds the
    2026-08-14 measurement that set this value.

Not changed

No Kustomization changes prune: false. Those stay as adoption staging.

Verification

  • bash tests/job-ttl-contract.sh — RED first with 9 failures on the current
    manifests, then GREEN: ok: 37 Job manifest(s) — 35 with a finish timeout, 2 that keep none with a written reason.
  • bash tests/binjovi-repository-bootstrap.sh — PASS.
  • bash tests/binjovi-collaborator-bootstrap.sh — PASS.
  • bash tests/every-guard-has-a-caller.sh — ok: 87 guard(s) — 76 gated by CI.
  • bash tests/check.sh — exit 0 in 4m15s.
  • kubectl apply --dry-run=client -f <file> on all eight edited manifests —
    every document is accepted. No server change was made.

https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

## The problem 22 finished Jobs set no `ttlSecondsAfterFinished`. Their pods stay until a person deletes the Job. On 2026-09-07 the cluster held 104 finished pods, 90 Completed and 14 Error. 88 of those pods belonged to those Jobs. 17 other Job manifests already set `ttlSecondsAfterFinished: 86400`. The absence is therefore a gap and not a convention. ## Read this before you add a finish timeout The Flux Kustomizations that carry these Jobs use `prune: false`. See `kubernetes/flux/clusters/seanfarm/infrastructure/binjovi.yaml`, and the same field in `forgejo-users.yaml`, `tenantbucket.yaml` and `forgejo-app.yaml`. Flux applies every object that its manifest declares, so Flux creates again any object that is absent from the cluster. The TTL controller deletes the Job 86400 seconds after the Job finishes. Flux then finds the Job absent and creates it at the next reconcile. **The Job runs again, about one time each day, for as long as its manifest stays in git.** The cluster shows the cycle. On 2026-09-07 all 29 Jobs that set a finish timeout were younger than their timeout, and each one was Complete. Example rows, with the age in hours: ``` minio minio-zot-v2-identity ttl=86400 age=7.1h Complete forgejo binjovi-forgejo-integration-v23 ttl=86400 age=8.9h Complete binjovi binjovi-release-cli-oauth-v3 ttl=86400 age=17.3h Complete forgejo forgejo-admin-token-issuer ttl=86400 age=23.6h Complete ``` A finish timeout is therefore correct ONLY for a Job that converges. A second run must reach the same state, and it must not fail. ## The new guard `tests/job-ttl-contract.sh` reads every Job document under `kubernetes/flux/infrastructure` with `yq`, and requires `ttlSecondsAfterFinished: 86400`. `tests/check.sh` calls the guard, so the guard gates a merge. `tests/every-guard-has-a-caller.sh` passes. The guard is defensive: - Zero Job manifests is a failure, not a pass. A broken search cannot look like success. - An allowlist entry that names a file that does not exist is a failure. - An allowlist entry that names a file with no Job document is a failure. - An allowlist entry that no longer describes the manifest is a failure, so the lists shrink instead of rot. - A file cannot be in both lists. Six mutation checks confirm that each of those paths goes red. ## Jobs that gained `ttlSecondsAfterFinished: 86400` Each Job was read first. Each comment in the manifest records why the repeat run converges. | Manifest | Job | | --- | --- | | `binjovi/availability-agent-bootstrap.yaml` | `binjovi-availability-agent-bootstrap-v6` | | `binjovi/buildkit-cache-v2-tombstone.yaml` | `binjovi-buildkit-admission-cache-v2-retire-v2` | | `binjovi/buildkit-cache-v2-tombstone.yaml` | `binjovi-buildkit-release-cache-v2-retire-v2` | | `binjovi/release-cli-oauth-v1-tombstone.yaml` | `binjovi-release-oauth-retire-v1` | | `binjovi/release-cli-oauth-v2-tombstone.yaml` | `binjovi-release-oauth-retire-v2` | | `binjovi/olly-protection-tombstone.yaml` | `olly-protection-retire-v1` | | `binjovi/ibmi-rpg-storage.yaml` | `binjovi-rpg-storage-v1` | | `tenantbucket/binjovi-logs.yaml` | `minio-binjovi-logs` | | `shipwright/zot-base-seed.yaml` | `zot-base-seed` | Why each repeat run converges: - **The four tombstones.** Each delete accepts HTTP 404, and each absence wait accepts HTTP 404. The target objects are already gone, so a second run changes nothing. `olly-protection-retire-v1` also requires the three Binjovi protections to be Ready first, which is a read. - **`minio-binjovi-logs`.** The manifest note already records the measurement: `mc ilm rule import` replaces the whole lifecycle configuration, so the result is exactly one rule with a fixed ID, however many times the Job runs. The bucket, policy and user steps accept the state that is already in place. The key comes from a Password generator with `refreshInterval: "0"`, so the key does not rotate. - **`binjovi-rpg-storage-v1`.** Same shape. It uses a bare `mc admin policy attach`, so I checked the cluster: `minio-zot-v2-identity` uses the same bare form, holds a 86400 timeout, and its 2026-09-06 pod log shows `Attached Policies: [zot-policy] To User: zot-v2` on a MinIO whose StatefulSets date from 2026-07-22. A repeat attach is accepted in this mc release. The probe objects are written to the same paths each time. - **`zot-base-seed`.** Its own note says that the Job is rebuild-durable and idempotent. It copies the same two amd64 manifests to the same two tags. - **`binjovi-availability-agent-bootstrap-v6`.** It reads the Deployment first, and it patches only when the observed spec differs from the declared one. It then waits for one ready replica. `ttlSecondsAfterFinished` is a mutable field on a Job, so Flux can patch the live Jobs in place. The change does not need a delete and recreate. ## Jobs in the allowlist, and why **`NO_FINISH_TIMEOUT` — these Jobs must keep no timeout.** - `forgejo-users/repository-bootstrap.yaml` (`binjovi-repository-bootstrap-v1`) — the 40 indexed completions are the import evidence for the 40 repositories. A timeout deletes that record, and Flux then imports again. `tests/binjovi-repository-bootstrap.sh` line 13 asserts the absence. - `forgejo-users/management-grants-bootstrap.yaml` (`binjovi-management-grants-bootstrap-v2`) — this Job is the evidence that the management grants were applied one time to the 41 repositories. `tests/binjovi-collaborator-bootstrap.sh` line 12 asserts the absence. **`OTHER_FINISH_TIMEOUT` — these Jobs use a different value on purpose.** The guard asserts the exact value, so a silent change is still a failure. - `argocd/manifest.yaml` (`argocd-redis-secret-init`) = 60 — rendered output of the upstream Argo CD chart. The value comes from the chart, and the Job is a Helm hook. - `shipwright-builds/kernel-chain/5-sign/manifest.yaml` (`cosign-sign-kernel-${KERNEL_TAG}`) = 5400 — the Job must live longer than the 1h reconcile interval of its Kustomization. The manifest holds the 2026-08-14 measurement that set this value. ## Not changed No Kustomization changes `prune: false`. Those stay as adoption staging. ## Verification - `bash tests/job-ttl-contract.sh` — RED first with 9 failures on the current manifests, then GREEN: `ok: 37 Job manifest(s) — 35 with a finish timeout, 2 that keep none with a written reason`. - `bash tests/binjovi-repository-bootstrap.sh` — PASS. - `bash tests/binjovi-collaborator-bootstrap.sh` — PASS. - `bash tests/every-guard-has-a-caller.sh` — `ok: 87 guard(s) — 76 gated by CI`. - `bash tests/check.sh` — exit 0 in 4m15s. - `kubectl apply --dry-run=client -f <file>` on all eight edited manifests — every document is accepted. No server change was made. https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w
test(jobs): require a finish timeout on repeatable Jobs
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
aa51ae75ff
 
22 finished Jobs set no ttlSecondsAfterFinished, so their pods stay until a
person deletes the Job. On 2026-09-07 the cluster held 104 finished pods, 90
Completed and 14 Error, and 88 of them belonged to those Jobs. 17 other Job
manifests already set ttlSecondsAfterFinished: 86400, so the absence is a gap
and not a convention.

tests/job-ttl-contract.sh reads every Job document under
kubernetes/flux/infrastructure and requires the timeout. tests/check.sh calls
the guard, so the guard gates a merge.

A finish timeout is NOT always correct. The Flux Kustomizations that carry
these Jobs use prune: false. Flux creates again any object that is absent from
the cluster. The TTL controller deletes the Job 86400 seconds after the Job
finishes, and Flux then creates the Job at the next reconcile. The Job RUNS
AGAIN, about one time each day, for as long as its manifest stays in git. A
timeout is therefore correct only for a Job that converges. The cluster shows
this cycle: on 2026-09-07 all 29 Jobs with a timeout were younger than their
timeout, and each one was Complete.

Nine Job documents in eight manifests take the timeout. Each Job was read
first, and each repeat run converges:

  - the four tombstones accept HTTP 404 on each delete and on each absence
    wait, so a second run changes nothing;
  - minio-binjovi-logs and binjovi-rpg-storage-v1 use mc commands that accept
    the state that is already in place, with keys from Password generators
    that do not rotate;
  - zot-base-seed copies the same two amd64 manifests to the same two tags,
    and its own note records that it is idempotent;
  - the availability bridge reads the Deployment first and patches only when
    the observed spec differs from the declared one.

Four manifests stay out of the requirement. The guard holds the reason for
each one:

  - forgejo-users/repository-bootstrap.yaml and
    forgejo-users/management-grants-bootstrap.yaml keep NO timeout. They are
    the import evidence and the grant evidence. A repeat run destroys that
    evidence. tests/binjovi-repository-bootstrap.sh and
    tests/binjovi-collaborator-bootstrap.sh assert the absence;
  - argocd/manifest.yaml uses 60 and
    shipwright-builds/kernel-chain/5-sign/manifest.yaml uses 5400. The guard
    asserts the exact value, so a silent change is still a failure.

Both lists are checked in two directions. An entry that names a file that does
not exist fails. An entry that no longer describes the manifest fails. Zero
Job manifests fails too, so a broken search cannot look like success.

No Kustomization changes prune: false. Those stay as adoption staging.

Claude-Session: https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/seanfarm!399
No description provided.