feat(hydra): add a one-off table rewrite Job #71

Merged

binjovi-bot merged 1 commit from feat/hydra-vacuum-maintenance into trunk 2026-09-07 03:24:28 +00:00

Conversation 0 Commits 1 Files changed 4 +381

sean commented 2026-09-07 02:51:02 +00:00

Owner

Copy link

What this does

This adds one file that an operator runs by hand:
manifests/maintenance/job-hydra-vacuum-full.yaml. It rewrites the
hydra_oauth2_access table and gives the disk back to the filesystem.

Why

The hourly hydra janitor CronJob deletes expired tokens. A DELETE keeps the
pages in the table for PostgreSQL to use again. It does not shrink the file on
disk. The janitor's own header comment already says this.

Measured on 2026-09-07 in the hydra database:

• hydra_oauth2_access holds 541 MB.
• It holds 160,151 live rows and 0 dead rows.

The cluster is CloudNativePG postgres in namespace cnpg. The PGDATA volume
is 10Gi with 9.2G free.

pg_repack is not in the image and is not in shared_preload_libraries. It is
rejected. VACUUM FULL is the remaining tool.

The Job refuses to run today

The node agent DaemonSet still runs an old image. It asks for a new token on
every call, about 14,000 tokens each hour. Those rows are live, not dead.
A VACUUM FULL today would reclaim almost nothing.

seanfarm pull request 400 rolls that DaemonSet to a release that keeps its
token. That roll must land first.

Until then the precheck init container stops the run. It reads n_live_tup
and exits non-zero when the count is above MAX_LIVE_ROWS (50,000). The guard
is the reason this Job is safe to keep in the repository.

The guard also fails closed. An empty or non-numeric row count refuses the run.
A test on that value inside an if would hide the error and let the rewrite run
unguarded.

The lock

VACUUM FULL takes an ACCESS EXCLUSIVE lock. Hydra writes this table on every
token grant. Every grant waits while the rewrite runs.

The Job sets lock_timeout to 10 seconds. It fails fast instead of queueing
behind the writers and stalling every later grant. It sets the bound twice: once
with PGOPTIONS when the session opens, and once with SET lock_timeout.

Argo CD does not sync this

manifests/maintenance/ is not referenced by any Argo CD Application. The two
Applications in apps/ read manifests/stack and manifests/users only. There
are no kustomization files in this repository.

This matters. A Job under manifests/stack/ would be re-created and re-run by
Argo CD after its TTL, with no operator watching.

Shape of the Job

• backoffLimit: 0. Never retry a table rewrite on its own.
• restartPolicy: Never.
• activeDeadlineSeconds: 900.
• ttlSecondsAfterFinished: 86400.
• automountServiceAccountToken: false. The Job speaks to PostgreSQL only.
• The same pod and container hardening as the janitor: runAsNonRoot,
  uid and gid 65534, fsGroup 65534, RuntimeDefault seccomp,
  allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, and all
  capabilities dropped. Both containers carry it.
• The image is the CloudNativePG PostgreSQL image, pinned by digest. It is the
  same digest the CNPG bootstrap uses.
• The DSN comes from secretKeyRef on hydra-secrets, key DSN, and nothing
  else. envFrom is not used, because the Job needs one key only. No connection
  string is spelled out in the file.

Tests

New tests/hydra-vacuum-contract.sh, wired into tests/check.sh. It follows
the style of tests/hydra-janitor-contract.sh.

The guard failed first on the missing manifest. Eleven separate mutations were
then tested. Each one made the guard fail:

1. Remove lock_timeout.
2. Add envFrom.
3. Move the file under manifests/stack/.
4. Unpin the image digest.
5. Remove the MAX_LIVE_ROWS comparison.
6. Weaken readOnlyRootFilesystem on the second container.
7. Spell out a postgres:// connection string.
8. Set backoffLimit to 3.
9. Copy the Job under manifests/stack/ as well.
10. Point an Argo CD Application at manifests/maintenance.
11. Stop naming the Job file in the README.

bash tests/check.sh passes. The Job also passes
kubectl apply --dry-run=client. Nothing was applied to the cluster.

README

New section "Hydra table maintenance (one-off, operator)". It gives the
read-only pre-check queries, the kubectl -n ory create -f ... command, how to
watch the run, and the one-day TTL.

It states plainly that the lock stalls every token grant. Expect about 5 to 20
seconds when the live rows are below 30,000. Expect 1 to 3 minutes if the table
is still large.

It says to stay away from the busy times. The CNPG scheduled backup runs every
six hours, at 00:00, 06:00, 12:00 and 18:00 UTC. The janitor runs at minute 23.

https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

## What this does This adds one file that an operator runs by hand: `manifests/maintenance/job-hydra-vacuum-full.yaml`. It rewrites the `hydra_oauth2_access` table and gives the disk back to the filesystem. ## Why The hourly `hydra janitor` CronJob deletes expired tokens. A DELETE keeps the pages in the table for PostgreSQL to use again. It does not shrink the file on disk. The janitor's own header comment already says this. Measured on 2026-09-07 in the `hydra` database: - `hydra_oauth2_access` holds **541 MB**. - It holds **160,151 live rows** and **0 dead rows**. The cluster is CloudNativePG `postgres` in namespace `cnpg`. The PGDATA volume is 10Gi with 9.2G free. `pg_repack` is not in the image and is not in `shared_preload_libraries`. It is rejected. `VACUUM FULL` is the remaining tool. ## The Job refuses to run today The node agent DaemonSet still runs an old image. It asks for a new token on every call, about **14,000 tokens each hour**. Those rows are live, not dead. A `VACUUM FULL` today would reclaim almost nothing. seanfarm pull request 400 rolls that DaemonSet to a release that keeps its token. **That roll must land first.** Until then the `precheck` init container stops the run. It reads `n_live_tup` and exits non-zero when the count is above `MAX_LIVE_ROWS` (50,000). The guard is the reason this Job is safe to keep in the repository. The guard also fails closed. An empty or non-numeric row count refuses the run. A test on that value inside an `if` would hide the error and let the rewrite run unguarded. ## The lock `VACUUM FULL` takes an ACCESS EXCLUSIVE lock. Hydra writes this table on every token grant. Every grant waits while the rewrite runs. The Job sets `lock_timeout` to 10 seconds. It fails fast instead of queueing behind the writers and stalling every later grant. It sets the bound twice: once with `PGOPTIONS` when the session opens, and once with `SET lock_timeout`. ## Argo CD does not sync this `manifests/maintenance/` is not referenced by any Argo CD Application. The two Applications in `apps/` read `manifests/stack` and `manifests/users` only. There are no kustomization files in this repository. This matters. A Job under `manifests/stack/` would be re-created and re-run by Argo CD after its TTL, with no operator watching. ## Shape of the Job - `backoffLimit: 0`. Never retry a table rewrite on its own. - `restartPolicy: Never`. - `activeDeadlineSeconds: 900`. - `ttlSecondsAfterFinished: 86400`. - `automountServiceAccountToken: false`. The Job speaks to PostgreSQL only. - The same pod and container hardening as the janitor: `runAsNonRoot`, uid and gid 65534, `fsGroup` 65534, `RuntimeDefault` seccomp, `allowPrivilegeEscalation: false`, `readOnlyRootFilesystem: true`, and all capabilities dropped. Both containers carry it. - The image is the CloudNativePG PostgreSQL image, pinned by digest. It is the same digest the CNPG bootstrap uses. - The DSN comes from `secretKeyRef` on `hydra-secrets`, key `DSN`, and nothing else. `envFrom` is not used, because the Job needs one key only. No connection string is spelled out in the file. ## Tests New `tests/hydra-vacuum-contract.sh`, wired into `tests/check.sh`. It follows the style of `tests/hydra-janitor-contract.sh`. The guard failed first on the missing manifest. Eleven separate mutations were then tested. Each one made the guard fail: 1. Remove `lock_timeout`. 2. Add `envFrom`. 3. Move the file under `manifests/stack/`. 4. Unpin the image digest. 5. Remove the `MAX_LIVE_ROWS` comparison. 6. Weaken `readOnlyRootFilesystem` on the second container. 7. Spell out a `postgres://` connection string. 8. Set `backoffLimit` to 3. 9. Copy the Job under `manifests/stack/` as well. 10. Point an Argo CD Application at `manifests/maintenance`. 11. Stop naming the Job file in the README. `bash tests/check.sh` passes. The Job also passes `kubectl apply --dry-run=client`. Nothing was applied to the cluster. ## README New section "Hydra table maintenance (one-off, operator)". It gives the read-only pre-check queries, the `kubectl -n ory create -f ...` command, how to watch the run, and the one-day TTL. It states plainly that the lock stalls every token grant. Expect about 5 to 20 seconds when the live rows are below 30,000. Expect 1 to 3 minutes if the table is still large. It says to stay away from the busy times. The CNPG scheduled backup runs every six hours, at 00:00, 06:00, 12:00 and 18:00 UTC. The janitor runs at minute 23. https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

sean added 1 commit 2026-09-07 02:51:02 +00:00

feat(hydra): add a one-off table rewrite Job ... c5e414088e

The janitor deletes expired tokens. A DELETE keeps the pages in the table
for PostgreSQL to use again. It does not give the disk back to the
filesystem. On 2026-09-07 hydra_oauth2_access held 541 MB and 160,151 live
rows with 0 dead rows. Only a table rewrite returns that space.

Add manifests/maintenance/job-hydra-vacuum-full.yaml. It runs
VACUUM (FULL, ANALYZE, VERBOSE) on hydra_oauth2_access. pg_repack is not in
the image and is not in shared_preload_libraries, so it is rejected.

Argo CD does not read manifests/maintenance/. The two Applications in apps/
sync manifests/stack and manifests/users only. A Job under manifests/stack
would be re-created and re-run after its TTL, with no operator watching.

The Job refuses to run while the rows are still live. The node agent
DaemonSet still runs an old image. It asks for a new token on every call,
about 14,000 tokens an hour. Those rows are live, not dead, so a rewrite
gives back almost nothing. seanfarm pull request 400 rolls the agent to a
release that keeps its token. Until that lands, the precheck init container
exits non-zero above MAX_LIVE_ROWS (50,000).

VACUUM FULL takes an ACCESS EXCLUSIVE lock. Hydra writes this table on
every token grant. The Job sets lock_timeout to 10 seconds, so it fails
fast instead of queueing behind the writers and stalling every later grant.

Bounds: backoffLimit 0, restartPolicy Never, activeDeadlineSeconds 900,
ttlSecondsAfterFinished 86400, automountServiceAccountToken false. The pod
and container hardening copies the janitor. The image is the CloudNativePG
PostgreSQL image, pinned by digest. The DSN comes from one key of the
hydra-secrets Secret. No credential is spelled out.

Add tests/hydra-vacuum-contract.sh and wire it into tests/check.sh. It
keeps the Job out of the sync path, keeps the lock bound, keeps the guard,
and keeps the README procedure. Add the README section "Hydra table
maintenance (one-off, operator)".

Claude-Session: https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

sean force-pushed feat/hydra-vacuum-maintenance from c5e414088e to 120625ee6b 2026-09-07 03:05:54 +00:00 Compare

sean force-pushed feat/hydra-vacuum-maintenance from 120625ee6b to 14c89d31eb 2026-09-07 03:12:05 +00:00 Compare

sean force-pushed feat/hydra-vacuum-maintenance from 14c89d31eb to cdf93f2a57 2026-09-07 03:20:49 +00:00 Compare

binjovi-bot merged commit cdf93f2a57 into trunk 2026-09-07 03:24:28 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/ory!71

No description provided.