sean/ory
Watch
1
0
Fork
You've already forked ory
 0
Code Issues Pull requests Projects Releases 68 Packages Wiki Activity

perf(hydra): remove expired tokens on a bounded schedule #69

Merged
binjovi-bot merged 1 commit from perf/hydra-token-janitor into trunk 2026-09-06 23:19:18 +00:00
Conversation 0 Commits 1 Files changed 3 +186
sean commented 2026-09-06 23:17:17 +00:00
Owner
Copy link

Problem

Hydra issues one access token per agent poll and nothing removed the expired rows. Measured on the cluster on 2026-09-06:

Signal Value
hydra_oauth2_access rows 305,218
hydra_oauth2_access size 513 MB
growth about 6,000 rows per hour
oldest row 2026-06-16
hydra database size 501 MB, the largest in the shared CNPG cluster
PGDATA volume 10Gi, 9% used

No hydra janitor ran anywhere: there were no CronJobs in the ory namespace.

The companion change in sean/binjovi makes the agents reuse one token per hour instead of one per poll, which stops most of the growth. This change removes the rows that already exist and the ones the remaining traffic still creates.

Change

An hourly CronJob/hydra-janitor in manifests/stack:

  • Runs hydra janitor --tokens on the pinned server image oryd/hydra:v2.3.0, so its idea of an expired token is the running server's idea.
  • Reads the server's own hydra-config ConfigMap with --config /etc/hydra/hydra.yaml, so it applies the configured ttl block rather than a second copy of those numbers that could drift.
  • Takes the DSN from the existing hydra-secrets Secret with --read-from-env. No connection string appears in the manifest.
  • Bounds each run: --keep-if-younger 1h, --batch-size 500, --limit 50000. The existing backlog clears over about seven runs; the steady state is far below one run's limit.
  • concurrencyPolicy: Forbid, startingDeadlineSeconds: 300, restartPolicy: Never, and the same pod hardening as the rest of the stack.

Scope is deliberately --tokens only. hydra_oauth2_code holds 13 rows and the login and consent request tables are under a megabyte, so --requests and --grants would delete from tables that have no growth problem.

What this does not do

DELETE leaves the pages in the table for PostgreSQL to reuse. The 513 MB file stays 513 MB and stops growing. Returning that space to the filesystem needs VACUUM FULL or pg_repack, which takes an exclusive lock on the table Hydra writes to on every token grant. That stays an operator decision.

Tests

tests/hydra-janitor-contract.sh, wired into tests/check.sh. It pins the scope, the bounds, the shared image and config, the credential source, and the pod hardening.

The guard is proved non-vacuous: from a green baseline, twelve damage cases are each rejected, including the image drifting from the server image, the scope widening to --requests, the row-age floor or per-run limit being removed, concurrent runs being permitted, a DSN being spelled out, and the pod being allowed to run as root or write its root filesystem.

kubectl apply --dry-run=server accepts the manifest.

https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

## Problem Hydra issues one access token per agent poll and nothing removed the expired rows. Measured on the cluster on 2026-09-06: | Signal | Value | |---|---| | `hydra_oauth2_access` rows | 305,218 | | `hydra_oauth2_access` size | 513 MB | | growth | about 6,000 rows per hour | | oldest row | 2026-06-16 | | `hydra` database size | 501 MB, the largest in the shared CNPG cluster | | PGDATA volume | 10Gi, 9% used | No `hydra janitor` ran anywhere: there were no CronJobs in the `ory` namespace. The companion change in `sean/binjovi` makes the agents reuse one token per hour instead of one per poll, which stops most of the growth. This change removes the rows that already exist and the ones the remaining traffic still creates. ## Change An hourly `CronJob/hydra-janitor` in `manifests/stack`: - Runs `hydra janitor --tokens` on the pinned server image `oryd/hydra:v2.3.0`, so its idea of an expired token is the running server's idea. - Reads the server's own `hydra-config` ConfigMap with `--config /etc/hydra/hydra.yaml`, so it applies the configured `ttl` block rather than a second copy of those numbers that could drift. - Takes the DSN from the existing `hydra-secrets` Secret with `--read-from-env`. No connection string appears in the manifest. - Bounds each run: `--keep-if-younger 1h`, `--batch-size 500`, `--limit 50000`. The existing backlog clears over about seven runs; the steady state is far below one run's limit. - `concurrencyPolicy: Forbid`, `startingDeadlineSeconds: 300`, `restartPolicy: Never`, and the same pod hardening as the rest of the stack. Scope is deliberately `--tokens` only. `hydra_oauth2_code` holds 13 rows and the login and consent request tables are under a megabyte, so `--requests` and `--grants` would delete from tables that have no growth problem. ## What this does not do `DELETE` leaves the pages in the table for PostgreSQL to reuse. The 513 MB file stays 513 MB and stops growing. Returning that space to the filesystem needs `VACUUM FULL` or `pg_repack`, which takes an exclusive lock on the table Hydra writes to on every token grant. That stays an operator decision. ## Tests `tests/hydra-janitor-contract.sh`, wired into `tests/check.sh`. It pins the scope, the bounds, the shared image and config, the credential source, and the pod hardening. The guard is proved non-vacuous: from a green baseline, twelve damage cases are each rejected, including the image drifting from the server image, the scope widening to `--requests`, the row-age floor or per-run limit being removed, concurrent runs being permitted, a DSN being spelled out, and the pod being allowed to run as root or write its root filesystem. `kubectl apply --dry-run=server` accepts the manifest. https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w
perf(hydra): remove expired tokens on a bounded schedule
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
768ead527f
 
Hydra issued one access token per agent poll and never removed the
expired rows. On 2026-09-06 hydra_oauth2_access held 305,218 rows in
513 MB, growing about 6,000 rows an hour, and `hydra` had become the
largest database in the shared CNPG cluster on a 10Gi volume.

Add an hourly janitor. It reads the server's own configuration file, so
it removes rows by the same lifespans the server issues them with, and
it takes at most 50,000 rows per run in batches of 500. Scope is
`--tokens`: hydra_oauth2_code holds 13 rows and the request tables are
under a megabyte.

DELETE does not return the pages to the filesystem. The table stops
growing and reuses its own space; reclaiming the 513 MB needs VACUUM
FULL or pg_repack, which takes an exclusive lock and stays an operator
decision.

Claude-Session: https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/ory!69
No description provided.