sean/ory

Ory SSO manifests (Hydra / Kratos / Ory-UI / Oathkeeper / hydra-token-hook + the auth.sean.farm gateway route + the Vault/ESO credential plane), synced by the argocd control plane into ns ory. Migrated out of the Crossplane apis/oidc composition (the ory-alice → ory singleton collapse).

• apps/ — app-of-apps child Applications (ory-stack, ory-users), synced by the root ory-argocd Application (seeded from seanfarm kubernetes/flux/infrastructure/ory-argocd/).
• manifests/stack/ — the Ory server stack (ns ory).
• manifests/users/ — the 4 operator-managed Kratos identities: Alice, Bob, Sean, and Steve.

The Loki route accepts browser sessions and short-lived Hydra service JWTs. Loki does not use Oathkeeper identity headers, so this route uses the no-op mutator. The shared user-header mutator depends on Kratos browser fields and must not run for the service JWT path.

Hydra table maintenance (one-off, operator)

manifests/maintenance/job-hydra-vacuum-full.yaml rewrites the hydra_oauth2_access table and gives the disk back to the filesystem. Argo CD does not sync manifests/maintenance/. The two Applications in apps/ read manifests/stack and manifests/users only. Run this Job by hand.

The hourly janitor deletes expired tokens. A DELETE keeps the pages in the table for PostgreSQL to use again. It does not shrink the file. On 2026-09-07 the table held 541 MB and 160,151 live rows with 0 dead rows.

Read this before you run it. VACUUM FULL takes an ACCESS EXCLUSIVE lock. Hydra writes this table on every token grant. Every token grant waits until the rewrite ends. The Job sets lock_timeout to 10 seconds, so it fails fast instead of queueing behind the writers.

When to run it

Two conditions must be true first.

1. The node agent roll must land. The DaemonSet still runs an old image. It asks for a new token on every call, about 14,000 tokens an hour. Those rows are live, not dead, so a rewrite gives back almost nothing. seanfarm pull request 400 rolls the agent to a release that keeps its token.
2. The janitor must drain the backlog. It runs at minute 23 of every hour and removes at most 50,000 rows in one run.

The Job refuses to start while the live rows are above MAX_LIVE_ROWS (50,000). The precheck init container makes that test and exits non-zero.

Pick a quiet time. The CNPG scheduled backup runs every six hours, at 00:00, 06:00, 12:00 and 18:00 UTC. The janitor runs at minute 23. Stay away from those times.

Pre-checks (read-only)

Open a psql session on the cluster and run these. They change nothing.

kubectl -n cnpg exec -it postgres-1 -- psql -d hydra

SELECT pg_size_pretty(pg_total_relation_size('hydra_oauth2_access'));
SELECT n_live_tup, n_dead_tup, last_autovacuum
  FROM pg_stat_user_tables WHERE relname = 'hydra_oauth2_access';
SELECT count(*) FROM pg_stat_activity
  WHERE datname = 'hydra' AND pid <> pg_backend_pid() AND state <> 'idle';
SELECT count(*) FROM pg_locks
  WHERE NOT granted AND relation = 'hydra_oauth2_access'::regclass;

Go on only when n_live_tup is below 50,000. The PGDATA volume is 10Gi. The rewrite needs free space for a second copy of the table, so keep at least 1 GB free.

Run it

kubectl -n ory create -f manifests/maintenance/job-hydra-vacuum-full.yaml

Use create, not apply. The Job name carries a date. A later rewrite needs a new name.

Watch it

kubectl -n ory get job hydra-vacuum-full-20260907 -w
kubectl -n ory logs -f job/hydra-vacuum-full-20260907 -c precheck
kubectl -n ory logs -f job/hydra-vacuum-full-20260907 -c vacuum

Expect about 5 to 20 seconds when the live rows are below 30,000. Expect 1 to 3 minutes if the table is still large. The Job stops itself after 900 seconds. It never retries.

If the pod fails with a lock timeout, a writer held the table. Wait, then create the Job again under a new name.

The Job removes itself one day after it ends (ttlSecondsAfterFinished: 86400). Read the logs before then.

Binjovi release authentication

Binjovi is the only Ory build and release authority. Run binjovictl authority ory to verify that the provider-guarded handoff is authorized before you request a release.

The exact Binjovi release endpoints for bytey, binjovi, hello-lib, libseanfarm-operator, olly, ory, platform-seanfarm, seanfarm, and tetragon accept either a Kratos browser session or a Hydra bearer token. Oathkeeper verifies the bearer token and puts its subject and scope in a five-minute Binjovi identity token. Binjovi then permits only the binjovi-release-cli subject with the binjovi.release scope. Other Binjovi mutation routes do not accept this service identity, except the exact failed-staging-Deploy retry route, the exact Tetragon trunk-adoption route, and the exact Tetragon legacy-collaborator retirement route. The retirement route can remove only tetragon-pipeline-bot, after Binjovi verifies provider authority. The adoption route remains as an idempotent record of the completed shadow handoff. Binjovi rejects production retries and a second staging retry.

Laptop is read-only to Forgejo; push content with scripts/ory-push.sh <dir> (in-cluster pipeline-bot). That helper git push -fs the whole branch from a local directory, so it is last-writer-wins: diff a fresh clone against your tree before pushing, or you will silently revert someone else's work.

Binjovi health evidence

GET and HEAD requests below /health/ are anonymous. Deployment gates must read readiness and activity evidence without a browser session. The dashboard, assets, API, and all mutation routes remain authenticated.

The backend de-alias is DONE (2026-08-06), so the note that used to sit here — "DB names + Vault creds stay alice-named" — no longer applies. Postgres databases and roles are bare kratos / hydra; the OpenBaoAuth and OpenBaoDynamicSecret CRs carry no spec.namespace (root), with roles {kratos,hydra}-database and paths static-creds/{kratos,hydra}.

Kratos metadata handoff

Eve's identity CR was delivered with deletionPolicy: Retain in v0.1.35 and retired in v0.1.36. Alice, Bob, and Steve received Retain in v0.1.38. This change removes those three CRs so Argo can prune them while their remote identities and credentials remain. Password Secrets stay in the credential plane. Sean stays in platform bootstrap for the initial operator login.

Binjovi manages these four accounts' names, groups, and active states through kratos/seanfarm/identities/USERNAME. Each initial declaration binds the existing UUID, username, and email. It must verify the old CR is absent before it can adopt metadata ownership. Exact GET grants on those four CR names and the Kratos admin ingress grant stay in this repository so Binjovi can verify that no legacy writer returns.

Password rotation is not part of metadata ownership. Changing a retained password Secret no longer rotates the remote password after its CR is removed. Credentials, account creation, and UUID replacement require a separate path. Restore Kratos and Binjovi database backups together to keep UUID bindings.

Native pipeline timing

Binjovi uses native agents for this project's Build, Release, and Deploy tasks. The configured deployment target is ory-development. The deployment verifies the Ory Argo CD applications at the released Git revision.

Select an exact successful Build when you release:

binjovictl builds ory
binjovictl release ory --build BUILD_ID --wait
binjovictl status ory

Open the project dashboard and expand an execution attempt to inspect its task timeline and available profiles. The latest completed pipeline summary includes Build and Release time as well as deployment time. Check the Deploy result separately after Release completes. Telemetry explains execution; the stored revision and verification evidence establish completion.

The native node-image route accepts an agent JWT. Binjovi also checks the admission lane, current lease, and fence before it controls the isolated Job. The gateway does not grant a public node-image endpoint.

Deploy the node-image gateway route before you activate a project plan that uses node_image_job.v1. A gateway rejection must leave the Job unstarted.