feat(archive): flip git.kri.pt to LE production, build the packet.town archive #748

Merged
binjovi-bot merged 1 commit from feat/forge-archive-packettown into trunk 2026-09-14 15:47:54 +00:00
Owner

Phase 2 of moving off Codeberg. Two gaps blocked it; both are now closed.

TLS is production

git.kri.pt served a staging certificate. The guard was tightened first
— production-or-fail, with a FORGE_TLS_EXPECT escape hatch — because the
issuer string is the only signal available: this estate installs the LE staging
roots on purpose (bootstrap/freebsd.env builds them into the fbsd-k8s CA
bundle), so a failed flip verifies cleanly from every client we own. Verified
against a stock Mozilla bundle, which carries no staging root.

That flip exposed a latent bug in phase 1's own script: the "is there a cert"
test asked whether a file existed, so changing ACME_CA silently kept the old
CA. It now decides from the installed certificate's issuer and passes
--force, which is required to cross CAs. Re-running is a no-op again.

The archive

archive.packet.town — bare git repositories on rpool/archive, pushed to over
SSH by one Forgejo mirror key per repository. Its authorization model is a
single primitive, a line in authorized_keys, and the absence of everything else
is deliberate: no accounts, no web UI, no OIDC, no Forgejo.

That isolation is not a preference. Ory Hydra runs inside the cluster
make rebuild destroys
, and git.kri.pt exists to be the rebuild seed — a
federated login would mean destroying the cluster locks you out of the host
holding the seed you need to rebuild it. The three realms share no identity at
all, only one-way machine credentials pointing downstream:

sean.farm ──[ Forgejo PAT ]──▶ kri.pt ──[ per-repo SSH key ]──▶ packet.town

Snapshots are the backup; the repos are a cache. gitmirrors/README.md
records that a push mirror can delete refs on its remote and that the rule is
unknown — a chain propagates that faithfully. archive-snapshot runs hourly,
keeps 48 hourly and 30 daily, prunes only names carrying its own prefix, and is
unreachable from the git-shell-pinned key that pushes in.

Measured

  • sync_on_commit chains. This was the open "verify, don't assume" question.
    A push into git.kri.pt reached the archive with no manual sync; Forgejo's
    last_update moved 3 seconds later. The git remote update fallback isn't
    needed.
  • A branch_filter naming a branch the repo lacks fails the entire push,
    atomically — main,trunk against a main-only repo lost main too. Same shape
    as the hermes incident. Default is now *.
  • command="git-shell ..." is the obvious authorized_keys line and is wrong
    here
    — sshd runs a forced command through the user's shell, which is
    already git-shell, so it double-wraps and git-shell rejects its own name.
    restrict is used instead. Separately shown: git-shell refuses id even for
    an unrestricted key, so the two are defence in depth.
  • Recreating a mirror mints a new keypair, orphaning the old key line.
  • illumos: useradd doesn't create a matching group (the account lands in
    other, gid 1, shared by every unprivileged account), and install(1) is the
    SysV one that searches for its source with find.

CI caught a real defect

tests/no-sigpipe-readers.sh flagged | grep -q under pipefail — it reports
141 for a match when the producer is still writing, which an if reads as
"not found". Fixed in the new files and in the phase-1 script it had already
shipped in, including a tr </dev/urandom | head -c 48 that had only ever passed
by winning a race.

Verification

  • Full hermetic suite green (tests/check.sh, exit 0).
  • Both live guards pass via their make targets; test-forge-archive was
    refutation-tested (an unrestricted key makes it fail, removing it makes it pass).
  • Both converge scripts are idempotent — second run changes nothing.
  • Canary rpg-hello: 13 refs match exactly between forge and archive.
  • Neither k3s node disturbed: fbsd-k8s up, bridge100 up, 3 rdr rules in the
    k8s anchor; OmniOS kubelet on 10250, zt-gateway and controlplane running.

Deliberately not done

Codeberg stays. The cluster's 37 mirrors (phase 3), the guards (phase 4) and
BOOTSTRAP_REPO_URL (phase 5) are untouched; 158 live files still name it.
The archive is a second site, not a redundant disk — it protects against a bad
push, not against that disk dying. And kri.pt expires 23 October 2026.

https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

Phase 2 of moving off Codeberg. Two gaps blocked it; both are now closed. ## TLS is production `git.kri.pt` served a **staging** certificate. The guard was tightened **first** — production-or-fail, with a `FORGE_TLS_EXPECT` escape hatch — because the issuer string is the only signal available: this estate installs the LE staging roots on purpose (`bootstrap/freebsd.env` builds them into the fbsd-k8s CA bundle), so a *failed* flip verifies cleanly from every client we own. Verified against a stock Mozilla bundle, which carries no staging root. That flip exposed a latent bug in phase 1's own script: the "is there a cert" test asked whether a *file existed*, so changing `ACME_CA` silently kept the old CA. It now decides from the installed certificate's **issuer** and passes `--force`, which is required to cross CAs. Re-running is a no-op again. ## The archive `archive.packet.town` — bare git repositories on `rpool/archive`, pushed to over SSH by **one Forgejo mirror key per repository**. Its authorization model is a single primitive, a line in `authorized_keys`, and the absence of everything else is deliberate: no accounts, no web UI, no OIDC, no Forgejo. That isolation is not a preference. **Ory Hydra runs inside the cluster `make rebuild` destroys**, and `git.kri.pt` exists to be the rebuild seed — a federated login would mean destroying the cluster locks you out of the host holding the seed you need to rebuild it. The three realms share no identity at all, only one-way machine credentials pointing downstream: ``` sean.farm ──[ Forgejo PAT ]──▶ kri.pt ──[ per-repo SSH key ]──▶ packet.town ``` **Snapshots are the backup; the repos are a cache.** `gitmirrors/README.md` records that a push mirror can *delete* refs on its remote and that the rule is unknown — a chain propagates that faithfully. `archive-snapshot` runs hourly, keeps 48 hourly and 30 daily, prunes only names carrying its own prefix, and is unreachable from the `git-shell`-pinned key that pushes in. ## Measured - **`sync_on_commit` chains.** This was the open "verify, don't assume" question. A push into `git.kri.pt` reached the archive with no manual sync; Forgejo's `last_update` moved **3 seconds** later. The `git remote update` fallback isn't needed. - **A `branch_filter` naming a branch the repo lacks fails the *entire* push**, atomically — `main,trunk` against a main-only repo lost `main` too. Same shape as the hermes incident. Default is now `*`. - **`command="git-shell ..."` is the obvious `authorized_keys` line and is wrong here** — sshd runs a forced command *through the user's shell*, which is already `git-shell`, so it double-wraps and git-shell rejects its own name. `restrict` is used instead. Separately shown: `git-shell` refuses `id` even for an *unrestricted* key, so the two are defence in depth. - Recreating a mirror mints a **new** keypair, orphaning the old key line. - illumos: `useradd` doesn't create a matching group (the account lands in `other`, gid 1, shared by every unprivileged account), and `install(1)` is the SysV one that *searches* for its source with `find`. ## CI caught a real defect `tests/no-sigpipe-readers.sh` flagged `| grep -q` under `pipefail` — it reports 141 for a **match** when the producer is still writing, which an `if` reads as "not found". Fixed in the new files *and* in the phase-1 script it had already shipped in, including a `tr </dev/urandom | head -c 48` that had only ever passed by winning a race. ## Verification - Full hermetic suite green (`tests/check.sh`, exit 0). - Both live guards pass via their make targets; `test-forge-archive` was refutation-tested (an unrestricted key makes it fail, removing it makes it pass). - Both converge scripts are idempotent — second run changes nothing. - Canary `rpg-hello`: 13 refs match exactly between forge and archive. - Neither k3s node disturbed: `fbsd-k8s` up, `bridge100` up, 3 `rdr` rules in the `k8s` anchor; OmniOS kubelet on 10250, `zt-gateway` and `controlplane` running. ## Deliberately not done **Codeberg stays.** The cluster's 37 mirrors (phase 3), the guards (phase 4) and `BOOTSTRAP_REPO_URL` (phase 5) are untouched; **158 live files still name it**. The archive is a second *site*, not a redundant disk — it protects against a bad push, not against that disk dying. And **`kri.pt` expires 23 October 2026**. https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
feat(archive): flip git.kri.pt to LE production, build the packet.town archive
All checks were successful
binjovi/ci Binjovi completed the frozen plan
6d01d3a868
Two gaps blocked the Codeberg migration. git.kri.pt served a STAGING
certificate, and there was no second copy of it anywhere: rpool on
freebsd-fsn1 is a stripe of two NVMe with no redundancy. Both are now closed.

TLS IS PRODUCTION. The guard was tightened FIRST, to production-or-fail with a
FORGE_TLS_EXPECT escape hatch, because the issuer string is the only signal
available: this estate installs the LE staging roots on purpose, so a failed
flip verifies cleanly from every client we own. Verified against a stock
Mozilla bundle, which carries no staging root.

That flip also exposed a latent bug in phase 1's own script: the "is there a
cert" test asked whether a file EXISTED, so changing ACME_CA silently kept the
old CA. It now decides from the installed certificate's ISSUER and passes
--force, which is required to cross CAs. Re-running is a no-op again.

THE ARCHIVE. archive.packet.town is bare git repositories on rpool/archive,
pushed to over SSH by one Forgejo mirror key per repository. Its authorization
model is a single primitive -- a line in authorized_keys -- and it is deliberate
that there is nothing else: no accounts, no web UI, no OIDC, no Forgejo.

That isolation is not a preference. Ory Hydra runs INSIDE the cluster that
`make rebuild` destroys, and git.kri.pt exists to be the rebuild seed; a
federated login would mean destroying the cluster locks you out of the host
holding the seed you need to rebuild it. So the three realms share no identity
at all, only one-way machine credentials pointing downstream.

SNAPSHOTS ARE THE BACKUP; THE REPOS ARE A CACHE. gitmirrors/README.md records
that a push mirror can DELETE refs on its remote and that the rule is unknown.
A chain propagates that faithfully. archive-snapshot runs hourly from cron,
keeps 48 hourly and 30 daily, prunes only names carrying its own prefix, and is
unreachable from the git-shell-pinned key that pushes in.

Measured, and each kept as a comment where it bit:

  - sync_on_commit CHAINS. This was the open "verify, do not assume" question.
    A push into git.kri.pt reached the archive with no manual sync; Forgejo's
    last_update moved 3 seconds later. The git-remote-update fallback is not
    needed.
  - A branch_filter naming a branch the repo LACKS fails the ENTIRE push, since
    the push is atomic: "main,trunk" against a main-only repo lost main too.
    Same shape as the hermes incident. The default is now "*", which cannot
    have that failure and is anyway the right filter for a backup.
  - command="git-shell ..." is the obvious authorized_keys line and is WRONG
    here: sshd runs a forced command through the user's shell, which is already
    git-shell, so it double-wraps and git-shell rejects its own name. `restrict`
    is used instead -- and it denies capabilities a future OpenSSH may add,
    rather than silently permitting them. git-shell alone was separately shown
    to refuse `id` even for an unrestricted key.
  - Recreating a mirror mints a NEW keypair, orphaning the old authorized_keys
    line.
  - illumos: useradd does not create a matching group (the account lands in
    `other`, gid 1, shared by every unprivileged account), and install(1) is the
    SysV one that SEARCHES for its source with find and fails.

tests/no-sigpipe-readers.sh caught a real defect: `| grep -q` under pipefail
reports 141 for a MATCH when the producer is still writing, which an `if` reads
as "not found". Fixed in the new files and in the phase-1 script it was already
shipped in, including a `tr </dev/urandom | head -c 48` that had only ever
passed by winning a race.

Both live guards are laptop-only and registered as make targets, since
every-guard-has-a-caller rejects an unreachable guard and a bare NOT_IN_CI entry
would fail a different way.

Codeberg still stays. The cluster's 37 mirrors (phase 3), the guards (phase 4)
and BOOTSTRAP_REPO_URL (phase 5) are untouched; 158 live files still name it.
The archive is a second SITE, not a redundant disk.

Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
binjovi-bot deleted branch feat/forge-archive-packettown 2026-09-14 15:47:54 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/seanfarm!748
No description provided.