feat(forge): stand up the off-cluster Forgejo mirror host #740

Merged

binjovi-bot merged 3 commits from feat/forge-jail-kript into trunk

2026-09-14 13:57:35 +00:00

sean commented

2026-09-14 13:45:55 +00:00

Owner

Phase 1 of moving off Codeberg: a reproducible Forgejo at https://git.kri.pt,
in a FreeBSD jail on freebsd-fsn1. It serves. Nothing mirrors to it yet.

Build or converge: scripts/freebsd/forge-jail.sh, as root on the host.
Prove: tests/forge-jail.sh, from anywhere. Both are bash + jq. No Python.

Why

Codeberg does four jobs, not one. Count them before retiring it:

push-mirror backup for 37 repos
the cold-rebuild importSource seed
the Flux bootstrap source (BOOTSTRAP_REPO_URL in the Makefile)
the sean/linux backup check-kernel-source-backup.sh proves before make rebuild

166 tracked files name Codeberg; 158 are live config, scripts or tests.
This host replaces jobs 1 and 2 later. Because it runs outside the cluster it
also survives make rebuild, which makes it a better bootstrap source than
Codeberg rather than only a replacement.

The host is a live k3s node, and that shaped every choice

fbsd-k8s owns bridge100 and the k8s pf anchor, whose rdr rules carry
zdns on 53, 443 and 853. pfctl -f /etc/pf.conf empties anchors declared in the
main ruleset, so a pf reload here takes production DNS down.

The jail is therefore non-vnet and shares the host address: no nat rule, no
rdr rule, no edit to /etc/pf.conf. The script asserts the node is healthy
before it starts and again after it finishes. The guard fails if forge rules
ever appear in pf.conf.

The sharp edge

The kernel rewrites 127.0.0.1 to the jail's first address, so the private
alias 127.0.1.1 is listed first and the public address second. A daemon that
naively binds localhost then lands somewhere private.

nginx alone holds a public port. Forgejo binds 127.0.1.1:3000. PostgreSQL has
no TCP listener at all. The guard probes 3000 and 5432 from outside the host,
because from inside both look reachable either way.

Three defects the red-green loop caught

Each is kept as a comment where it bit:

secret() logged to stdout from inside a command substitution, so a log
line was captured into PG_PASS. app.ini got a PASSWD line holding a log
message and a bare secret on the next line, and Forgejo refused to boot.
step() and info() now write to stderr, closing the class.
jls -h prints its column header even when no jail matches, so the guard's
original "is it running" check could never fail. It asks for the jid now.
Forgejo rewrites app.ini to add LFS_JWT_SECRET and [oauth2] JWT_SECRET.
The script now supplies both, so converging no longer mints new ones and
invalidates every LFS token and OAuth2 session.

Verification

Two consecutive runs report app.ini unchanged — genuinely idempotent.
Guard passes; its new exposure checks were refutation-tested (nc detects an
open 443, does not detect 3000; the reversed address order is rejected).
The k3s node is intact after every run: fbsd-k8s up, bridge100 up, 3 rdr
rules in the k8s anchor.
Base pinned by SHA256; every installed package version asserted after the fact.
acme.sh, not certbot — verified the dependency closure of nginx, acme.sh,
forgejo15 and postgresql17-server pulls in no python3.

Deliberately not done

TLS is Let's Encrypt STAGING. AGENTS.md: production enforces a hard
per-identifier limit and this estate burns it. Switching is ACME_CA=letsencrypt,
once, when the host is proven. A staging cert will make git clone refuse.
No mirrors are configured. That is phase 3, and it stays dual-write with
Codeberg until proven.
Version gap: cluster runs Forgejo 16.0.2, this host 15.0.7 (newest in the
FreeBSD ports tree). Mirroring is plain git over HTTPS and the seed uses the
Gitea-compatible API 15 serves, so it should be fine — prove it on one repo
before trusting it with 37.

Load-bearing warning

rpool on this host is a stripe of two NVMe devices, no redundancy. The
OmniOS box is a single disk. Codeberg is still the only second copy of
sean/linux. Do not retire Codeberg until a second copy of this host exists.

Also: kri.pt expires 23 October 2026.

https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

Phase 1 of moving off Codeberg: a reproducible Forgejo at **https://git.kri.pt**, in a FreeBSD jail on `freebsd-fsn1`. It serves. **Nothing mirrors to it yet.** Build or converge: `scripts/freebsd/forge-jail.sh`, as root on the host. Prove: `tests/forge-jail.sh`, from anywhere. Both are bash + jq. No Python. ## Why Codeberg does four jobs, not one. Count them before retiring it: 1. push-mirror backup for 37 repos 2. the cold-rebuild `importSource` seed 3. the Flux bootstrap source (`BOOTSTRAP_REPO_URL` in the Makefile) 4. the `sean/linux` backup `check-kernel-source-backup.sh` proves before `make rebuild` **166 tracked files name Codeberg; 158 are live config, scripts or tests.** This host replaces jobs 1 and 2 later. Because it runs outside the cluster it also survives `make rebuild`, which makes it a *better* bootstrap source than Codeberg rather than only a replacement. ## The host is a live k3s node, and that shaped every choice `fbsd-k8s` owns `bridge100` and the `k8s` `pf` anchor, whose `rdr` rules carry zdns on 53, 443 and 853. `pfctl -f /etc/pf.conf` empties anchors declared in the main ruleset, so a `pf` reload here takes production DNS down. The jail is therefore **non-vnet and shares the host address**: no `nat` rule, no `rdr` rule, no edit to `/etc/pf.conf`. The script asserts the node is healthy before it starts and again after it finishes. The guard fails if `forge` rules ever appear in `pf.conf`. ## The sharp edge The kernel rewrites `127.0.0.1` to the jail's **first** address, so the private alias `127.0.1.1` is listed first and the public address second. A daemon that naively binds `localhost` then lands somewhere private. nginx alone holds a public port. Forgejo binds `127.0.1.1:3000`. PostgreSQL has no TCP listener at all. The guard probes 3000 and 5432 **from outside the host**, because from inside both look reachable either way. ## Three defects the red-green loop caught Each is kept as a comment where it bit: - `secret()` logged to **stdout from inside a command substitution**, so a log line was captured into `PG_PASS`. `app.ini` got a `PASSWD` line holding a log message and a bare secret on the next line, and Forgejo refused to boot. `step()` and `info()` now write to stderr, closing the class. - `jls -h` prints its column header even when no jail matches, so the guard's original "is it running" check **could never fail**. It asks for the jid now. - Forgejo **rewrites `app.ini`** to add `LFS_JWT_SECRET` and `[oauth2] JWT_SECRET`. The script now supplies both, so converging no longer mints new ones and invalidates every LFS token and OAuth2 session. ## Verification - Two consecutive runs report `app.ini unchanged` — genuinely idempotent. - Guard passes; its new exposure checks were refutation-tested (`nc` detects an open 443, does not detect 3000; the reversed address order is rejected). - The k3s node is intact after every run: `fbsd-k8s` up, `bridge100` up, 3 `rdr` rules in the `k8s` anchor. - Base pinned by SHA256; every installed package version asserted after the fact. - `acme.sh`, not `certbot` — verified the dependency closure of nginx, acme.sh, forgejo15 and postgresql17-server pulls in **no python3**. ## Deliberately not done - **TLS is Let's Encrypt STAGING.** AGENTS.md: production enforces a hard per-identifier limit and this estate burns it. Switching is `ACME_CA=letsencrypt`, once, when the host is proven. A staging cert will make `git clone` refuse. - **No mirrors are configured.** That is phase 3, and it stays dual-write with Codeberg until proven. - **Version gap:** cluster runs Forgejo 16.0.2, this host 15.0.7 (newest in the FreeBSD ports tree). Mirroring is plain git over HTTPS and the seed uses the Gitea-compatible API 15 serves, so it should be fine — prove it on one repo before trusting it with 37. ## Load-bearing warning `rpool` on this host is a **stripe of two NVMe devices, no redundancy**. The OmniOS box is a **single disk**. Codeberg is still the only second copy of `sean/linux`. **Do not retire Codeberg until a second copy of this host exists.** Also: **`kri.pt` expires 23 October 2026.** https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

sean added 1 commit

2026-09-14 13:45:55 +00:00

feat(forge): stand up the off-cluster Forgejo mirror host

binjovi/ci Binjovi failed the frozen plan

Details

eaa2311ad5

Codeberg has to go. This is phase 1: a reproducible Forgejo at
https://git.kri.pt, in a FreeBSD jail on freebsd-fsn1. It serves. Nothing
mirrors to it yet.

Codeberg does four jobs, not one: the push-mirror backup for 37 repos, the
cold-rebuild importSource seed, the Flux bootstrap source named by
BOOTSTRAP_REPO_URL, and the sean/linux backup that check-kernel-source-backup.sh
proves before `make rebuild` may destroy the cluster. 158 live files name it.
This host replaces the first two later; because it runs outside the cluster it
also survives `make rebuild`, which makes it a better bootstrap source than
Codeberg rather than only a replacement.

The host is a LIVE k3s node, and that shaped every choice. fbsd-k8s owns
bridge100 and the `k8s` pf anchor, whose rdr rules carry zdns on 53/443/853.
`pfctl -f /etc/pf.conf` empties anchors declared in the main ruleset, so a pf
reload here takes production DNS down. The jail is therefore non-vnet and
shares the host address: no nat rule, no rdr rule, no edit to /etc/pf.conf.
The script asserts the node is healthy before it starts and again after it
finishes; the guard fails if forge rules ever appear in pf.conf.

Sharing the address has one sharp edge. The kernel rewrites 127.0.0.1 to the
jail's FIRST address, so the private loopback alias 127.0.1.1 is listed first
and the public address second: a daemon that naively binds localhost then lands
somewhere private. nginx alone holds a public port, Forgejo binds 127.0.1.1:3000,
and PostgreSQL has no TCP listener at all. The guard probes 3000 and 5432 from
OUTSIDE the host, because from inside both look reachable either way.

acme.sh, not certbot: certbot is Python. Verified that the dependency closure of
nginx, acme.sh, forgejo15 and postgresql17-server pulls in no python3.

The base is pinned by SHA256 and every installed package version is asserted
after the fact, never inferred from what was requested.

Three defects the red-green loop caught, each kept as a comment where it bit:

  - secret() logged to stdout from inside a command substitution, so a log line
    was captured into PG_PASS. app.ini got a PASSWD line holding a log message
    and a bare secret on the next line, and Forgejo refused to boot. step() and
    info() now write to stderr, which closes the whole class.
  - `jls -h` prints its column header even when no jail matches, so the guard's
    original "is it running" check could never fail. It asks for the jid now.
  - Forgejo REWRITES app.ini to add LFS_JWT_SECRET and [oauth2] JWT_SECRET. The
    script now supplies both, so converging no longer mints new ones and
    invalidates every LFS token and OAuth2 session. app.ini is byte-stable
    across runs; two consecutive runs report it unchanged.

The role password is set on every run rather than only at creation, so a role
left wrong by a failed run repairs itself.

Certificates come from Let's Encrypt STAGING by default. AGENTS.md: production
enforces a hard per-identifier limit and this estate burns it. Switching is one
deliberate env var, once the host is proven.

Not addressed, and load-bearing for what comes next: rpool here is a STRIPE of
two NVMe devices with no redundancy, and the OmniOS box is a single disk.
Codeberg is still the only second copy of sean/linux. Do not retire it until a
second copy of this host exists.

Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

sean added 1 commit

2026-09-14 13:50:02 +00:00

fix(forge): give the forge-jail guard a caller

binjovi/ci Binjovi failed the frozen plan

Details

5eac2a10b3

tests/every-guard-has-a-caller.sh failed the build on the previous commit:
`tests/forge-jail.sh has NO caller`. It is right. A guard nothing runs
protects nothing.

The guard offers three ways out, and only one fits. tests/check.sh is out
because the guard needs ssh to freebsd-fsn1 and a public DNS and TLS probe,
neither of which a CI step has. A bare NOT_IN_CI entry is ALSO out, and that
is the non-obvious part: the guard rejects an exemption for a file that is not
a directory suite, because tests/run-all.sh auto-discovers `tests/*/run-all.sh`
and cannot discover a bare `tests/*.sh` -- the exemption would leave it
unreachable by anything. Every existing NOT_IN_CI entry is a run-all.sh for
exactly that reason.

So: a make target, which is the same shape test-kernel-source-backup already
uses for the same reason. It is a weaker caller -- a human has to run it, and
it does not gate a merge -- and the guard counts it as such rather than as CI
coverage.

Verified locally: 118 guards, 107 gated by CI, 1 reachable only via make,
10 declared not-in-CI with a reason.

Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

sean force-pushed feat/forge-jail-kript from 5eac2a10b3

binjovi/ci Binjovi failed the frozen plan

Details

to 4d010225be

binjovi/ci Binjovi completed the frozen plan

Details

2026-09-14 13:55:16 +00:00

Compare