test(archive): prove the packet.town archive restores, not just that it exists #752
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/archive-restore-drill"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
You asked to verify the archive actually works and the chain is configured as
expected, before retiring Codeberg. It did not, quite. Here is what was wrong,
what is now proven, and what is still unexplained.
The defect that mattered
Every ref and every object on the archive was byte-identical to the cluster —
and a plain clone checked out zero files:
git init --barepoints HEAD at the local git default (masteron OmniOS) anda mirror push never sets it. In a recovery that reads as an empty backup.
forge-archive-mirror.shnow points HEAD at the source's default branch afterthe sync, and the guard asserts the property an operator actually needs — a
plain clone, no flags, yields a real object id — rather than the property that
is easy to measure.
Two defects in my own checks
revision through an ssh command string, git never resolved it, and the literal
HEAD^{tree}came back and was reported as PASS. Both the drill and theguard now require 40 or 64 hex, refutation-tested against the literal, the
empty string, and a real id.
--allow-empty, soits tree was git's empty tree
4b825dc6…— every downstream check would havepassed even if the restore lost all file content. It now writes a canary
file and asserts the contents survive byte-for-byte.
What
make archive-restore-drillprovesthrowaway dataset, asserting the right survivors and that a hand-made
snapshot with no
auto-prefix is untouched. This is the only part of thearchive that destroys data and it had never executed when it shipped.
<mountpoint>/.zfs/snapshot/<name>/, which restores one repository withoutrolling the dataset back and taking every other one with it. Identical refs,
clean
fsck, a real object id, canary intact. RTO under 1 second.Operator-invoked and deliberately absent from
tests/—check.shstayscluster-free — and it refuses any repository or dataset name not ending in
-drill.Verified separately
16:00:00.rpg-hello: 13 identical refs and the sametrunktreehash on cluster, forge and archive.
git-shellcontains even an unrestricted key — a throwaway key withoutrestrictstill gotfatal: unrecognized command 'id', so the twocontainments are independent rather than one doing all the work.
One thing left unexplained, on purpose
Pushing to a just-created repository with a just-minted token intermittently
returns
remote: Credentials are incorrect or have expired, and succeedsimmediately after. Seen twice; not reproducible on demand — an isolated
create-then-push worked every time, as did a second push with the same token, and
four drill runs since have needed no retry.
The cause is not established. The drill retries three times and prints that
it retried, rather than hiding it behind
|| true. If that NOTE appears, theintermittency is still live.
Verification
tests/check.sh, exit 0).Codeberg stays. This makes the archive trustworthy enough to start phase 3 —
it does not retire anything.
https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
The archive was complete and unproven. This adds the drill that makes it a backup rather than a hypothesis, and fixes three defects the drill found. THE DEFECT THAT MATTERED. Every ref and every object on the archive was byte-identical to the cluster, and git clone /srv/git/rpg-hello.git warning: remote HEAD refers to nonexistent ref, unable to checkout checked out ZERO files. `git init --bare` points HEAD at the local git default -- master on OmniOS -- and a mirror push never sets it. In a recovery that reads as an empty backup. forge-archive-mirror.sh now points HEAD at the source's default branch after the sync, and the guard asserts the property an operator actually needs: a plain clone, no flags, yields a real object id. A CHECK THAT COULD NOT FAIL. The drill's first clone assertion interpolated the revision through an ssh command string, git never resolved it, and the literal `HEAD^{tree}` came back and was reported as PASS. Both the drill and the guard now require 40 or 64 hex, refutation-tested against the literal, the empty string and a real id. AN ASSERTION ON AN EMPTY TREE. The drill committed with --allow-empty, so its tree was git's empty tree 4b825dc6... and every downstream check would have passed even if the restore lost all file content. It now writes a canary FILE and asserts the contents survive byte-for-byte, the same reason cnpg-restore-drill.sh writes a canary row. WHAT THE DRILL PROVES, in order: 1. RETENTION -- 21 synthetic snapshots over 6 days pruned against a THROWAWAY dataset, asserting the right survivors and that a hand-made snapshot with no auto- prefix is untouched. This is the only part of the archive that destroys data and it had never executed when it shipped. SNAP_DATASET now overrides the baked-in dataset so it can be exercised safely. 2. DELIVERY -- 8 to 9 seconds, no manual sync, over four runs. 3. RECOVERY -- the repository deleted outright and rebuilt from <mountpoint>/.zfs/snapshot/<name>/, which restores ONE repository without rolling the dataset back and taking every other one with it. Identical refs, clean fsck, a real object id, and the canary intact. RTO under 1s. It is operator-invoked and deliberately absent from tests/ -- check.sh stays cluster-free -- and refuses any repository or dataset name not ending in -drill. ALSO VERIFIED, separately: the hourly cron genuinely fires (an unattended snapshot appeared at 16:00:00); rpg-hello has 13 identical refs and the same trunk tree hash on cluster, forge and archive; and git-shell refuses `id` even for a key carrying no restrict, so the two containments are independent. ONE THING LEFT UNEXPLAINED, on purpose. Pushing to a just-created repository with a just-minted token intermittently returns "Credentials are incorrect or have expired" and succeeds immediately after. Seen twice, not reproducible on demand, and four drill runs since have needed no retry. The cause is NOT established, so the drill retries three times and PRINTS that it retried rather than hiding it behind `|| true`. If that NOTE appears, the intermittency is live. Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6