docs: record that a rebuild is deferred and unproven #731

Merged

binjovi-bot merged 4 commits from docs/rebuild-is-deferred into trunk 2026-09-14 13:41:06 +00:00

Conversation 0 Commits 4 Files changed 1 +42 -5

sean commented 2026-09-14 12:37:47 +00:00

Owner

Copy link

The north star said make rebuild must complete with zero manual
intervention. That reads as an operative rule. It is an aspiration, and the
gap to it is large.

The cluster has run for months without a rebuild, and the probability that
it comes back up is very far from 100 percent. Changes are tested in
production -- the deliberate choice for now -- but rebuild safety drifted away
a long time ago.

Why this is worth a whole section

The old wording already misled a reader. In this session an agent read it as
a live constraint and proposed running make rebuild as the verification gate
for retiring a component. The most dangerous operation available here was
offered as a safety step. The next reader makes the same inference, because the
text invites it.

So the file now says plainly: do not run it, do not propose it as a gate. It
lists the three conditions that must hold first -- stable operations, radical
simplification, and a rebuild proven in a separate environment -- and says
what to verify instead, since work still needs some way to be checked:
fidelity to live state, ordering written into Git rather than implied by a
controller, and live checks in production.

A node roll keeps its own section

Rolling nodes replaces them one at a time against a live cluster. It is a
different operation, the deferral does not apply to it, and the kernel-security
argument for it is unchanged. The two were adjacent in one block and easy to
conflate.

The LE-prod rule loses a stale reason

It was justified by "the dev cluster gets rebuilt often," which is no longer
true. The rule stands on the rate limit alone -- 5 certs per identifier per 168
hours does not care what causes the re-issues.

bash tests/check.sh exits 0, including the four guards that assert on
AGENTS.md content. The CLAUDE.md -> AGENTS.md symlink is untouched.

https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

The north star said `make rebuild` must complete with **zero manual intervention**. That reads as an operative rule. It is an aspiration, and the gap to it is large. The cluster has run for **months** without a rebuild, and the probability that it comes back up is very far from 100 percent. Changes are tested in production -- the deliberate choice for now -- but rebuild safety drifted away a long time ago. ### Why this is worth a whole section The old wording already misled a reader. **In this session an agent read it as a live constraint and proposed running `make rebuild` as the verification gate for retiring a component.** The most dangerous operation available here was offered as a safety step. The next reader makes the same inference, because the text invites it. So the file now says plainly: do not run it, do not propose it as a gate. It lists the three conditions that must hold first -- stable operations, radical simplification, and a rebuild proven in a **separate** environment -- and says what to verify instead, since work still needs some way to be checked: fidelity to live state, ordering written into Git rather than implied by a controller, and live checks in production. ### A node roll keeps its own section Rolling nodes replaces them one at a time against a live cluster. It is a different operation, the deferral does not apply to it, and the kernel-security argument for it is unchanged. The two were adjacent in one block and easy to conflate. ### The LE-prod rule loses a stale reason It was justified by "the dev cluster gets rebuilt often," which is no longer true. The rule stands on the rate limit alone -- 5 certs per identifier per 168 hours does not care what causes the re-issues. `bash tests/check.sh` exits 0, including the four guards that assert on AGENTS.md content. The `CLAUDE.md -> AGENTS.md` symlink is untouched. https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean added 1 commit 2026-09-14 12:37:48 +00:00

docs: record that a rebuild is deferred and unproven ... 813c3830fa

The north star said `make rebuild` must complete with zero manual
intervention. That reads as an operative rule. It is an aspiration, and
the gap to it is large.

The cluster has run for months without a rebuild and the probability that
it comes back up is very far from 100 percent. Changes are tested in
production, which is the deliberate choice for now, but rebuild safety
drifted away a long time ago.

This section exists because the old wording already misled a reader. In
this session an agent read it as a live constraint and proposed running
`make rebuild` as the verification gate for retiring a component. The
most dangerous operation available here was offered as a safety step. The
next reader would have made the same inference.

So the file now says plainly: do not run it, and do not propose it as a
gate. It also lists the three conditions that must hold first, and says
what to verify instead, because work still needs a way to be checked.

A node roll is a separate operation and it keeps its own section. It
replaces nodes one at a time against a live cluster, so the deferral does
not apply to it and the kernel security argument is unchanged.

The LE-prod rule loses its stale reason. It was justified by the cluster
being rebuilt often, which is no longer true. The rule stands on the rate
limit alone, which does not care what causes the re-issues.

tests/check.sh exits 0.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean added 1 commit 2026-09-14 12:38:33 +00:00

chore: retrigger after a keyring quota failure on the build node ... fc33d72608

Build afd416c1 never reached the guards:

  runc run failed: unable to start container process: error during
  container init: unable to join session keyring: unable to create
  session key: disk quota exceeded

That is the kernel keyring quota on the build node, not this change, and
a documentation edit cannot cause it. This commit is empty and only moves
the head so the webhook fires again.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean added 1 commit 2026-09-14 13:00:08 +00:00

chore: retrigger now that buildkit has been rolled ... 44a5071fe1

The keyring quota that failed builds afd416c1 and 275b7275 belonged to the
previous buildkitd pod, up 45 hours. It was replaced at 12:45:33, after
both failures. uid 1000 now holds 3 keys against a 20000 limit.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean added 1 commit 2026-09-14 13:31:44 +00:00

chore: retrigger with the cache claim reclaimed ... afe986a976

The two transfer snapshots are destroyed. The dataset went from used
89.8G / usedbysnapshots 77.8G / available 10.2G to used 12.0G /
usedbysnapshots 0B / available 88.0G, so buildkitd's 12GB minFreeSpace
is reachable again and GC is no longer starved.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean force-pushed docs/rebuild-is-deferred from afe986a976 to 39e3166b78 2026-09-14 13:40:26 +00:00 Compare

binjovi-bot merged commit 39e3166b78 into trunk 2026-09-14 13:41:06 +00:00

binjovi-bot deleted branch docs/rebuild-is-deferred 2026-09-14 13:41:06 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!731

No description provided.