docs(buildkit): record the snapshots that keep filling the cache claim #736

Merged

binjovi-bot merged 1 commit from fix/buildkit-minfreespace into trunk

2026-09-14 13:41:10 +00:00

sean commented

2026-09-14 13:17:35 +00:00

Owner

The 2026-09-10 worker replacement copied this cache to a surviving worker and
left its transfer snapshots behind. On 2026-09-14 the dataset read:

used             89.0G
referenced       11.6G
usedbysnapshots  77.4G
available        10.8G     <-- below the 12GB minFreeSpace
quota            100G

Two snapshots, @arm-20260911-base and @arm-20260911-final, held 77 percent
of a 100G quota carrying 11.6G of live cache.

This is the second time, not the first

The note already in this file about a filesystem with "only 57.9GB and no free
space" on 2026-09-12 is this same dataset, already losing room to these same
snapshots. GC was retuned to cope. The snapshots were never found, so it came
back two days later — this time with available below the floor, and seanfarm
builds failing in gitops-select.

The guard was right and I was wrong

My first draft lowered minFreeSpace to 6GB to make the numbers fit.
tests/binjovi-buildkit.sh refused it:

binjovi-buildkit: FAIL: GC must preserve free space on the shared ZFS pool

That 12GB floor is pinned so a full cache cannot fill the shared pool — the
failure that cost 35 hours on 2026-08-09. No GC setting changes in this PR.

What the comment now carries

available under the floor means the space went somewhere buildkitd cannot
reach; GC cannot win it back. Check zfs get used,referenced,usedbysnapshots
first.
Do not lower minFreeSpace to make it fit.
Nothing in this platform prunes ZFS snapshots. containerd's snapshotter
manages its own (1029 of them, all 0B), the OpenEBS ZFSSnapshot CRD has zero
instances, and no cron or script here creates or reaps any. A snapshot taken
by hand lives until a human destroys it.
maxUsedSpace in the older comment said 80GB. It is 24GB.

Documentation only. The reclaim itself is zfs destroy on the two snapshots,
done out of band.

bash tests/check.sh exits 0.

https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

The 2026-09-10 worker replacement copied this cache to a surviving worker and **left its transfer snapshots behind**. On 2026-09-14 the dataset read: ``` used 89.0G referenced 11.6G usedbysnapshots 77.4G available 10.8G <-- below the 12GB minFreeSpace quota 100G ``` Two snapshots, `@arm-20260911-base` and `@arm-20260911-final`, held **77 percent of a 100G quota carrying 11.6G of live cache**. ### This is the second time, not the first The note already in this file about a filesystem with *"only 57.9GB and no free space"* on 2026-09-12 is **this same dataset, already losing room to these same snapshots**. GC was retuned to cope. The snapshots were never found, so it came back two days later — this time with `available` below the floor, and seanfarm builds failing in `gitops-select`. ### The guard was right and I was wrong My first draft lowered `minFreeSpace` to 6GB to make the numbers fit. `tests/binjovi-buildkit.sh` refused it: ``` binjovi-buildkit: FAIL: GC must preserve free space on the shared ZFS pool ``` That 12GB floor is pinned so a full cache cannot fill the **shared pool** — the failure that cost 35 hours on 2026-08-09. **No GC setting changes in this PR.** ### What the comment now carries - `available` under the floor means the space went somewhere buildkitd cannot reach; GC cannot win it back. Check `zfs get used,referenced,usedbysnapshots` first. - Do not lower `minFreeSpace` to make it fit. - **Nothing in this platform prunes ZFS snapshots.** containerd's snapshotter manages its own (1029 of them, all 0B), the OpenEBS `ZFSSnapshot` CRD has zero instances, and no cron or script here creates or reaps any. A snapshot taken by hand lives until a human destroys it. - `maxUsedSpace` in the older comment said 80GB. It is 24GB. Documentation only. The reclaim itself is `zfs destroy` on the two snapshots, done out of band. `bash tests/check.sh` exits 0. https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean added 1 commit

2026-09-14 13:17:35 +00:00

docs(buildkit): record the snapshots that keep filling the cache claim

binjovi/ci Binjovi failed the frozen plan

Details

f6ae9c2ec6

The 2026-09-10 worker replacement copied this cache to a surviving
worker and left its transfer snapshots behind. On 2026-09-14 the dataset
read: used 89.0G, referenced 11.6G, usedbysnapshots 77.4G, available
10.8G. Two snapshots, @arm-20260911-base and @arm-20260911-final, held
77 percent of a 100G quota that carries 11.6G of live cache.

This is the second time the same cause has shown up. The note above
about a filesystem with "only 57.9GB and no free space" on 2026-09-12 is
this same dataset, already losing room to these same snapshots. GC was
retuned to cope and the snapshots were not found, so it returned two
days later, this time with available below the 12GB minFreeSpace, and
seanfarm builds failed in gitops-select.

The first draft of this commit lowered minFreeSpace to 6GB to make the
numbers fit. tests/binjovi-buildkit.sh refused it: the 12GB floor is
pinned so a full cache cannot fill the SHARED pool, which is what cost
35 hours on 2026-08-09. The guard was right. Nothing in the GC settings
changes here.

So the comment now carries what the next reader needs: available under
the floor means the space went somewhere buildkitd cannot reach, and
snapshots are the first place to look. It also records that nothing in
this platform prunes them. containerd's snapshotter manages its own, the
OpenEBS ZFSSnapshot CRD has no instances, and no cron or script here
creates or reaps any. A snapshot taken by hand lives until a human
destroys it.

maxUsedSpace in the older comment said 80GB. It is 24GB.

tests/check.sh exits 0.

Claude-Session: https://claude.ai/code/session_01GfkEuwuvGSqGyXVuxwT7PA

sean force-pushed fix/buildkit-minfreespace from f6ae9c2ec6

binjovi/ci Binjovi failed the frozen plan

Details

to 46f10788d6

binjovi/ci Binjovi completed the frozen plan

Details

2026-09-14 13:40:46 +00:00

Compare