feat(buildkit-bench): three daemons that differ only in storage class #432

Merged

binjovi-bot merged 1 commit from feat/buildkit-bench-daemons into trunk

2026-09-07 20:42:39 +00:00

sean commented

2026-09-07 20:37:57 +00:00

Owner

buildkit-32k, buildkit-128k, buildkit-1m in namespace buildkit-bench, on zfs-nvme-32k / -128k / -1m. Same image digest, buildkitd.toml, resources, 30Gi cache, TLS chain and scheduling — generated from one template so they cannot drift, and tests/buildkit-bench.sh normalises each daemon-s document set and diffs the three. Anything differing beyond the name and the class fails.

That guard is the point: a daemon that quietly picked up a different parallelism or cache size would still build, still produce a number, and the number would be attributable to nothing.

What it can and cannot answer

Recorded in the manifest header rather than discovered later. These three classes are not a clean recordsize experiment — their parent datasets differ in more than recordsize:

class	dataset	properties
`zfs-nvme-32k`	`tank/databases`	sync=standard, primarycache=all, logbias=latency
`zfs-nvme-128k`	`tank/ephemeral`	sync=disabled
`zfs-nvme-1m`	`tank/bulk`	sync=disabled, primarycache=metadata

So a difference is attributable to the class, not the block size alone — 32k in particular also pays for synchronous writes. That is the right question to ask first, because the class is what a cache would actually be placed on. If the answer turns out to hinge on recordsize rather than durability policy, then a matched-property family earns its place — the opposite order from the zfs-buildkit-* classes this repo just removed for duplicating what already existed.

Isolation

They carry the production daemon-s cluster-wide one-per-node anti-affinity, so a benchmark never lands on a host already running builds. They use their own CA: BuildKit authorises by CA, not CN, so sharing the production ClusterIssuer would let a benchmark client certificate open the production daemon.

Fenced by a separate CiliumClusterwideNetworkPolicy rather than by widening buildkitd-restrict, whose selector is an AND of two In sets — adding this namespace and these three names there widens it by the whole cross product. This namespace holds nothing but benchmark daemons, so selecting the namespace alone is exact. minio is denied here though the production daemon needs it: a benchmark builds one Containerfile and pushes nothing.

Method deliberately not encoded. Run them one at a time with the cache pruned between runs; three daemons building at once contend for ARC and the same NVMe.

Guard verified against seven mutations, each detected.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

`buildkit-32k`, `buildkit-128k`, `buildkit-1m` in namespace `buildkit-bench`, on `zfs-nvme-32k` / `-128k` / `-1m`. Same image digest, buildkitd.toml, resources, 30Gi cache, TLS chain and scheduling — **generated from one template** so they cannot drift, and `tests/buildkit-bench.sh` normalises each daemon-s document set and diffs the three. Anything differing beyond the name and the class fails. That guard is the point: a daemon that quietly picked up a different parallelism or cache size would still build, still produce a number, and the number would be attributable to nothing. ### What it can and cannot answer Recorded in the manifest header rather than discovered later. These three classes are **not** a clean recordsize experiment — their parent datasets differ in more than recordsize: | class | dataset | properties | |---|---|---| | `zfs-nvme-32k` | `tank/databases` | sync=standard, primarycache=all, logbias=latency | | `zfs-nvme-128k` | `tank/ephemeral` | sync=disabled | | `zfs-nvme-1m` | `tank/bulk` | sync=disabled, primarycache=metadata | So a difference is attributable to the **class**, not the block size alone — 32k in particular also pays for synchronous writes. That is the right question to ask first, because the class is what a cache would actually be placed on. If the answer turns out to hinge on recordsize rather than durability policy, *then* a matched-property family earns its place — the opposite order from the `zfs-buildkit-*` classes this repo just removed for duplicating what already existed. ### Isolation They carry the production daemon-s cluster-wide one-per-node anti-affinity, so a benchmark never lands on a host already running builds. They use their **own CA**: BuildKit authorises by CA, not CN, so sharing the production ClusterIssuer would let a benchmark client certificate open the production daemon. Fenced by a **separate** CiliumClusterwideNetworkPolicy rather than by widening `buildkitd-restrict`, whose selector is an AND of two `In` sets — adding this namespace and these three names there widens it by the whole cross product. This namespace holds nothing but benchmark daemons, so selecting the namespace alone is exact. `minio` is denied here though the production daemon needs it: a benchmark builds one Containerfile and pushes nothing. **Method deliberately not encoded.** Run them one at a time with the cache pruned between runs; three daemons building at once contend for ARC and the same NVMe. Guard verified against seven mutations, each detected. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit

2026-09-07 20:37:57 +00:00

feat(buildkit-bench): three daemons that differ only in storage class

binjovi/ci Binjovi completed the frozen plan

Details

3f113f290a

buildkit-32k, buildkit-128k and buildkit-1m in namespace buildkit-bench, on
zfs-nvme-32k, zfs-nvme-128k and zfs-nvme-1m. Same image digest, same
buildkitd.toml, same resources, same 30Gi cache, same TLS chain, same
scheduling. They are generated from one template so they cannot drift, and
tests/buildkit-bench.sh normalises each daemon's document set and diffs the
three: anything differing beyond the name and the class fails.

That guard is the point. A daemon that quietly picked up a different
parallelism or cache size would still build, still produce a number, and the
number would be attributable to nothing.

WHAT IT CAN AND CANNOT ANSWER, recorded in the manifest header rather than
discovered later. The three classes are not a clean recordsize experiment: their
parent datasets differ in more than recordsize -- tank/databases is
sync=standard with primarycache=all, tank/ephemeral and tank/bulk are
sync=disabled, and tank/bulk caches only metadata. So a difference is
attributable to the CLASS, not the block size alone; 32k in particular also pays
for synchronous writes. That is the right question to ask first, because the
class is what a cache would actually be placed on. If the answer turns out to
hinge on recordsize rather than durability policy, THEN a matched-property
family earns its place -- which is the opposite order from the zfs-buildkit-*
classes this repo just removed for duplicating what already existed.

They carry the production daemon's cluster-wide one-per-node anti-affinity, so a
benchmark never lands on a host already running builds, and they use their OWN
CA: BuildKit authorises by CA, not CN, so sharing the production ClusterIssuer
would make a benchmark client certificate open the production daemon.

Fenced by a separate CiliumClusterwideNetworkPolicy rather than by widening
buildkitd-restrict, whose selector is an AND of two In sets -- adding this
namespace and these three names there would widen it by the whole cross product.
This namespace holds nothing but benchmark daemons, so selecting the namespace
alone is exact. minio is denied here though the production daemon needs it: a
benchmark builds one Containerfile and pushes nothing.

Method deliberately not encoded. Run them ONE AT A TIME with the cache pruned
between runs; three daemons building at once contend for ARC and the same NVMe.

Guard verified against seven mutations, each detected: differing parallelism,
cache size or memory limit on one daemon, chaining to the production CA,
reintroducing a bespoke storage class family, un-denying minio, and making the
directory non-prunable.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

binjovi-bot merged commit 3f113f290a into trunk

2026-09-07 20:42:39 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!432

No description provided.

Rows
Columns