feat(buildkit): make binjovi-buildkit the one shared daemon #419

Merged

binjovi-bot merged 1 commit from feat/one-shared-buildkit into trunk 2026-09-07 15:44:54 +00:00

Conversation 0 Commits 1 Files changed 6 +367 -104

sean commented 2026-09-07 15:35:16 +00:00

Owner

Copy link

Six BuildKit caches on five nodes, 200Gi of claim, for largely the same base-layer set. binjovi-buildkit becomes the single daemon for every build in the cluster — Binjovi Change and Release, Zelnet package, IBM i RPG, and every Shipwright build including the kernel chain — on one 100Gi cache on zfs-nvme-128k.

The four daemons it replaces keep running until the sean/binjovi cutover repoints their clients. Only the Shipwright path moves in this PR. The redundant zfs-buildkit-* classes go away in the cleanup commit.

The toml is a superset, not the Binjovi one

The Shipwright daemon was the only one mirroring docker.io, gcr.io and codeberg.org to zot, and those are required: the worker is rootless and slirp4netns cannot reach public registry auth endpoints, so without them every kernel-chain FROM fails. It was also the only one with no [worker.oci] block at all, so GC now governs the kernel ccache too — maxUsedSpace stops at 80GB inside the 100Gi claim. ZFS thin-provisions, so a cache allowed to exceed its claim does not fail; it fills the pool and kubelet evicts. That is the 2026-08-09 incident, 35 hours to clear.

mTLS everywhere, because arithmetic left no choice

buildkitd-s --tlscert/--tlskey/--tlscacert are daemon-global — one gRPC server, one credential set, shared by every listener a repeated --addr creates (verified against the running image-s --help). A daemon serving the Binjovi lanes over mTLS cannot also offer the plaintext door the Shipwright strategy used. The ClusterIssuer signs a third client cert into shipwright-build, and buildkit-cached passes --tlsdir: buildctl has no env var for TLS, but --tlsdir takes a directory of (ca.crt, tls.crt, tls.key) — exactly what cert-manager writes. This removes the last unauthenticated BuildKit endpoint in the cluster.

buildkitd-restrict is retargeted, and that edit is the whole safety of this change

It is the only thing that subtracts egress: allow-cluster-mesh selects every endpoint and grants [cluster, kube-apiserver, world], Cilium unions allow rules, so the daemon-s own allow list documents intent and restricts nothing. Left on shipwright-build/buildkitd the selector matches nothing and the crown-jewel deny list evaporates for every build — fail open, and green in every render and dry run. Checked against live identity labels.

minio is no longer denied — a real reduction

It was denied while the policy fenced only the Shipwright daemon, which never touches the object store. The same daemon now carries the package and RPG lanes, whose builds use minio:9000. Deny beats allow, so leaving it would break both lanes at build time with a timeout and nothing pointing at the cause. Scoped per-project credentials, the H29/H30 isolation checks, and binjovi-untrusted-credential-guard are what still stand in the way.

Consolidation also merges five trust boundaries into one: every daemon authorises by CA, not CN, and the four namespaced Issuers were the only thing stopping one lane-s cert from opening another lane-s daemon. Inherent to one shared daemon.

A guard that was checking a comment

check-kernel-build-cache.sh probe 3 matched buildctl --addr tcp://buildkitd — a string that existed only inside a comment. It would have passed a strategy repointed at any other daemon, and failed a correct repoint that updated the comment. It now asserts BUILDKIT_HOST and the client certificate, and its negative control mutates those instead of checking that its own sed worked.

Verified against six mutations, each detected: policy left on the retired namespace, minio re-denied, docker.io mirror dropped, cache left at one lane-s size, client certificate removed, strategy left pointing at the deleted daemon.

https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

Six BuildKit caches on five nodes, 200Gi of claim, for largely the same base-layer set. `binjovi-buildkit` becomes the single daemon for **every** build in the cluster — Binjovi Change and Release, Zelnet package, IBM i RPG, and every Shipwright build including the kernel chain — on one 100Gi cache on `zfs-nvme-128k`. The four daemons it replaces keep running until the `sean/binjovi` cutover repoints their clients. Only the Shipwright path moves in this PR. The redundant `zfs-buildkit-*` classes go away in the cleanup commit. ### The toml is a superset, not the Binjovi one The Shipwright daemon was the only one mirroring `docker.io`, `gcr.io` and `codeberg.org` to zot, and those are **required**: the worker is rootless and slirp4netns cannot reach public registry auth endpoints, so without them every kernel-chain `FROM` fails. It was also the only one with no `[worker.oci]` block at all, so GC now governs the kernel ccache too — `maxUsedSpace` stops at 80GB inside the 100Gi claim. ZFS thin-provisions, so a cache allowed to exceed its claim does not fail; it fills the pool and kubelet evicts. That is the 2026-08-09 incident, 35 hours to clear. ### mTLS everywhere, because arithmetic left no choice `buildkitd`-s `--tlscert/--tlskey/--tlscacert` are **daemon-global** — one gRPC server, one credential set, shared by every listener a repeated `--addr` creates (verified against the running image-s `--help`). A daemon serving the Binjovi lanes over mTLS cannot also offer the plaintext door the Shipwright strategy used. The ClusterIssuer signs a third client cert into `shipwright-build`, and `buildkit-cached` passes `--tlsdir`: buildctl has no env var for TLS, but `--tlsdir` takes a directory of `(ca.crt, tls.crt, tls.key)` — exactly what cert-manager writes. This removes the last unauthenticated BuildKit endpoint in the cluster. ### `buildkitd-restrict` is retargeted, and that edit is the whole safety of this change It is the only thing that subtracts egress: `allow-cluster-mesh` selects every endpoint and grants `[cluster, kube-apiserver, world]`, Cilium unions allow rules, so the daemon-s own allow list documents intent and restricts nothing. Left on `shipwright-build/buildkitd` the selector matches nothing and the crown-jewel deny list evaporates for every build — **fail open, and green in every render and dry run**. Checked against live identity labels. ### minio is no longer denied — a real reduction It was denied while the policy fenced only the Shipwright daemon, which never touches the object store. The same daemon now carries the package and RPG lanes, whose builds use `minio:9000`. Deny beats allow, so leaving it would break both lanes at **build** time with a timeout and nothing pointing at the cause. Scoped per-project credentials, the H29/H30 isolation checks, and `binjovi-untrusted-credential-guard` are what still stand in the way. Consolidation also merges five trust boundaries into one: every daemon authorises by CA, not CN, and the four namespaced Issuers were the only thing stopping one lane-s cert from opening another lane-s daemon. Inherent to one shared daemon. ### A guard that was checking a comment `check-kernel-build-cache.sh` probe 3 matched `buildctl --addr tcp://buildkitd` — a string that existed **only inside a comment**. It would have passed a strategy repointed at any other daemon, and failed a correct repoint that updated the comment. It now asserts `BUILDKIT_HOST` and the client certificate, and its negative control mutates those instead of checking that its own `sed` worked. Verified against six mutations, each detected: policy left on the retired namespace, minio re-denied, docker.io mirror dropped, cache left at one lane-s size, client certificate removed, strategy left pointing at the deleted daemon. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean added 1 commit 2026-09-07 15:35:17 +00:00

feat(buildkit): make binjovi-buildkit the one shared daemon ... 3440365376

Six BuildKit caches on five nodes, 200Gi of claim, for what is largely the same
base-layer set. This makes binjovi-buildkit the single daemon for every build in
the cluster -- Binjovi Change and Release, Zelnet package, IBM i RPG, and every
Shipwright build including the kernel chain -- on one 100Gi cache on
zfs-nvme-128k. The four daemons it replaces keep running until the sean/binjovi
cutover repoints their clients; only the Shipwright path moves here.

The zfs-buildkit-* classes were redundant. zfs-nvme-* already existed, and a
recordsize comparison belongs to purpose-built benchmark daemons, not to
production caches on a bespoke storage family.

THE TOML IS A SUPERSET, not the Binjovi one. The Shipwright daemon was the only
one mirroring docker.io, gcr.io and codeberg.org to zot, and those are required
rather than an optimisation: the worker is rootless and slirp4netns cannot reach
public registry auth endpoints, so without them every kernel-chain FROM fails.
It was also the only one with no [worker.oci] block at all -- no GC, no space
limits -- so the GC policy now governs the kernel ccache too, and maxUsedSpace
stops at 80GB inside the 100Gi claim. ZFS thin-provisions, so a cache allowed to
exceed its claim does not fail: it fills the pool and kubelet starts evicting.
That is the 2026-08-09 incident, which took 35 hours to clear.

mTLS EVERYWHERE, because arithmetic left no choice. buildkitd's --tlscert,
--tlskey and --tlscacert are DAEMON-GLOBAL -- one gRPC server, one credential
set, shared by every listener a repeated --addr creates -- so a daemon serving
the Binjovi lanes over mTLS cannot also offer the plaintext door the Shipwright
strategy used. Verified against the running image's --help. The ClusterIssuer
now signs a third client certificate, into shipwright-build, and buildkit-cached
passes --tlsdir: buildctl has no env var for TLS, but --tlsdir takes a directory
of (ca.crt, tls.crt, tls.key), which is exactly what cert-manager writes. This
removes the last unauthenticated BuildKit endpoint in the cluster.

buildkitd-restrict is RETARGETED, and that edit is the whole safety of this
change. It is the only thing that subtracts egress: allow-cluster-mesh selects
every endpoint and grants [cluster, kube-apiserver, world], Cilium unions allow
rules, so the daemon's own allow list documents intent and restricts nothing.
Left on shipwright-build/buildkitd the selector would match nothing and the
crown-jewel deny list would evaporate for every build in the cluster -- fail
open, and green in every render and dry run. Checked against the live identity
labels: k8s:app=binjovi-buildkit, k8s:io.kubernetes.pod.namespace=binjovi-builds.

minio IS NO LONGER DENIED, and that is a real reduction. It was denied while the
policy fenced only the Shipwright daemon, which never touches the object store.
The same daemon now carries the package and RPG lanes, whose builds fetch and
push over minio:9000. Cilium resolves deny before allow, so leaving it would
have broken both lanes at BUILD time with a timeout and nothing pointing here.
Per-project scoped credentials, the H29/H30 bucket-isolation checks, and
binjovi-untrusted-credential-guard are what still stand in the way.

Consolidation also merges five trust boundaries into one: every daemon
authorises by CA, not CN, and the four namespaced Issuers were the only thing
stopping one lane's client certificate from opening another lane's daemon. That
is inherent to one shared daemon, not an oversight.

check-kernel-build-cache.sh probe 3 WAS VACUOUS. It matched
'buildctl --addr tcp://buildkitd', a string that existed only inside a comment
at the top of the strategy -- so it would have passed a strategy repointed at
any other daemon, and failed a correct repoint that updated the comment. It now
asserts BUILDKIT_HOST and the client certificate, and its negative control
mutates those rather than checking that its own sed worked.

Verified against six mutations, each detected: the policy left on the retired
namespace, minio re-denied, the docker.io mirror dropped, the cache left at one
lane's size, the client certificate removed, and the strategy left pointing at
the deleted daemon.

Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76

sean force-pushed feat/one-shared-buildkit from 3440365376 to 74104f4d29 2026-09-07 15:40:52 +00:00 Compare

binjovi-bot merged commit 74104f4d29 into trunk 2026-09-07 15:44:54 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!419

No description provided.