fix(build-images): publish into zot /build/**, not the codeberg mirror namespace #2050

Merged

pipeline-bot merged 1 commit from fix/zot-push-namespace into trunk

2026-08-15 11:11:19 +00:00

sean commented

2026-08-15 11:10:51 +00:00

Owner

The in-cluster-first cutover (a68d66a) moved the build push from codeberg.org/someara to registry.sean.farm/someara. It changed the host and kept the path — and that path is a mirror. Every builder-images build has failed since:

failed to push registry.sean.farm/someara/zig-builder:<sha>:
  unknown: blob upload unknown to registry

Why the namespace decides this

Secret/zot-config in ns registry declares onDemand sync from codeberg:

urls: [https://codeberg.org]  onDemand: true
  prefix: someara/**   destination: /seanfarm
  prefix: someara/**   destination: /someara

/someara/** and /seanfarm/** are read-through caches, not local repos. On a push zot's CheckBlob misses locally, the sync extension asks codeberg, codeberg 404s, and the upload is refused. The zot log shows the whole chain: cache miss → failed to stat blob s3aws: Path not found → failed to request manifest head codeberg.org/someara/...: 404 → image is filtered out by sync config.

The error names a blob upload, so it reads like a zot storage fault. It is a namespace collision.

/build/** is in no sync rule — a real local repo. Shipwright already publishes build/linux-kernel, build/node-bootc, build/forgejo-operator there. This applies that existing convention to the one publisher that missed it.

What changes

Two constructing lines in pipeline-build-images.yaml: the push ref and the MANIFEST line that carries it forward. pipeline-promote-images derives its ref from that MANIFEST, so it follows without an edit — only its comments move.

The codeberg backup leg is unchanged. Promote still copies the released :vN to codeberg.org/someara, bounded and best-effort. This change protects that leg: the alternative fix — dropping the someara/** sync rule so pushes are allowed there — would remove the mirror the backup exists to feed.

Checks

registry-admin holds read/create/update/delete on ** in zot accessControl, and registry-credentials authenticates as registry-admin — /build/** is writable, no policy change needed.
Both templates parse; all 129 pipeline guards pass.

Not in scope (recorded separately)

The four Dockerfile FROMs in builder-images still resolving against codeberg, and the registry-package-prune policy that deletes :vN tags because sha CI tags crowd them out of keep-2 (pipeline-tools, hetzner-builder, go-builder, zig-builder, sysadmin have no version tags left on codeberg).

The in-cluster-first cutover (`a68d66a`) moved the build push from `codeberg.org/someara` to `registry.sean.farm/someara`. It changed the **host** and kept the **path** — and that path is a mirror. Every builder-images build has failed since: ``` failed to push registry.sean.farm/someara/zig-builder:<sha>: unknown: blob upload unknown to registry ``` ## Why the namespace decides this `Secret/zot-config` in ns `registry` declares onDemand sync from codeberg: ``` urls: [https://codeberg.org] onDemand: true prefix: someara/** destination: /seanfarm prefix: someara/** destination: /someara ``` `/someara/**` and `/seanfarm/**` are **read-through caches**, not local repos. On a push zot's `CheckBlob` misses locally, the sync extension asks codeberg, codeberg 404s, and the upload is refused. The zot log shows the whole chain: `cache miss` → `failed to stat blob s3aws: Path not found` → `failed to request manifest head codeberg.org/someara/...: 404` → `image is filtered out by sync config`. The error names a blob upload, so it reads like a zot storage fault. It is a namespace collision. `/build/**` is in **no** sync rule — a real local repo. Shipwright already publishes `build/linux-kernel`, `build/node-bootc`, `build/forgejo-operator` there. This applies that existing convention to the one publisher that missed it. ## What changes Two constructing lines in `pipeline-build-images.yaml`: the push ref and the MANIFEST line that carries it forward. `pipeline-promote-images` derives its ref from that MANIFEST, so it follows without an edit — only its comments move. **The codeberg backup leg is unchanged.** Promote still copies the released `:vN` to `codeberg.org/someara`, bounded and best-effort. This change *protects* that leg: the alternative fix — dropping the `someara/**` sync rule so pushes are allowed there — would remove the mirror the backup exists to feed. ## Checks - `registry-admin` holds `read/create/update/delete` on `**` in zot accessControl, and `registry-credentials` authenticates as `registry-admin` — `/build/**` is writable, no policy change needed. - Both templates parse; **all 129 pipeline guards pass**. ## Not in scope (recorded separately) The four Dockerfile `FROM`s in builder-images still resolving against codeberg, and the `registry-package-prune` policy that deletes `:vN` tags because sha CI tags crowd them out of keep-2 (`pipeline-tools`, `hetzner-builder`, `go-builder`, `zig-builder`, `sysadmin` have no version tags left on codeberg).

sean added 1 commit

2026-08-15 11:10:51 +00:00

fix(build-images): publish into zot /build/**, not the codeberg mirror ns

pipeline/ci CI green @ 8d83d6e496d6

Details

8d83d6e496

The in-cluster-first cutover (a68d66a) moved the build push from
codeberg.org/someara to registry.sean.farm/someara. It changed the host
and kept the path. That path is a mirror, so every builder-images build
has failed since:

  failed to push registry.sean.farm/someara/zig-builder:<sha>:
    unknown: blob upload unknown to registry

WHY THE NAMESPACE DECIDES THIS

Secret zot-config in namespace registry declares onDemand sync from
codeberg:

  urls: [https://codeberg.org]  onDemand: true
    prefix: someara/**   destination: /seanfarm
    prefix: someara/**   destination: /someara

So /someara/** and /seanfarm/** are READ-THROUGH CACHES of codeberg, not
local repositories. On a push, zot's CheckBlob misses locally, the sync
extension asks codeberg for the blob, codeberg answers 404, and the
upload session is refused. The zot log shows the full chain: "cache
miss", then "failed to stat blob s3aws: Path not found", then "failed to
request manifest head codeberg.org/someara/...: 404", then "image is
filtered out by sync config".

The error names a blob upload, so it reads like a zot storage fault. It
is not. It is a namespace collision.

/build/** appears in NO sync rule, so it is a real local repository.
Shipwright already publishes build/linux-kernel, build/node-bootc and
build/forgejo-operator there. This applies that existing convention to
the one publisher that missed it.

WHAT CHANGES

Two constructing lines in pipeline-build-images.yaml: the push ref, and
the MANIFEST line that carries it forward. pipeline-promote-images
derives its whole ref from that MANIFEST line, so it follows without an
edit; only its comments move, to keep them true.

The codeberg backup leg in promote is UNCHANGED. It still copies the
released :vN to codeberg.org/someara, bounded and best-effort. This
change protects that leg rather than weakening it: the alternative fix,
dropping the someara/** sync rule so pushes are allowed there, would
remove the mirror the backup exists to feed.

CHECKS

  - registry-admin holds read/create/update/delete on `**` in the zot
    accessControl, and registry-credentials authenticates as
    registry-admin, so /build/** is writable with no policy change.
  - Both templates still parse, and all 129 pipeline guards pass.

Not in scope, and recorded separately: the four Dockerfile FROMs in
builder-images that still resolve against codeberg, and the
registry-package-prune policy that deletes :vN tags because sha CI tags
crowd them out of keep-2.

pipeline-bot merged commit 8d83d6e496 into trunk

2026-08-15 11:11:19 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/pipelines!2050

No description provided.

Rows
Columns