docs(build-images): retract a wrong root-cause claim about the push namespace #2051

Merged

pipeline-bot merged 1 commit from fix/correct-zot-ns-rationale into trunk

2026-08-15 11:18:53 +00:00

sean commented

2026-08-15 11:18:21 +00:00

Owner

The comment added in 8d83d6e claimed a push into a sync-mirrored namespace fails with blob upload unknown to registry. That is wrong, and the evidence was available before I wrote it.

builder-images-build-auto-gxvwr pushed all 14 images to registry.sean.farm/someara/ and succeeded, 10:30–11:00 UTC today. The sync rule does not block a push. I read the zot log's failed to stat blob and image is filtered out by sync config lines as the cause; they are ordinary CheckBlob misses and the sync extension reacting to them.

What is actually observed, still unexplained

the same error happens on both someara/ and build/
my build rtqs7 failed on someara/ at 10:51 — inside the window where gxvwr was succeeding on that same namespace
within a run it is repeatable per image, not random: cast-render and envtest-tester fail while go-builder, debian-repo-signer, hetzner-builder, ibmi-builder succeed
zot is healthy: 0 restarts, livez/readyz 200, and a 134 MB blob PUT returns 201 in the same minute as a failure
MinIO is not full (/export 22% used)
dedupe: false, cacheDriver: null — neither explains it
failures begin when two builds push concurrently

The namespace change stays

/build/** is where Shipwright already publishes, so per-commit candidates belong there rather than mixed into the namespaces that mirror released artifacts. That is hygiene, and the comment now says so instead of claiming a fix.

Root cause of the intermittent push failure is OPEN. 129/129 pipeline guards pass.

The comment added in `8d83d6e` claimed a push into a sync-mirrored namespace fails with `blob upload unknown to registry`. **That is wrong**, and the evidence was available before I wrote it. `builder-images-build-auto-gxvwr` pushed **all 14 images to `registry.sean.farm/someara/` and succeeded**, 10:30–11:00 UTC today. The sync rule does not block a push. I read the zot log's `failed to stat blob` and `image is filtered out by sync config` lines as the cause; they are ordinary CheckBlob misses and the sync extension reacting to them. ## What is actually observed, still unexplained - the same error happens on **both** `someara/` and `build/` - my build `rtqs7` failed on `someara/` at 10:51 — *inside* the window where `gxvwr` was succeeding on that same namespace - within a run it is **repeatable per image**, not random: `cast-render` and `envtest-tester` fail while `go-builder`, `debian-repo-signer`, `hetzner-builder`, `ibmi-builder` succeed - zot is healthy: 0 restarts, `livez`/`readyz` 200, and a 134 MB blob PUT returns 201 in the same minute as a failure - MinIO is not full (`/export` 22% used) - `dedupe: false`, `cacheDriver: null` — neither explains it - failures begin when two builds push concurrently ## The namespace change stays `/build/**` is where Shipwright already publishes, so per-commit candidates belong there rather than mixed into the namespaces that mirror released artifacts. That is **hygiene**, and the comment now says so instead of claiming a fix. **Root cause of the intermittent push failure is OPEN.** 129/129 pipeline guards pass.

sean added 1 commit

2026-08-15 11:18:21 +00:00

docs(build-images): retract a wrong root-cause claim about the push namespace

pipeline/ci CI green @ 9affd356bea3

Details

9affd356be

The comment added in 8d83d6e said a push into a sync-mirrored namespace
fails with "blob upload unknown to registry". That is WRONG, and the
evidence was available before it was written.

Build builder-images-build-auto-gxvwr pushed ALL 14 images to
registry.sean.farm/someara/ and succeeded, at 2026-08-15 10:30-11:00 UTC.
So the sync rule does not block a push. The mistake was to read the zot
log's "failed to stat blob" and "image is filtered out by sync config"
lines as the cause. They are ordinary CheckBlob misses and the sync
extension's reaction to them, not the failure.

What is actually observed, and still UNEXPLAINED:

  - the same "blob upload unknown to registry" happens on BOTH the
    someara/ and build/ namespaces
  - build builder-images-build-images-rtqs7 failed on someara/ at 10:51,
    INSIDE the window where gxvwr was succeeding on the same namespace
  - within one run it is repeatable per image, not random: cast-render and
    envtest-tester fail while go-builder, debian-repo-signer,
    hetzner-builder and ibmi-builder succeed
  - zot is healthy: 0 restarts, livez and readyz 200, and a 134 MB blob
    PUT returns 201 in the same minute as a failure
  - MinIO is not full: /export is 22% used
  - dedupe is off and there is no cacheDriver, so neither explains it
  - the failures start when two builds push concurrently

The namespace change itself STAYS. /build/** is where Shipwright already
publishes, so per-commit candidates belong there rather than mixed into
the namespaces that mirror released artifacts. That is hygiene, and the
comment now says so instead of claiming a fix.

Root cause of the intermittent push failure is open.

pipeline-bot merged commit 9affd356be into trunk

2026-08-15 11:18:53 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/pipelines!2051

No description provided.

Rows
Columns