feat(registry): collect garbage, and keep what cannot be rebuilt #51
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/zot-retention"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The registry has never garbage-collected: 165 GiB, 2,840 tags, 124 repos, 13 images running, 1,131 tags belonging to a deleted system. GC on at a 24h interval -- the bound is the interval, not the switch, because zot v2.1 holds ONE image-store write lock across each repo scan (imagestore.go:47, gc.go:125) and has no offline collector. Retention lands in dry-run; blob collection is not rehearsed and reclaims only true orphans. The roll's hard-coded LEGACY_IMAGES rollback digest is pinned unconditionally, since it predates almost every other tag. Seven negative controls, two of which were inert before they were right. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
This registry has never garbage-collected. By 2026-09-09 it held 165 GiB and 2,840 tags across 124 repositories, of which 13 images were running and 1,131 tags belonged to Pipelines, a system that was deleted. build/binjovi alone is 44.71 GiB in 662 tags -- one per release, kept forever. RETENTION IS ON IN DRY-RUN. Tag removal is rehearsed, not performed. Blob collection is NOT rehearsed and never was: gc.cleanRepo runs removeUnreferencedBlobs outside the dryRun guard (gc.go:155 vs :164), and because the index is not written back in a dry run, the only blobs it can reach are ones already referenced by nothing. That is the orphan set -- about 24 GiB, the gap between the bucket's 165 GiB and zot's own 140.6 GiB accounting -- and reclaiming it is the intent. THE LOCK CONCERN THAT KEPT THIS OFF IS REAL, and the guard that recorded it was right. Zot v2.1 guards its image store with ONE sync.RWMutex (imagestore.go:47) and gc.cleanRepo takes the write lock for a whole repository scan, S3 blob LIST included (gc.go:125). A pass blocks manifest requests across the WHOLE registry. There is no offline collector either -- zot's subcommands are serve, scrub, verify and verify-feature -- so the only collector available runs in the serving process. So the bound is the INTERVAL, not the switch: 24h, where the fleet scan costs about a second per repository spread by the scheduler's random delay. Hourly would be a few percent of the registry's time spent blocked, and this registry is what every node pulls from. THE POLICY, and why each part is shaped the way it is. zot's rules are not obvious from the config, so they were read from v2.1.15's source: a tag matching NO keepTags pattern is deleted, with no implicit keep (retention.go:278) -- so every policy needs a pattern that covers cosign ".sig" tags, which get no exemption in the tag loop (gc.go:395-432). A kept image that loses its signature fails `cosign verify` on the next roll. repository globs are first-match-wins (retention.go:243), so "**" is last. patterns are UNANCHORED regexes (matcher.go:24), so all of them are anchored; "v[0-9]" would otherwise also match "sha-v1abc". a patterns-only policy with no rules retains everything it matches (retention.go:207) -- the only way to pin an artifact permanently. That last one matters more than it looks. maintenance_service.py hard-codes sha256:d57a9611 as the fleet's LEGACY_IMAGES rollback target. It lives in build/node-bootc under three tags, all pushed 2026-06-04, older than almost everything else there -- so every count- or window-based rule eventually drops it and takes rollback with it. It is now pinned by an unconditional patterns-only policy, placed FIRST because candidates are grouped by the first matching tag policy. Kernel and node repositories also keep untagged manifests. Everything on the roll path is pulled by digest, and deleteUntagged there turns an aged-out tag into an unpullable image. tests/zot-storage-hygiene.sh is the contract: collection on, both delays above their floors, every policy proven against a sample signature tag rather than merely inspected, the catch-all last, the rollback pin unconditional and first, roll-path repositories keeping untagged manifests, and every pattern anchored and compilable -- an uncompilable one is a registry that does not boot. Seven negative controls, each confirmed red. Two of those controls were wrong before they were right, which is the argument for running them. A bare '"**"' anchor mutated the access-control wildcard earlier in the file instead of the retention catch-all. And breaking an explicit ".sig" pattern proved nothing, because every policy also carries "^.*$", which matches signature tags too; the control now breaks the one policy where "^.*$" is the only cover. scripts/zot-keepset-check.sh is the gate between dryRun true and false. It resolves every running pod digest, the current and previous kernel images, and the legacy rollback digest, and with --log also requires a keep decision for each. It lives in scripts/ because it needs a cluster; tests/check.sh is the cluster-free suite. Baseline before this change: 18 digests, all resolving. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76