fix(registry): take the S3 credential out of the config zot logs #49
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/zot-config-secretless"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Zot dumps its config at info level and Sanitize() does not redact storageDriver, so this repo pinned log level warn as a workaround. That also silenced every retention and GC decision, which the 165 GiB prune needs. The credential now arrives as AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (distribution v3.0.0 falls back to the SDK credential chain when accesskey/secretkey are absent), the rendered config is secret-free, and info is safe. The stale log-level guard is rewritten to forbid the secret rather than the level, with five negative controls each confirmed red. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
Zot dumps its whole configuration at info level (pkg/api/controller.go:321, Interface("params", c.Config.Sanitize())), and Config.Sanitize() redacts only the LDAP bind password and the OpenID client secrets. It does NOT touch storage.storageDriver. So the MinIO secret key this file templated into config.json was one log-level change away from the pod log, and from Loki behind it. This repository knew, and worked around it by pinning "log": { "level": "warn" }, with a comment saying so. That workaround had a second cost nobody priced: EVERY retention and garbage-collection decision zot makes is logged at info. At warn the store cannot be pruned with any evidence of what is about to be deleted -- which is exactly what the next change needs, on a 165 GiB bucket that has never garbage-collected. So fix the cause instead of the symptom. distribution v3.0.0 uses static credentials only when BOTH accesskey and secretkey are non-empty (registry/storage/driver/s3-aws/s3.go:514); with neither set the AWS SDK falls back to its default credential chain, which reads AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Same Secret, same rotation, out of the dump. Those two template values were the ONLY interpolations in config.json -- everything else in it is a file path -- so the rendered config is now secret-free and info is safe. THE SECRET HAD TO LEARN TO PUBLISH THE KEYS. An ExternalSecret whose template declares `data` renders ONLY the keys it names: the fetched values are template variables, not automatic Secret entries. The live Secret held config.json and nothing else, so an env var reading minio_access_key would have found no such key and left the container in CreateContainerConfigError. Both are now rendered explicitly. The mount is unaffected -- it is a subPath on config.json. tests/zot-binjovi-publisher.sh pinned the warn level and refused info. That guard is rewritten rather than deleted: the invariant was never the log level, it was the secret, so tests/zot-config-secretless.sh now forbids ANY interpolation into config.json, forbids accesskey/secretkey/sessiontoken on the storageDriver, and requires both AWS_* variables to come from Secret keys the template actually renders. Raise or lower the log level freely; never put a credential back in the file. Five negative controls, each confirmed red: the credential put back in config.json, AWS_ACCESS_KEY_ID unset, a credential env set to a literal, an env reading a Secret key the template does not render, and config.json left interpolating any fetched value. The controls mutate with python, not sed. Multi-line sed expressions behave differently under BSD and GNU sed, so a control written with one can pass locally and be inert in CI -- which is the exact failure a control exists to prevent. The first draft of the third control was inert for that reason. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76