fix(olly): let single-replica rings find themselves #14
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/memberlist-publish-notready"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
loki-0has been in CrashLoopBackOff since 2026-08-05:This is a deadlock, not a transient fault.
loki-memberlistis headless withoutpublishNotReadyAddresses, so it publishes an address only once a pod is ready. Loki runs one replica and must join its own ring to BECOME ready. After a restart the only member is not ready, DNS answers NXDOMAIN, the ring stays empty, and it can never recover.The endpoint shows it exactly:
tempo-memberlisthas the identical shape and one replica. It survives only because it has not restarted since creation — a node roll restarts it and it deadlocks the same way. Fixed both.Found while a kernel roll refused to start: the deploy green-gate fails on any CrashLoopBackOff pod, and
loki-0was the only one blocking.loki-0 has been in CrashLoopBackOff since 2026-08-05, repeating: Failed to resolve loki-memberlist: lookup loki-memberlist ... no such host error getting ingester clients: empty ring The cause is a deadlock, not a transient fault. loki-memberlist is a headless Service without publishNotReadyAddresses, so it publishes an address only after a pod is ready. Loki runs one replica and must join its own memberlist ring to BECOME ready. After any restart the only member is not ready, DNS answers NXDOMAIN, the ring stays empty, and the pod can never recover. The endpoint shows this exactly: notReadyAddresses=[loki-0], no ready addresses. Set publishNotReadyAddresses: true so a starting member can resolve itself. tempo-memberlist has the identical shape and one replica. It survives only because it has not restarted since it was created. A node roll restarts it and it would deadlock the same way, so fix both now rather than after the next roll. Found while a kernel roll refused to start: the deploy green-gate fails on any CrashLoopBackOff pod, and loki-0 was the only one.