fix(loki): let the WAL replay finish before a probe kills the pod #17

Merged

sean merged 1 commit from fix/loki-wal-replay-startupprobe into trunk

2026-08-09 23:32:06 +00:00

sean commented

2026-08-09 23:12:00 +00:00

Owner

loki-0 crashlooped without end.

The WAL is 3.6 GiB in 124 segments. A replay of that size needs many minutes, and /ready answers 503 for all of it. So livenessProbe killed the container 148 s after each start (60 s initial delay, then 3 failures at 30 s). Each kill replayed the same WAL from the start, so the replay could never finish and the ingester could never flush to MinIO.

A startupProbe suspends BOTH the readiness and the liveness probe until it passes. This gives the replay a 20 minute budget (120 tries at 10 s). The liveness probe then starts, unchanged, for steady state.

Found while clearing DiskPressure on seanfarm-worker-069094.

loki-0 crashlooped without end. The WAL is 3.6 GiB in 124 segments. A replay of that size needs many minutes, and `/ready` answers 503 for all of it. So `livenessProbe` killed the container 148 s after each start (60 s initial delay, then 3 failures at 30 s). Each kill replayed the same WAL from the start, so the replay could never finish and the ingester could never flush to MinIO. A `startupProbe` suspends BOTH the readiness and the liveness probe until it passes. This gives the replay a 20 minute budget (120 tries at 10 s). The liveness probe then starts, unchanged, for steady state. Found while clearing DiskPressure on seanfarm-worker-069094.

sean added 1 commit

2026-08-09 23:12:00 +00:00

fix(loki): let the WAL replay finish before a probe kills the pod

pipeline/ci CI green @ dce90fa3393e

Details

dce90fa339

loki-0 crashlooped without end. The WAL is 3.6 GiB in 124 segments. A
replay of that size needs many minutes. The /ready endpoint answers 503
for all of that time. So livenessProbe killed the container 148 seconds
after each start (60 s initial delay, then 3 failures at 30 s).

Each kill replayed the same WAL from the start. So the replay could
never finish, the ingester could never flush to MinIO, and the WAL could
never get smaller.

A startupProbe suspends BOTH the readiness probe and the liveness probe
until it passes. This gives the replay a 20 minute budget (120 tries at
10 s). The liveness probe then starts, unchanged, for steady state.

sean merged commit dce90fa339 into trunk

2026-08-09 23:32:06 +00:00

sean referenced this pull request

2026-08-09 23:42:02 +00:00

fix(loki): cap WAL replay memory below the container limit #18