sean/olly
Watch
1
0
Fork
You've already forked olly
 0
Code Issues Pull requests Projects Releases 60 Packages Wiki Activity

fix(loki): cap WAL replay memory below the container limit #18

Merged
sean merged 1 commit from fix/loki-wal-replay-memory-ceiling into trunk 2026-08-09 23:43:41 +00:00
Conversation 0 Commits 1 Files changed 1 +2 -1
sean commented 2026-08-09 23:42:02 +00:00
Owner
Copy link

Follow-up to #17, which is confirmed working: the startupProbe let the replay run 6m21s instead of being killed at 148s. The replay then hit the REAL ceiling and was OOMKilled.

Lokis default ingester.wal.replay_memory_ceiling is 4GB; the container limit is 3Gi. The default therefore lets replay ask for more memory than the container can ever have, and a 3.6 GiB WAL gets there.

The ceiling makes the ingester flush to storage during replay rather than holding the whole WAL in memory. 1GB keeps it well below 3Gi, leaving room for the chunk cache and in-memory indexes.

Both changes are needed: the probe stops the premature kill, the ceiling stops the OOM the probe exposed.

Follow-up to #17, which is confirmed working: the startupProbe let the replay run 6m21s instead of being killed at 148s. The replay then hit the REAL ceiling and was **OOMKilled**. Lokis default `ingester.wal.replay_memory_ceiling` is 4GB; the container limit is 3Gi. The default therefore lets replay ask for more memory than the container can ever have, and a 3.6 GiB WAL gets there. The ceiling makes the ingester flush to storage during replay rather than holding the whole WAL in memory. 1GB keeps it well below 3Gi, leaving room for the chunk cache and in-memory indexes. Both changes are needed: the probe stops the premature kill, the ceiling stops the OOM the probe exposed.
fix(loki): cap WAL replay memory below the container limit
All checks were successful
pipeline/ci CI green @ ec118fadbb92
Details
ec118fadbb
 
The startupProbe from the previous commit did its job: it let the replay
run for 6m21s, where the liveness probe used to kill it at 148s. The
replay then hit the REAL ceiling and the container was OOMKilled.

Loki's default `ingester.wal.replay_memory_ceiling` is 4GB. The
container limit is 3Gi. So the default permits the replay to ask for
more memory than the container can ever have. A 3.6 GiB WAL reaches
that point.

The ceiling makes the ingester flush to storage during replay, instead
of holding the whole WAL in memory. 1GB keeps replay well below the 3Gi
limit, with room for the chunk cache and the in-memory indexes that
share it.

Both changes are necessary. The probe stops the premature kill. The
ceiling stops the OOM that the probe then exposed.
sean merged commit ec118fadbb into trunk 2026-08-09 23:43:41 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/olly!18
No description provided.