sean/olly
Watch
1
0
Fork
You've already forked olly
 0
Code Issues Pull requests Projects Releases 60 Packages Wiki Activity

fix(olly): give loki enough memory to stay up #16

Merged
sean merged 1 commit from fix/loki-memory-limit into trunk 2026-08-08 07:55:23 +00:00
Conversation 0 Commits 1 Files changed 1 +5 -2
sean commented 2026-08-08 07:54:38 +00:00
Owner
Copy link

loki-0 was OOMKilled 62 times against a 1Gi limit:

lastState.terminated: reason=OOMKilled exit=137
restarts: 62
limits.memory: 1Gi

The visible symptom was a memberlist resolve failure, but that was downstream — the container never lived long enough to join its own ring. (PR #14 fixed a real, separate headless-Service deadlock that would have bitten on the next restart regardless.)

Limit 1Gi -> 3Gi, request 512Mi -> 1Gi. Chunk cache, WAL and in-memory indexes exceed 1Gi under normal fleet ingest, so this is a steady-state shortfall, not a spike. Workers sit at 30-45% memory, so 3Gi has room.

Found because a kernel roll refused to start: the deploy green-gate fails on any CrashLoopBackOff pod and loki-0 was the only one.

`loki-0` was OOMKilled 62 times against a 1Gi limit: lastState.terminated: reason=OOMKilled exit=137 restarts: 62 limits.memory: 1Gi The visible symptom was a memberlist resolve failure, but that was downstream — the container never lived long enough to join its own ring. (PR #14 fixed a real, separate headless-Service deadlock that would have bitten on the next restart regardless.) Limit 1Gi -> 3Gi, request 512Mi -> 1Gi. Chunk cache, WAL and in-memory indexes exceed 1Gi under normal fleet ingest, so this is a steady-state shortfall, not a spike. Workers sit at 30-45% memory, so 3Gi has room. Found because a kernel roll refused to start: the deploy green-gate fails on any CrashLoopBackOff pod and `loki-0` was the only one.
fix(olly): give loki enough memory to stay up
All checks were successful
pipeline/ci CI green @ 038e76fd4b89
Details
038e76fd4b
 
loki-0 was OOMKilled 62 times (lastState.terminated exit=137) against a 1Gi
limit. The visible symptom was a memberlist failure, but that was downstream:
the container never lived long enough to join its ring.

Raise the limit to 3Gi and the request to 1Gi. Loki's chunk cache, WAL and
in-memory indexes exceed 1Gi under normal fleet ingest, so this is a
steady-state shortfall rather than a spike. The workers sit at 30-45% memory
use, so 3Gi has room.

The request moves to 1Gi as well so the scheduler reserves what the process
actually needs at rest, instead of packing against a 512Mi figure the workload
has never respected.

Found because a kernel roll refused to start: the deploy green-gate fails on
any CrashLoopBackOff pod, and loki-0 was the only one.
sean merged commit 038e76fd4b into trunk 2026-08-08 07:55:23 +00:00
sean deleted branch fix/loki-memory-limit 2026-08-08 07:55:23 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/olly!16
No description provided.