sean/olly
Watch
1
0
Fork
You've already forked olly
 0
Code Issues Pull requests Projects Releases 60 Packages Wiki Activity

Collect pipeline profiles on every platform node #107

Merged
binjovi-bot merged 1 commit from feat/profiler-platform-pool into trunk 2026-09-13 13:42:58 +00:00
Conversation 0 Commits 1 Files changed 3 +20 -12
sean commented 2026-09-13 13:40:22 +00:00
Owner
Copy link

Flame graphs looked "randomly unavailable" on the dashboard: build #1736 showed "No profile samples were recorded for this interval" for every profile kind, while other builds had samples.

Root cause: the pipeline-profiler DaemonSet selected nodes by hostname, and the list named the four amd64 workers that the ARM migration replaced (seanfarm-worker-07b866, -42afba, -a40389, -b6ceba). The current workers are seanfarm-worker-arm-*. Since the replacement the profiler ran on the canary only. The shared binjovi-buildkit daemon runs on seanfarm-worker-arm-a40389, so every Build on the admission and release lanes had no samples, for CPU, off-CPU and the syscall kinds alike. Bytey Builds run on the canary and had samples. That is the "random" pattern.

Change: placement follows the node-pool=platform label. Every node in the pool gets a profiler and a node roll cannot remove one silently. The canary toleration stays. tests/check.sh refuses a worker name in the manifest. docs/pipeline-profiles.md says the pool, not a count.

Verify after the deploy: kubectl get pods -n alloy -l app.kubernetes.io/name=pipeline-profiler -o wide shows one Running pod per platform node, and a new admission-lane Build shows samples on its Performance tab.

The matching binjovi change (the raw capture node list in config/runtime.exs) goes in the next binjovi pull request.

https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6

Flame graphs looked "randomly unavailable" on the dashboard: build #1736 showed "No profile samples were recorded for this interval" for every profile kind, while other builds had samples. Root cause: the `pipeline-profiler` DaemonSet selected nodes by hostname, and the list named the four amd64 workers that the ARM migration replaced (`seanfarm-worker-07b866`, `-42afba`, `-a40389`, `-b6ceba`). The current workers are `seanfarm-worker-arm-*`. Since the replacement the profiler ran on the canary only. The shared `binjovi-buildkit` daemon runs on `seanfarm-worker-arm-a40389`, so every Build on the admission and release lanes had no samples, for CPU, off-CPU and the syscall kinds alike. Bytey Builds run on the canary and had samples. That is the "random" pattern. Change: placement follows the `node-pool=platform` label. Every node in the pool gets a profiler and a node roll cannot remove one silently. The canary toleration stays. `tests/check.sh` refuses a worker name in the manifest. `docs/pipeline-profiles.md` says the pool, not a count. Verify after the deploy: `kubectl get pods -n alloy -l app.kubernetes.io/name=pipeline-profiler -o wide` shows one Running pod per platform node, and a new admission-lane Build shows samples on its Performance tab. The matching binjovi change (the raw capture node list in `config/runtime.exs`) goes in the next binjovi pull request. https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
Collect pipeline profiles on every platform node
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
f4aec4fbfa
 
The profiler DaemonSet selected nodes by hostname. The list named the
four amd64 workers that the ARM migration replaced, so after the
replacement the profiler ran on the canary only. A Build whose BuildKit
daemon ran on another worker had no samples for any profile kind, and
the dashboard reported "No profile samples were recorded" for it.

Placement now follows the node-pool label. Every node in the platform
pool gets a profiler, and a node roll does not silently remove one. A
check refuses a worker name in the manifest.

Claude-Session: https://claude.ai/code/session_01XdBRc9CHgFhAxCyHbdVtt6
binjovi-bot deleted branch feat/profiler-platform-pool 2026-09-13 13:42:58 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/olly!107
No description provided.