fix: keep the apiserver to kubelet log path alive and monitored #34

Merged

pipeline-bot merged 3 commits from fix/http2-kubelet-health into trunk 2026-08-28 08:41:38 +00:00

Conversation 0 Commits 3 Files changed 5 +134

sean commented 2026-08-28 08:04:06 +00:00

Owner

Copy link

Durable (IaC) form of the 2026-08-27 zombie-HTTP/2 fix, both parts already applied live and verified:

1. k3s-create.sh writes HTTP2_READ_IDLE_TIMEOUT_SECONDS=30 / HTTP2_PING_TIMEOUT_SECONDS=10 into k3s.service.env at master install, so a dead pooled apiserver->kubelet connection is ping-detected and evicted in <=40s instead of hanging log requests forever.
2. New flux-managed kubelet-path-probe CronJob (kube-system, every 5m) that GETs /api/v1/nodes//proxy/healthz for every node and fails loudly on a wedged path - the failure mode node heartbeats can never see.

https://claude.ai/code/session_01L5p8SUBvUajDvfFcCLZWux

Durable (IaC) form of the 2026-08-27 zombie-HTTP/2 fix, both parts already applied live and verified: 1. k3s-create.sh writes HTTP2_READ_IDLE_TIMEOUT_SECONDS=30 / HTTP2_PING_TIMEOUT_SECONDS=10 into k3s.service.env at master install, so a dead pooled apiserver->kubelet connection is ping-detected and evicted in <=40s instead of hanging log requests forever. 2. New flux-managed kubelet-path-probe CronJob (kube-system, every 5m) that GETs /api/v1/nodes/<node>/proxy/healthz for every node and fails loudly on a wedged path - the failure mode node heartbeats can never see. https://claude.ai/code/session_01L5p8SUBvUajDvfFcCLZWux

sean added 1 commit 2026-08-28 08:04:06 +00:00

fix: keep the apiserver to kubelet log path alive and monitored ... f6087cc801

A node replacement on 2026-08-27 dropped the remotedialer tunnel under
the API server's pooled HTTP/2 connection to one kubelet with no TCP
reset. The dead connection was never removed. Log requests to that node
hung for hours while the node stayed Ready, and two binjovi release
trains failed on required_log_evidence_timeout.

Set HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS on
the k3s server at master install, so a dead pooled connection is
detected by ping and evicted in at most 40 seconds.

Add a kubelet-path-probe CronJob that checks the exact broken path,
GET /api/v1/nodes/<node>/proxy/healthz, for every node each 5 minutes
and fails loudly when a node does not answer. Node heartbeats travel
the healthy direction and can never catch this.

Both changes are already applied to the live cluster and verified;
this commit makes them durable.

Claude-Session: https://claude.ai/code/session_01L5p8SUBvUajDvfFcCLZWux

sean added 1 commit 2026-08-28 08:07:18 +00:00

fix: pin the probe to the signed house toolchain image ... 29c0680abb

The upstream-inventory guard rejects images that are not inventoried.
Use the digest-pinned pipeline-tools image that image-gc and
sysctl-tuner already use.

Claude-Session: https://claude.ai/code/session_01L5p8SUBvUajDvfFcCLZWux

sean added 1 commit 2026-08-28 08:07:35 +00:00

chore: remove test-run bytecode cache from the tree ... 83f82ffac4

Claude-Session: https://claude.ai/code/session_01L5p8SUBvUajDvfFcCLZWux

pipeline-bot merged commit 83f82ffac4 into trunk 2026-08-28 08:41:38 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!34

No description provided.