fix(sysctl-tuner): size the host route MTU for encrypted node traffic #161

Merged

binjovi-bot merged 1 commit from feat/host-route-mtu into trunk

2026-09-02 01:56:11 +00:00

sean commented

2026-09-02 01:48:20 +00:00

Owner

Problem

Cilium runs encrypt-node: true, so host-to-host traffic (nvme-tcp replication, kubelet, etcd) goes through cilium_wg0 (MTU 1355). The dhcpcd route to the private network keeps the interface MTU (1450), so the host builds 1450-byte segments; each one plus WireGuard framing is larger than 1450 and the kernel fragments it.

Measured 2026-09-01:

42afba lifetime: Ip: FragOKs 3.8 billion, ReasmFails 14k
a40389, 10 s sample during a Mayastor rebuild: 25,790 fragmented packets (2 fragments each)
host sockets: mss:1398 pmtu:1450; the control plane (NodeEncryption OptedOut) shows 0 fragments

Pod routes are fine (mtu 1305); only the host routes are wrong. Jumbo frames are not available on Hetzner cloud networks.

Change

The sysctl-tuner DaemonSet gets a route-mtu container. It reads its own CiliumNode once a minute (get only, new ServiceAccount + one-rule ClusterRole) to learn if the node encrypts, and every 5 s:

lists the host routes unfiltered (ip route show proto dhcp hides the proto word, and ip route change without it would flip the route to proto boot);
picks the DHCP gateway routes whose src is the node's own address (status.hostIP), so the public default route, on-link routes, Cilium's routes and operator static routes are never touched;
sets their mtu to the cilium_wg0 MTU on encrypting nodes, and back to the interface MTU on opted-out nodes (the control plane) or when cilium_wg0 is gone.

It reconciles because the private interface belongs to dhcpcd 10 (hc-net-ifup@enp7s0), which puts the interface MTU back on the route at every rebind, and dhcpcd has no knob for a route MTU below the interface MTU. Until the CiliumNode has been read once it does nothing; after that the last good read stands through API outages.

Wiring: hostPID + privileged so ip and sysfs come from the host through nsenter --target=1 --mount --net (the image has no iproute2); system-node-critical so node pressure cannot evict the reconciler; a memory limit and no CPU limit (a CFS quota throttles every pass and paints the throttling dashboard red for nothing).

Guard

tests/host-route-mtu-contract.sh runs the real inline script under fake nsenter/ip/curl/sleep and asserts the exact ip route change argv in ten cases: encrypting, converged, opted-out, API unreachable, API answers garbage, API lost after a good read (state stands, only two reads in 13 passes), cilium_wg0 gone, route without an mtu word, kernel state words (linkdown dead) stripped, and a clean exit within 5 s of SIGTERM. The fakes check their own calls: ip is only reachable through nsenter with the host namespaces, curl must present -f, the CA, the bearer token and this node's CiliumNode URL. The CiliumNode answer is parsed with yq -p=json (the same binary in pipeline-tools and the admission image, which has no jq); an empty or bad answer keeps the last good read. Wired into tests/check.sh; green locally and inside the admission image (2026-09-02).

Mutation-checked: 22 manifest mutations each turn the test red (widened RBAC, wrong target MTU, opted-out read as encrypting, unreadable/empty/failed read as key 0, YAML parse instead of JSON, assume-unencrypted at start, CiliumNode read every pass, 30 s route pass, dropped proto dhcp or src filter, foreground sleep, CPU limit back, priority class removed, unprivileged, no hostPID, nsenter without --net, curl without -f, kept mtu/linkdown words).

Verify after deploy

kubectl -n sysctl-tuner logs ds/sysctl-tuner -c route-mtu shows node encryption: yes and one route mtu 1355 (was 1450) line per worker; the control plane logs node encryption: no and changes nothing
workers: ss -tin on :8420 / :6443 shows pmtu:1355; Ip: FragOKs in /proc/net/snmp stops climbing
control plane: routes still mtu 1450, FragOKs still 0
Grafana namespace-resources: no throttling in sysctl-tuner

https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

## Problem Cilium runs `encrypt-node: true`, so host-to-host traffic (nvme-tcp replication, kubelet, etcd) goes through `cilium_wg0` (MTU 1355). The dhcpcd route to the private network keeps the interface MTU (1450), so the host builds 1450-byte segments; each one plus WireGuard framing is larger than 1450 and the kernel fragments it. Measured 2026-09-01: - 42afba lifetime: `Ip: FragOKs` 3.8 billion, `ReasmFails` 14k - a40389, 10 s sample during a Mayastor rebuild: 25,790 fragmented packets (2 fragments each) - host sockets: `mss:1398 pmtu:1450`; the control plane (NodeEncryption OptedOut) shows 0 fragments Pod routes are fine (`mtu 1305`); only the host routes are wrong. Jumbo frames are not available on Hetzner cloud networks. ## Change The `sysctl-tuner` DaemonSet gets a `route-mtu` container. It reads its own CiliumNode once a minute (`get` only, new ServiceAccount + one-rule ClusterRole) to learn if the node encrypts, and every 5 s: - lists the host routes unfiltered (`ip route show proto dhcp` hides the `proto` word, and `ip route change` without it would flip the route to `proto boot`); - picks the DHCP gateway routes whose `src` is the node's own address (`status.hostIP`), so the public default route, on-link routes, Cilium's routes and operator static routes are never touched; - sets their `mtu` to the `cilium_wg0` MTU on encrypting nodes, and back to the interface MTU on opted-out nodes (the control plane) or when `cilium_wg0` is gone. It reconciles because the private interface belongs to dhcpcd 10 (`hc-net-ifup@enp7s0`), which puts the interface MTU back on the route at every rebind, and dhcpcd has no knob for a route MTU below the interface MTU. Until the CiliumNode has been read once it does nothing; after that the last good read stands through API outages. Wiring: `hostPID` + privileged so `ip` and sysfs come from the host through `nsenter --target=1 --mount --net` (the image has no iproute2); `system-node-critical` so node pressure cannot evict the reconciler; a memory limit and **no CPU limit** (a CFS quota throttles every pass and paints the throttling dashboard red for nothing). ## Guard `tests/host-route-mtu-contract.sh` runs the real inline script under fake `nsenter`/`ip`/`curl`/`sleep` and asserts the exact `ip route change` argv in ten cases: encrypting, converged, opted-out, API unreachable, API answers garbage, API lost after a good read (state stands, only two reads in 13 passes), `cilium_wg0` gone, route without an `mtu` word, kernel state words (`linkdown dead`) stripped, and a clean exit within 5 s of SIGTERM. The fakes check their own calls: `ip` is only reachable through nsenter with the host namespaces, `curl` must present `-f`, the CA, the bearer token and this node's CiliumNode URL. The CiliumNode answer is parsed with `yq -p=json` (the same binary in pipeline-tools and the admission image, which has no jq); an empty or bad answer keeps the last good read. Wired into `tests/check.sh`; green locally and inside the admission image (2026-09-02). Mutation-checked: 22 manifest mutations each turn the test red (widened RBAC, wrong target MTU, opted-out read as encrypting, unreadable/empty/failed read as key 0, YAML parse instead of JSON, assume-unencrypted at start, CiliumNode read every pass, 30 s route pass, dropped `proto dhcp` or `src` filter, foreground sleep, CPU limit back, priority class removed, unprivileged, no hostPID, nsenter without `--net`, curl without `-f`, kept `mtu`/`linkdown` words). ## Verify after deploy - `kubectl -n sysctl-tuner logs ds/sysctl-tuner -c route-mtu` shows `node encryption: yes` and one `route mtu 1355 (was 1450)` line per worker; the control plane logs `node encryption: no` and changes nothing - workers: `ss -tin` on :8420 / :6443 shows `pmtu:1355`; `Ip: FragOKs` in `/proc/net/snmp` stops climbing - control plane: routes still `mtu 1450`, `FragOKs` still 0 - Grafana namespace-resources: no throttling in `sysctl-tuner` https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean added 1 commit

2026-09-02 01:48:20 +00:00

fix(sysctl-tuner): size the host route MTU for encrypted node traffic

binjovi/ci Binjovi failed the frozen plan

Details

b7fc634ca1

Cilium encrypts node-to-node traffic through cilium_wg0 (MTU 1355) but
the dhcpcd route to the private network keeps the interface MTU (1450),
so every full-size segment the host itself sends (nvme-tcp replication,
kubelet, etcd) leaves as two IP fragments. Measured on 42afba: 3.8
billion fragmented packets, 14k failed reassemblies.

Add a route-mtu container to the sysctl-tuner DaemonSet. It reads its
own CiliumNode once a minute to learn whether the node encrypts, and
every 5 s gives the DHCP gateway routes sourced from the node's own
address the cilium_wg0 MTU (encrypting nodes) or the interface MTU
(opted-out control plane). dhcpcd puts the interface MTU back at every
rebind, so this reconciles instead of running once.

Wiring: hostPID + privileged so `ip` runs on the host through nsenter
(the image has no iproute2); a one-rule ClusterRole (get ciliumnodes);
system-node-critical so node pressure cannot evict it; no CPU limit
because a CFS quota throttles every pass into a false alarm on the
throttling dashboard.

tests/host-route-mtu-contract.sh runs the real script against fake
nsenter/ip/curl/sleep and checks the exact route edits: encrypting,
converged, opted-out, API unreachable, API garbage, API lost after a
good read, cilium_wg0 gone, route without an mtu word, kernel state
words, and a clean stop on SIGTERM. Nineteen manifest mutations each
turn it red.

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean force-pushed feat/host-route-mtu from b7fc634ca1

binjovi/ci Binjovi failed the frozen plan

Details

to 0a1d98a05b

binjovi/ci Binjovi completed the frozen plan

Details

2026-09-02 01:54:52 +00:00

Compare