fix(sysctl-tuner): split the host route MTU by encryption mark #165
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/host-route-mtu-table"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
#161 lowered the DHCP route to the private network from 1450 to the
cilium_wg0MTU (1355). Host sockets now build 1355-byte segments (ss -tin: 71 sockets atpmtu:1355), butIp: FragOKskept climbing on every worker (10 s sample, 2026-09-02: 42afba +30,398, b6ceba +7,422, 2a9194 +6,409). WireGuard's encrypted packets take the same route: a 1355-byte segment plus 60 bytes of WireGuard framing is 1415 bytes, larger than the route's 1355, and WireGuard sends its UDP without DF, so the kernel fragments the outer packet instead of the inner one. One route needs two MTUs.Change
Cilium gives
cilium_wg0the fwmarkMARK_MAGIC_ENCRYPT(0x0E00underMARK_MAGIC_HOST_MASK 0x0F00,bpf/lib/common.h). The reconciler now keeps:cilium_wg0MTU (ip route replace 10.0.0.0/16 via 10.0.0.1 dev enp7s0 src <node> mtu 1355 table 900);not fwmark 0xe00/0xf00 lookup 900, so every packet without the encryption mark looks up table 900 first, and WireGuard's marked packets fall through to dhcpcd's route at 1450.dhcpcd's route is not edited any more; the route #161 lowered is put back to the interface MTU (a one-time repair, kept because dhcpcd only re-asserts it at rebind). On opted-out nodes (the control plane) or when
cilium_wg0is gone, the table routes and the rule are removed. Stale table routes (an old lease's prefix, a changed gateway or MTU) are replaced or dropped. Reconciled every 30 s together with the CiliumNode read; a failed route listing edits nothing. Table 900 and rule 900 are clear of Cilium's (rules 9–112, tables 200/202/2004/2005/10+ifindex).Verified end to end on a real kernel (AlmaLinux 10 container with NET_ADMIN, iproute2 6.17, the real script with only nsenter/curl/sleep faked): after one pass
ip route get 10.0.0.9reportstable 900 … cache mtu 1355andip route get 10.0.0.9 mark 0xe00reportscache mtu 1450; opting out, losingcilium_wg0, an interface bounce and a stray rule at pref 900 all converge in one pass with no log noise.Guard
tests/host-route-mtu-contract.shruns the real inline script under fakensenter/ip/curl/sleep. The fakeipmodels the main table, table 900 and rule 900 the way the kernel renders and refuses them (a never-created table answersFIB table does not exist), records every edit as argv and asserts the exact sequence in 16 cases: fresh encrypting node, converged, #161's lowered route repaired, opted-out with and without a table, API unreachable / garbage / lost after a good read (mode stands, table rebuilt after an interface bounce),cilium_wg0gone, stray rule replaced, stale table MTU / prefix / gateway, failed listing edits nothing, route without anmtuword, kernel state words stripped, clean exit within 5 s of SIGTERM. Green locally (4 s) and inside the admission image (11 s).Mutation-checked: 37 manifest mutations each turn the test red (wrong mark, rule without
not, rule always re-added, any rule accepted, rule or table kept when opted out or whencilium_wg0is gone, table route at the interface MTU / withoutsrc, never replaced, MTU or gateway ignored, stale routes kept, failed listing read as empty, no repair / repair to the wrong MTU, filters dropped, 5 s cadence, foreground sleep, reconcile skipped, plus the wiring and CiliumNode-read mutations from #161).Verify after deploy
kubectl -n sysctl-tuner logs ds/sysctl-tuner -c route-mtu: workers logdhcp route mtu 1450 (was 1355),table 900: 10.0.0.0/16 … mtu 1355,rule 900: …; the control plane logsnode encryption: noand nothing elseip ruleshows900: not from all fwmark 0xe00/0xf00 lookup 900;ip route get <peer>showstable 900 … mtu 1355,ip route get <peer> mark 0xe00showsmtu 1450Ip: FragOKsin/proc/net/snmpstops climbing on every workersysctl-tunerhttps://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5