sean/builder-images
Watch
1
0
Fork
You've already forked builder-images
 0
Code Issues Pull requests Projects Releases 144 Packages Wiki Activity

perf(node-bootc): take 40 seconds off every reboot #200

Merged
binjovi-bot merged 1 commit from feat/fast-boot into trunk 2026-09-04 15:19:46 +00:00
Conversation 0 Commits 1 Files changed 1 +29 -12
sean commented 2026-09-04 14:29:36 +00:00
Owner
Copy link

Measured on a fresh seanfarm47 node: userspace boot 44.7s, and 57.9s on a rebooted node with more to restore. Three things in the image account for most of it, and none does work.

cloud-init — 31.5s, on the critical chain

systemd-analyze critical-chain k3s-agent.service runs through network-online.target ← cloud-init.service ← cloud-init-local.service +31.482s. The journal shows why:

url_helper.py[ERROR]: Timed out, no response from urls: ['http://169.254.169.254/hetzner/v1/metadata/instance-id']
dhcpcd[846]: dummy0: soliciting a DHCP lease
dhcpcd[846]: timed out                                   <- 30 seconds later
DataSourceHetzner.py[ERROR]: Bailing, DHCP Exception

It DHCPs on dummy0 — an interface cloud-init itself created on an earlier boot (/etc/NetworkManager/system-connections/cloud-init-dummy0.nmconnection) — and can never reach metadata before the real NIC is up. The sshd step in this file has documented since 2026-06-13 that the datasource never resolves and that both SSH keys are baked via tmpfiles.d instead. Hostname comes from k3s config.yaml; NetworkManager carries a static DHCP profile. Nothing consumes cloud-init. All four units are now masked; sshd.service Wants=sshd-keygen.target independently, so host keys are unaffected.

loglevel=7 → 4 — ~12s

The serial console runs at 115200 baud, ~11.5 KB/s, and a loglevel=7 boot printed 133 KB of kernel messages to it. The console loglevel gates only what is printed; the ring buffer, journald and pstore capture everything regardless, so the seanfarm33 panic forensics lose nothing.

fwupd — 20s of contention

A firmware-update daemon on a Hetzner VM. Masked with its refresh timer.

What is not a cost

Shutdown: 8s from systemctl reboot to the last journal line. The ~120s a node spends down decomposes as 15s reboot delay, 8s shutdown, ~30s Hetzner firmware, 45-60s userspace boot. This PR attacks the last; sean/pipelines takes the first.

Expected: userspace boot ~45-60s → ~15-20s. Verify after the next kernel roll with systemd-analyze blame on any node.

Heredocs balanced, scripts/lint-kernel-cache.sh passes.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Measured on a fresh seanfarm47 node: **userspace boot 44.7s**, and 57.9s on a rebooted node with more to restore. Three things in the image account for most of it, and none does work. ## cloud-init — 31.5s, on the critical chain `systemd-analyze critical-chain k3s-agent.service` runs through `network-online.target ← cloud-init.service ← cloud-init-local.service +31.482s`. The journal shows why: ``` url_helper.py[ERROR]: Timed out, no response from urls: ['http://169.254.169.254/hetzner/v1/metadata/instance-id'] dhcpcd[846]: dummy0: soliciting a DHCP lease dhcpcd[846]: timed out <- 30 seconds later DataSourceHetzner.py[ERROR]: Bailing, DHCP Exception ``` It DHCPs on **`dummy0`** — an interface cloud-init itself created on an earlier boot (`/etc/NetworkManager/system-connections/cloud-init-dummy0.nmconnection`) — and can never reach metadata before the real NIC is up. The sshd step in this file has documented since 2026-06-13 that the datasource never resolves and that **both SSH keys are baked via tmpfiles.d instead**. Hostname comes from k3s `config.yaml`; NetworkManager carries a static DHCP profile. Nothing consumes cloud-init. All four units are now masked; `sshd.service` `Wants=sshd-keygen.target` independently, so host keys are unaffected. ## `loglevel=7` → `4` — ~12s The serial console runs at 115200 baud, ~11.5 KB/s, and a `loglevel=7` boot printed **133 KB** of kernel messages to it. The console loglevel gates only what is *printed*; the ring buffer, journald and pstore capture everything regardless, so the seanfarm33 panic forensics lose nothing. ## fwupd — 20s of contention A firmware-update daemon on a Hetzner VM. Masked with its refresh timer. ## What is *not* a cost Shutdown: **8s** from `systemctl reboot` to the last journal line. The ~120s a node spends down decomposes as 15s reboot delay, 8s shutdown, ~30s Hetzner firmware, 45-60s userspace boot. This PR attacks the last; `sean/pipelines` takes the first. **Expected: userspace boot ~45-60s → ~15-20s.** Verify after the next kernel roll with `systemd-analyze blame` on any node. Heredocs balanced, `scripts/lint-kernel-cache.sh` passes. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
perf(node-bootc): take 40 seconds off every reboot
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Details
2ea31b194d
 
Measured on a fresh seanfarm47 node with systemd-analyze: userspace boot
was 44.7s, and on a rebooted node with more to restore, 57.9s. Three
things in the image account for most of it, and none of them does work.

cloud-init: 31.5s, ON THE CRITICAL CHAIN to k3s-agent via
network-online.target. Its init-local stage tried DHCP on dummy0 — an
interface cloud-init itself had created on an earlier boot — waited the
full 30s solicit timeout, then bailed with "DataSourceHetzner ... failed".
The sshd step in this file has documented since 2026-06-13 that the
datasource never resolves and that both SSH keys are BAKED via tmpfiles.d
instead; hostname comes from k3s config.yaml and NetworkManager carries a
static DHCP profile. Nothing consumes cloud-init. All four units are now
masked and the dead 99-hetzner.cfg is no longer written. sshd host keys are
unaffected: sshd.service Wants=sshd-keygen.target, which is independent.

loglevel=7 -> 4: the serial console runs at 115200 baud, ~11.5 KB/s, and a
loglevel=7 boot printed 133 KB of kernel messages to it — about 12s spent
waiting on a UART. The console loglevel gates only what is PRINTED; the
ring buffer, journald and pstore capture everything regardless, so the
seanfarm33 panic forensics lose nothing.

fwupd: a firmware-update daemon on a Hetzner VM, 20s of boot-time
contention. Masked with its refresh timer.

Shutdown was measured too and is NOT a cost: 8s from `systemctl reboot` to
the last journal line. The ~120s a node spends down is 15s reboot delay,
8s shutdown, ~30s Hetzner firmware, and 45-60s of userspace boot. This
commit attacks the last of those; sean/pipelines takes the first.

Expected: userspace boot from ~45-60s to ~15-20s. Verify after the next
kernel roll with `systemd-analyze` and `systemd-analyze blame` on any node.

Verified: all heredocs balanced, scripts/lint-kernel-cache.sh passes.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/builder-images!200
No description provided.