fix: token leak in make dry runs, and OpenBao join targets for all five replicas #248

Merged

binjovi-bot merged 2 commits from fix/defects into trunk 2026-09-04 09:00:55 +00:00

Conversation 0 Commits 2 Files changed 2 +29 -9

sean commented 2026-09-04 08:57:55 +00:00

Owner

Copy link

Two unrelated defects found while auditing before the node roll. Separate commits.

1. make -n printed both tokens in full

Any dry run of any target leaked HCLOUD_TOKEN and CODEBERG_TOKEN, because:

• both are := assignments, so they decrypt at Makefile parse time on every make invocation regardless of the target; and
• five recipe lines read them as make variables, "$(HCLOUD_TOKEN)", which expands the value into the recipe text.

The @ prefix hides that during a real run. make -n ignores @ and prints the expanded text.

Both variables are already exported, so the fix is to read them from the environment: "$$HCLOUD_TOKEN" reaches the shell as $HCLOUD_TOKEN and never enters the recipe text. Five sites, identical behaviour.

Before / after on make -n preflight:

-   test -n "<the actual token>"  || (echo "ERROR: HCLOUD_TOKEN not available" ...)
+   test -n "$HCLOUD_TOKEN"       || (echo "ERROR: HCLOUD_TOKEN not available" ...)

This was found by running make -n, so both values are already exposed in that session's transcript. Rotating them is an operator action this PR does not perform.

2. OpenBao could not rejoin through two of its five replicas

The raft config declared three retry_join blocks for a five-replica StatefulSet, and its comment still described a replicas=1 deployment:

"with replicas=1 only openbao-0 exists and self-bootstraps a single-node Raft ... Scale to 3/5 to form real HA."

The scale to 5 happened; the config did not follow. A rejoining member can only join through a peer named in its own list, so openbao-3 and openbao-4 were reachable only via 0, 1 or 2 — and losing those three at once stranded them, even though the StatefulSet's own comment says five voters exist to tolerate two simultaneous losses.

Node rolls take one worker at a time, so this was latent rather than live. Worth fixing before a roll anyway: the point of five voters is surviving the unplanned second failure during a planned first one, and a three-entry join list quietly capped that.

Now one retry_join per replica. A target that is itself down is harmless — members try the list in order until one answers.

Checks

bash tests/check.sh exits 0. The rendered config parses with 5 leader_api_addr targets. Note commit 2 will roll the OpenBao StatefulSet (OrderedReady, 5 pods) — deliberately landed before the kernel roll rather than during it.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Two unrelated defects found while auditing before the node roll. Separate commits. ## 1. `make -n` printed both tokens in full Any dry run of any target leaked `HCLOUD_TOKEN` and `CODEBERG_TOKEN`, because: - both are `:=` assignments, so they decrypt at Makefile **parse** time on every `make` invocation regardless of the target; and - five recipe lines read them as **make** variables, `"$(HCLOUD_TOKEN)"`, which expands the value into the recipe **text**. The `@` prefix hides that during a real run. `make -n` ignores `@` and prints the expanded text. Both variables are already `export`ed, so the fix is to read them from the environment: `"$$HCLOUD_TOKEN"` reaches the shell as `$HCLOUD_TOKEN` and never enters the recipe text. Five sites, identical behaviour. Before / after on `make -n preflight`: ``` - test -n "<the actual token>" || (echo "ERROR: HCLOUD_TOKEN not available" ...) + test -n "$HCLOUD_TOKEN" || (echo "ERROR: HCLOUD_TOKEN not available" ...) ``` **This was found by running `make -n`, so both values are already exposed in that session's transcript. Rotating them is an operator action this PR does not perform.** ## 2. OpenBao could not rejoin through two of its five replicas The raft config declared **three** `retry_join` blocks for a **five**-replica StatefulSet, and its comment still described a `replicas=1` deployment: > "with replicas=1 only openbao-0 exists and self-bootstraps a single-node Raft ... Scale to 3/5 to form real HA." The scale to 5 happened; the config did not follow. A rejoining member can only join through a peer named in its own list, so `openbao-3` and `openbao-4` were reachable only via 0, 1 or 2 — and losing those three at once stranded them, even though the StatefulSet's own comment says five voters exist to tolerate two simultaneous losses. Node rolls take one worker at a time, so this was latent rather than live. Worth fixing **before** a roll anyway: the point of five voters is surviving the *unplanned* second failure during a planned first one, and a three-entry join list quietly capped that. Now one `retry_join` per replica. A target that is itself down is harmless — members try the list in order until one answers. ## Checks `bash tests/check.sh` exits 0. The rendered config parses with 5 `leader_api_addr` targets. Note commit 2 will roll the OpenBao StatefulSet (`OrderedReady`, 5 pods) — deliberately landed before the kernel roll rather than during it. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 2 commits 2026-09-04 08:57:55 +00:00

fix(make): stop expanding token values into recipe text ... 0b411ccffa

`make -n` printed HCLOUD_TOKEN and CODEBERG_TOKEN in full. Any dry run of
any target did, because:

  - both are `:=` assignments, so they decrypt at Makefile PARSE time on
    every make invocation regardless of the target asked for; and
  - five recipe lines read them as MAKE variables, "$(HCLOUD_TOKEN)",
    which expands the value into the recipe TEXT.

The @ prefix hides that during a real run. `make -n` ignores @ and prints
the expanded text, so both tokens landed on stdout.

Both variables are already `export`ed, so the recipes can read them from
the environment instead. "$$HCLOUD_TOKEN" reaches the shell as
$HCLOUD_TOKEN and never appears in the recipe text. Five sites changed;
the checks behave identically.

Found by running `make -n` while auditing targets for dead references, so
the values are already exposed in that session's transcript. Rotating both
is a separate operator action this commit does not perform.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

fix(openbao): give every replica a retry_join target ... c6c7b98efe

The raft config declared three retry_join blocks for a five-replica
StatefulSet, and its comment still described a replicas=1 deployment that
has not existed for a long time: "with replicas=1 only openbao-0 exists
and self-bootstraps a single-node Raft ... Scale to 3/5 to form real HA."
The scale to 5 happened; the config did not follow.

A rejoining member can only ever join through a peer named in its own
retry_join list. So openbao-3 and openbao-4 were joinable only via 0, 1
or 2, and losing those three at once stranded them — even though the
StatefulSet comment correctly says five voters exist to tolerate two
simultaneous losses.

Node rolls take one worker at a time, so this was latent rather than
live. It is worth fixing before a roll anyway: the point of five voters
is surviving the UNPLANNED second failure during a planned first one, and
a three-entry join list quietly capped that.

Now one retry_join per replica. A target that is itself down is harmless
— members try the list in order until one answers.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

binjovi-bot merged commit c6c7b98efe into trunk 2026-09-04 09:00:55 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!248

No description provided.