fix(openbao): unseal a rebooted member in seconds, not on a 15s tick #253

Merged

binjovi-bot merged 1 commit from fix/openbao-unseal-responsiveness into trunk

2026-09-04 15:43:36 +00:00

sean commented

2026-09-04 15:38:17 +00:00

Owner

A rebooted member comes back sealed and stays NotReady until the unseal-helper notices, because the readinessProbe reads /sys/health, which answers 503 while sealed. The helper looped on a flat 15s, so that quantum sat directly on the kernel roll green gate, which waits for all five members. Every transient failure on the unseal path cost another full 15s, since both failure branches return without retrying.

On the 2026-09-04 fleet repartition, openbao was the single thing the green gate was most often waiting on.

The loop now polls every 2s while this member is sealed and settles back to 15s once it is not.

Bounded on purpose. The fast cadence stops after two minutes of a continuous sealed spell, and a later spell gets a fresh window. Two minutes covers a reboot, where the key is present and the unseal lands immediately. A member still sealed after that has a real problem, and polling every 2s forever would not fix it while adding a Kubernetes API GET per tick.

The two log lines are throttled to one per 30s each; the first still logs immediately.

No change to the unseal logic, only to when it is attempted. POSIX sh under set -u, sh -n clean, pacing verified against a state machine fixture.

Merging rolls the openbao StatefulSet, five pods one at a time.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

A rebooted member comes back **sealed** and stays NotReady until the unseal-helper notices, because the readinessProbe reads `/sys/health`, which answers 503 while sealed. The helper looped on a flat 15s, so that quantum sat directly on the kernel roll green gate, which waits for all five members. Every transient failure on the unseal path cost another full 15s, since both failure branches return without retrying. On the 2026-09-04 fleet repartition, openbao was the single thing the green gate was most often waiting on. The loop now polls every 2s while this member is sealed and settles back to 15s once it is not. **Bounded on purpose.** The fast cadence stops after two minutes of a continuous sealed spell, and a later spell gets a fresh window. Two minutes covers a reboot, where the key is present and the unseal lands immediately. A member still sealed after that has a real problem, and polling every 2s forever would not fix it while adding a Kubernetes API GET per tick. The two log lines are throttled to one per 30s each; the first still logs immediately. No change to the unseal logic, only to when it is attempted. POSIX sh under `set -u`, `sh -n` clean, pacing verified against a state machine fixture. Merging rolls the openbao StatefulSet, five pods one at a time. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit

2026-09-04 15:38:18 +00:00

fix(openbao): unseal a rebooted member in seconds, not on a 15s tick

binjovi/ci Binjovi completed the frozen plan

Details

e741cb407e

A rebooted member comes back SEALED and stays NotReady until the unseal-helper
notices: the readinessProbe reads /sys/health, which answers 503 while sealed.
The helper's loop slept a flat 15s, so that quantum sat directly on the kernel
roll's green gate, which waits for all five members. Worse, every transient
failure on the unseal path cost a further whole 15s, because both failure
branches return without retrying: read_unseal_key coming back empty logs "no
unseal key yet" and returns, and the /sys/unseal POST is fire-and-forget behind
`|| true`. On the 2026-09-04 fleet repartition openbao was the single thing the
green gate was most often waiting on.

The loop now polls every 2s while THIS member is sealed and settles back to 15s
the moment it is not.

BOUNDED, deliberately. The fast cadence stops after two minutes of a continuous
sealed spell, and a later spell gets a fresh window. Two minutes covers the case
this is for -- a reboot, where the key is present and the unseal lands on the
first or second try. A member still sealed after two minutes has a real problem,
and polling it every 2s forever would not fix it while adding a Kubernetes API
GET per tick, since read_unseal_key reads a Secret.

The two log lines in unseal_if_sealed are now throttled to one per 30s each.
At the 2s cadence they would otherwise be 30 lines a minute for as long as the
condition lasted. The first occurrence still logs immediately.

No change to the unseal logic itself, only to when it is attempted. Written for
POSIX sh under `set -u`, as the rest of this script is: the throttle timestamps
are initialised at the top rather than on first use, and `sh -n` is clean.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

binjovi-bot merged commit 3a0b10401a into trunk

2026-09-04 15:43:36 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!253

No description provided.

Rows
Columns