fix(argocd): terminate a sync operation after 7200 s #192

Merged

binjovi-bot merged 1 commit from feat/argocd-sync-timeout into trunk 2026-09-02 13:23:39 +00:00

Conversation 0 Commits 1 Files changed 4 +154 -1

sean commented 2026-09-02 13:18:51 +00:00

Owner

Copy link

Why

On 2026-09-02 an automated sync of the Application registry rolled a StatefulSet to an image that no node could pull. The operation stayed in phase Running for 34 minutes. Argo CD skips auto-sync while an operation is in progress, so the revert that Binjovi released 14 minutes later could not start until the hung operation ended.

What

• argocd-cmd-params-cm/controller.sync.timeout.seconds: "7200". The default 0 means never terminate. The Binjovi deployer (builder-images binjovi-argocd-deployer v6, PR 170) terminates an older operation when a newer release starts to observe; this value is the backstop for a sync that no deployer watches.
• 7200 s equals the cold-rebuild convergence budget in scripts/run-rebuild.sh (WAIT_FLUX_TIMEOUT_SECONDS). The registry Sync hook waits for Hydra for a long time by design; a shorter value can leave a fresh cluster without a registry until a human syncs.
• GitOps activation: the controller reads the key at start, so a ConfigMap edit alone is inert. The checksum/cmd-params pod-template annotation on argocd-application-controller now holds the sha256 of the ConfigMap data (compact JSON, sorted keys). This edit, and every later edit of that ConfigMap, changes the pod template and Flux rolls the controller in the same reconcile. No manual restart. Other components keep the Helm-frozen checksum; they read no key this repository edits.
• tests/argocd-sync-timeout.sh (wired into tests/check.sh) pins the value, its string type, the env wiring, and the checksum; it prints the checksum to pin after an edit. Red on a stale checksum and on a data edit without a bump; green as committed.

Effect on rollout

One restart of argocd-application-controller-0 (single replica). In-flight syncs resume after the restart; Argo CD re-reconciles every Application once.

Verify after deploy

• kubectl -n argocd get sts argocd-application-controller -o jsonpath='{.spec.template.metadata.annotations.checksum/cmd-params}' = 17e7b264…
• kubectl -n argocd exec argocd-application-controller-0 -- env | grep ARGOCD_APPLICATION_CONTROLLER_SYNC_TIMEOUT = 7200

https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

## Why On 2026-09-02 an automated sync of the Application `registry` rolled a StatefulSet to an image that no node could pull. The operation stayed in phase Running for 34 minutes. Argo CD skips auto-sync while an operation is in progress, so the revert that Binjovi released 14 minutes later could not start until the hung operation ended. ## What - `argocd-cmd-params-cm/controller.sync.timeout.seconds: "7200"`. The default `0` means never terminate. The Binjovi deployer (builder-images `binjovi-argocd-deployer` v6, PR 170) terminates an older operation when a newer release starts to observe; this value is the backstop for a sync that no deployer watches. - 7200 s equals the cold-rebuild convergence budget in `scripts/run-rebuild.sh` (`WAIT_FLUX_TIMEOUT_SECONDS`). The registry Sync hook waits for Hydra for a long time by design; a shorter value can leave a fresh cluster without a registry until a human syncs. - GitOps activation: the controller reads the key at start, so a ConfigMap edit alone is inert. The `checksum/cmd-params` pod-template annotation on `argocd-application-controller` now holds the sha256 of the ConfigMap data (compact JSON, sorted keys). This edit, and every later edit of that ConfigMap, changes the pod template and Flux rolls the controller in the same reconcile. No manual restart. Other components keep the Helm-frozen checksum; they read no key this repository edits. - `tests/argocd-sync-timeout.sh` (wired into `tests/check.sh`) pins the value, its string type, the env wiring, and the checksum; it prints the checksum to pin after an edit. Red on a stale checksum and on a data edit without a bump; green as committed. ## Effect on rollout One restart of `argocd-application-controller-0` (single replica). In-flight syncs resume after the restart; Argo CD re-reconciles every Application once. ## Verify after deploy - `kubectl -n argocd get sts argocd-application-controller -o jsonpath='{.spec.template.metadata.annotations.checksum/cmd-params}'` = `17e7b264…` - `kubectl -n argocd exec argocd-application-controller-0 -- env | grep ARGOCD_APPLICATION_CONTROLLER_SYNC_TIMEOUT` = `7200` https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

sean added 1 commit 2026-09-02 13:18:52 +00:00

fix(argocd): terminate a sync operation after 7200 s ... d528eb71bd

On 2026-09-02 an automated sync of the Application "registry" rolled a
StatefulSet to an image that no node could pull. The operation stayed
in phase Running for 34 minutes. Argo CD skips auto-sync while an
operation is in progress, so the revert that Binjovi released 14
minutes later could not start until the hung operation ended.

Set argocd-cmd-params-cm/controller.sync.timeout.seconds to "7200".
The default 0 means "never terminate". The Binjovi deployer terminates
an older operation when a newer release starts to observe; this value
is the backstop for a sync that no deployer watches. 7200 s equals the
cold-rebuild convergence budget in scripts/run-rebuild.sh, because the
registry Sync hook waits for Hydra for a long time by design and a
shorter value can leave a fresh cluster without a registry.

The controller reads the key at start, so a ConfigMap edit alone is
inert. The pod template annotation checksum/cmd-params on the
argocd-application-controller StatefulSet now holds the sha256 of the
ConfigMap data (compact JSON, sorted keys). This edit, and every later
edit of that ConfigMap, changes the pod template and Flux rolls the
controller. No manual restart. The other components keep the
Helm-frozen checksum; they read no key this repository edits.

tests/argocd-sync-timeout.sh pins the value, the string type, the env
wiring and the checksum, and prints the checksum to pin after an edit.

Claude-Session: https://claude.ai/code/session_01LeVatedQ4sCA1u6LHjo7B5

binjovi-bot merged commit d528eb71bd into trunk 2026-09-02 13:23:39 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!192

No description provided.