chore(forgejo): move the repository volume to hcloud-volumes #232

Merged

binjovi-bot merged 1 commit from chore/forgejo-hcloud-volume into trunk

2026-09-03 21:49:10 +00:00

sean commented

2026-09-03 21:46:34 +00:00

Owner

Forgejo held the last Mayastor claim. With this change every Mayastor PVC in the cluster is gone, which is what lets a later change remove Mayastor and return 4 CPU, 2 GiB of hugepages and 2 GiB of RAM per worker.

What moved

data-forgejo-m2 on mayastor-repl2 becomes data-forgejo-hc on hcloud-volumes. The volume holds bare git repositories only, 3.12 GB in 41,489 files. Attachments and LFS live in MinIO and the database is the shared CNPG.

The cutover already happened

A PVC storageClassName is immutable, so the claim had to be recreated and the data copied by hand. docs/DEVELOPMENT-WORKFLOW.md permits kubectl for temporary local verification immediately followed by the matching commit, which is this shape.

The commit has to follow the cutover rather than precede it, because every pull request in this migration lands through Forgejo and the release train needs Forgejo up to merge and promote.

What was done, in order:

kustomize.toolkit.fluxcd.io/reconcile=disabled on the Deployment and on the forgejo-mirror-recovery CronJob, then the CronJob suspended. The annotation comes first, or the Kustomization flips suspend back within ten minutes.
The new claim applied, Forgejo scaled to zero.
rsync -aH --numeric-ids --delete from the old claim mounted read-only.
A second dry-run pass, which reported an empty delta. Both trees count 41,489 entries.
The Job deleted before scaling up, so the Hetzner volume was free to attach on the pod's node.
claimName patched with a strategic merge, which merges volumes by name, so there is no array-index brittleness.

Downtime was about three minutes, 21:42:27Z to 21:45:37Z. Verified afterwards: /api/healthz passes on cache and database, git ls-remote returns refs/heads/trunk, the web UI answers, and the pod mounts data-forgejo-hc.

Rollback

forgejo-app sets prune: false, so deleting storage.yaml does not delete the old claim. data-forgejo-m2 survives untouched, having been mounted read-only throughout, and stays the rollback until the soak ends. Reverting is scale to zero, patch claimName back, scale to one.

The Deployment stays annotated until main is confirmed to carry claimName: data-forgejo-hc. Removing the annotation while main still said data-forgejo-m2 would send Forgejo back to the stale volume.

Checks

kustomize build kubernetes/flux/infrastructure/forgejo-app renders 40 objects with no mayastor reference. bash tests/check.sh exits 0.

https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

Forgejo held the **last Mayastor claim**. With this change every Mayastor PVC in the cluster is gone, which is what lets a later change remove Mayastor and return 4 CPU, 2 GiB of hugepages and 2 GiB of RAM per worker. ## What moved `data-forgejo-m2` on `mayastor-repl2` becomes `data-forgejo-hc` on `hcloud-volumes`. The volume holds bare git repositories only, 3.12 GB in 41,489 files. Attachments and LFS live in MinIO and the database is the shared CNPG. ## The cutover already happened A PVC `storageClassName` is immutable, so the claim had to be recreated and the data copied by hand. `docs/DEVELOPMENT-WORKFLOW.md` permits `kubectl` for temporary local verification immediately followed by the matching commit, which is this shape. The commit has to follow the cutover rather than precede it, because every pull request in this migration lands through Forgejo and the release train needs Forgejo up to merge and promote. What was done, in order: 1. `kustomize.toolkit.fluxcd.io/reconcile=disabled` on the Deployment and on the `forgejo-mirror-recovery` CronJob, then the CronJob suspended. The annotation comes first, or the Kustomization flips `suspend` back within ten minutes. 2. The new claim applied, Forgejo scaled to zero. 3. `rsync -aH --numeric-ids --delete` from the old claim mounted **read-only**. 4. A second dry-run pass, which **reported an empty delta**. Both trees count 41,489 entries. 5. The Job deleted before scaling up, so the Hetzner volume was free to attach on the pod's node. 6. `claimName` patched with a strategic merge, which merges `volumes` by name, so there is no array-index brittleness. **Downtime was about three minutes**, 21:42:27Z to 21:45:37Z. Verified afterwards: `/api/healthz` passes on cache and database, `git ls-remote` returns `refs/heads/trunk`, the web UI answers, and the pod mounts `data-forgejo-hc`. ## Rollback `forgejo-app` sets `prune: false`, so deleting `storage.yaml` does not delete the old claim. `data-forgejo-m2` survives untouched, having been mounted read-only throughout, and stays the rollback until the soak ends. Reverting is scale to zero, patch `claimName` back, scale to one. The Deployment stays annotated until `main` is confirmed to carry `claimName: data-forgejo-hc`. Removing the annotation while `main` still said `data-forgejo-m2` would send Forgejo back to the stale volume. ## Checks `kustomize build kubernetes/flux/infrastructure/forgejo-app` renders 40 objects with no `mayastor` reference. `bash tests/check.sh` exits 0. https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

sean added 1 commit

2026-09-03 21:46:34 +00:00

chore(forgejo): move the repository volume to hcloud-volumes

binjovi/ci Binjovi completed the frozen plan

Details

1ae869afd4

Forgejo held the last Mayastor claim. Mayastor reserves 4 CPU, 2 GiB of
hugepages and 2 GiB of RAM on every worker, which is 20 vCPU and 20 GiB
across the fleet, to serve 740 GiB that eight workloads used. Emptying it
is what lets that reservation be reclaimed in a later change.

The volume holds bare git repositories only: 3.12 GB in 41,489 files.
Attachments and LFS are in MinIO and the database is the shared CNPG.

The live cutover happened before this commit, which the development
workflow permits for a change that cannot be applied declaratively.
`volumeClaimTemplates` and a PVC `storageClassName` are both immutable, so
the claim had to be recreated. Forgejo was frozen with
`kustomize.toolkit.fluxcd.io/reconcile=disabled`, scaled to zero, and the
data copied with `rsync -aH --numeric-ids` from the old claim mounted
read-only. A second dry-run pass reported an empty delta and both trees
count 41,489 entries. Downtime was about three minutes.

The commit has to follow the cutover rather than precede it, because every
pull request in this migration lands through Forgejo, and the release train
needs Forgejo up to merge and promote.

`storage.yaml` is replaced rather than edited so the claim name changes with
the class. `forgejo-app` sets `prune: false`, so the old `data-forgejo-m2`
claim survives this commit and stays available as a rollback until the soak
ends.

Claude-Session: https://claude.ai/code/session_01MdSbMhzabSbpG8TtP9Ur3H

binjovi-bot merged commit 1ae869afd4 into trunk

2026-09-03 21:49:10 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!232

No description provided.

Rows
Columns