mirror of
https://github.com/immich-app/yucca.git
synced 2026-09-30 13:33:00 +08:00
Andy Molenda
63087f6850
* feat(ceph): import yucca-ceph ansible + terraform infrastructure
Imports the yucca-ceph Ansible tree into ansible/ceph/ and adds the
Terraform stack at tf/ that drives it. Cuts over from ansible-vault
to the hybrid secrets architecture (TF as inventory authority, 1P
as secrets store, op-inject at deploy time) in one atomic move.
Source: internal yucca-ceph working tree; fresh subtree-style
import, history not preserved. Andy continues operating sietch +
painbox post-merge; yucca-team hosts the code and reviews changes.
What it adds:
- sietch (3-node Austin, production Ceph S3 backend, untouched
by this PR)
- painbox (single-node Hetzner SX295 in Helsinki) as a second
deployable cluster
- Future clusters land by appending to clusters.auto.tfvars in
the matching environment stack (tf/deployment/<env>/ceph/) —
no per-cluster TF code required
How it works (full map: ansible/ceph/docs/architecture.md):
- tf/shared/modules/ceph-cluster renders inventory.ini variants
+ secrets.yml.tpl per cluster from clusters.auto.tfvars
- secrets.yml.tpl carries op:// refs; `op inject -f` resolves
them at deploy time from the matching yucca_tf_<env> vault
- State in OVH yucca-tf-state bucket (key ceph/<env>/<stack>/)
- 11 ADRs capture the decisions: ansible/ceph/docs/adr/
Out of scope (intentional):
- LUKS keys not yet in 1P (deferred until hybrid is stable)
- tf/shared/modules/ceph-cluster/secrets.tf.disabled is dormant;
today's 1P items via `op item create` per
ansible/ceph/docs/adding-a-cluster.md
- Talos K8s on sietch is a separate workstream
Atomicity + rollback: TF-rendered inventory + secrets-template
files are gitignored (TF generates them) and ansible-vault removal
is coupled to the op-inject path. Splitting this PR lands in a
non-bootable state — merge as one unit. The merge itself is
reversible via `git revert` until the post-merge `tf:apply` runs;
after apply, full rollback needs state restore or `tofu state mv`
(land + validate before applying).
Dev-env impact: adds opentofu + terragrunt to yucca root mise tools
plus a self-contained ansible/ceph/.mise.toml. No new commands or
prereqs for immich-side contributors who don't touch ceph or run
tf:* tasks.
Verification:
- `mise run lint` (from ansible/ceph/): 130 files, 0 warnings
- `mise run check`: 19 playbooks parse clean
- `mise run tf:plan`: succeeds; 7 expected file path-rename
replacements (3 painbox + 4 sietch). State drift from import,
no cluster-side change.
- painbox deployed 2026-04-26 on the new code path: Bookworm +
Ceph Tentacle, 15 OSDs (14 HDD + 1 SSD) up + in, mon/mgr/rgw
running. HEALTH_WARN is expected on a single-node cluster.
Post-merge: from the yucca root, `mise run tf:apply` flips the
bucket state to the new monorepo paths (the 7 renames above).
* fix(ceph): exempt ansible/ and tf/ subtrees from root prettier
The imported infrastructure subtrees enforce their own format
conventions (yamllint + ansible-lint inside ansible/ceph/; tofu fmt
inside tf/). Prettier on ansible YAML reflows long Jinja2 expressions
and shell command blocks in unwanted ways, so root prettier checks
are skipped for both subtrees.
Also reformat root README.md table column alignment to match prettier
conventions (only the imported subtrees are exempt; yucca-side files
including the root README still follow root prettier rules).
* fix(ceph): clean up secrets tmpfile after ansible-playbook exits
`ansible-play.sh` rendered the resolved secrets file via `op inject`
into a `mktemp` tmpfile, set up a `trap 'rm -f "$TMPFILE"' EXIT INT
TERM`, then `exec`'d ansible-playbook. The `exec` replaced the bash
shell entirely, so the EXIT trap never fired — every play left a
plaintext-secrets file in /tmp.
In practice this was masked because /tmp is tmpfs (RAM only on this
operator's setup), so files evaporate on reboot. But within an
operator session, files accumulated linearly with each playbook
invocation. Recent count on the import-PR session: 38 files.
Drop the `exec`. With `set -euo pipefail` already on, bash:
- propagates ansible-playbook's exit code (set -e)
- fires the EXIT trap before exiting (always)
- cleans up the tmpfile on success, failure, or signal
Verified: `CEPH_ENV=... scripts/ansible-play.sh status.yml
--syntax-check` creates and removes the tmpfile within the same
invocation — /tmp is clean before and after.
`scripts/preflight.sh` uses the same trap pattern but does not
`exec`, so its tmpfile cleanup was already correct (and the suffix
differs: `-secrets-test.yml` vs `-secrets.yml`, confirming
ansible-play.sh as the sole offender).
104 lines
3.5 KiB
YAML
104 lines
3.5 KiB
YAML
---
|
|
# RADOS benchmark — tests raw cluster I/O bypassing RGW/S3.
|
|
# Useful for isolating storage performance from S3 overhead.
|
|
#
|
|
# Usage:
|
|
# scripts/ansible-play.sh rados-bench.yml # 30s write, replicated
|
|
# scripts/ansible-play.sh rados-bench.yml -e rados_ec=true # EC 8+3 (matches RGW data pool)
|
|
# scripts/ansible-play.sh rados-bench.yml -e rados_mode=seq # sequential read
|
|
# scripts/ansible-play.sh rados-bench.yml -e rados_seconds=60
|
|
# scripts/ansible-play.sh rados-bench.yml -e rados_cleanup=false # keep data for read test
|
|
#
|
|
# Modes: write, seq (sequential read), rand (random read)
|
|
# Read modes require a prior write run with rados_cleanup=false.
|
|
|
|
- name: RADOS benchmark
|
|
hosts: ceph_nodes
|
|
become: true
|
|
gather_facts: false
|
|
|
|
vars:
|
|
rados_pool: "rados-bench"
|
|
rados_mode: write
|
|
rados_seconds: 30
|
|
rados_threads: 16
|
|
rados_cleanup: true
|
|
rados_ec: false
|
|
rados_ec_profile: "ec-k8m3-osd" # reuse the RGW EC profile (8+3)
|
|
|
|
tasks:
|
|
- name: Create EC bench pool if missing
|
|
ansible.builtin.shell: |
|
|
set -o pipefail
|
|
if ceph osd pool ls | grep -q '^{{ rados_pool }}$'; then
|
|
echo "EXISTS"
|
|
else
|
|
ceph osd pool create {{ rados_pool }} 64 erasure {{ rados_ec_profile }}
|
|
ceph osd pool set {{ rados_pool }} allow_ec_overwrites true
|
|
echo "CREATED"
|
|
fi
|
|
args:
|
|
executable: /bin/bash
|
|
when:
|
|
- inventory_hostname in groups['ceph_bootstrap']
|
|
- rados_ec | bool
|
|
register: ec_pool_create
|
|
changed_when: "'CREATED' in ec_pool_create.stdout | default('')"
|
|
|
|
- name: Create replicated bench pool if missing
|
|
ansible.builtin.shell: |
|
|
set -o pipefail
|
|
if ceph osd pool ls | grep -q '^{{ rados_pool }}$'; then
|
|
echo "EXISTS"
|
|
else
|
|
ceph osd pool create {{ rados_pool }} 32 replicated
|
|
ceph osd pool set {{ rados_pool }} size 2
|
|
ceph osd pool set {{ rados_pool }} min_size 1
|
|
echo "CREATED"
|
|
fi
|
|
args:
|
|
executable: /bin/bash
|
|
when:
|
|
- inventory_hostname in groups['ceph_bootstrap']
|
|
- not (rados_ec | bool)
|
|
register: rep_pool_create
|
|
changed_when: "'CREATED' in rep_pool_create.stdout | default('')"
|
|
|
|
- name: Show benchmark parameters
|
|
ansible.builtin.debug:
|
|
msg:
|
|
- "Node: {{ inventory_hostname }}"
|
|
- "Pool: {{ rados_pool }}"
|
|
- "Mode: {{ rados_mode }}"
|
|
- "Duration: {{ rados_seconds }}s"
|
|
- "Threads: {{ rados_threads }}"
|
|
|
|
- name: Run rados bench
|
|
ansible.builtin.command: >
|
|
rados bench -p {{ rados_pool }}
|
|
{{ rados_seconds }} {{ rados_mode }}
|
|
-t {{ rados_threads }}
|
|
--run-name {{ inventory_hostname }}
|
|
{{ '--no-cleanup' if not (rados_cleanup | bool) else '' }}
|
|
register: rados_result
|
|
changed_when: true
|
|
timeout: "{{ (rados_seconds | int * 3) + 60 }}"
|
|
|
|
- name: Display results
|
|
ansible.builtin.debug:
|
|
msg: "{{ rados_result.stdout_lines[-20:] }}"
|
|
|
|
- name: Clean up bench pool
|
|
ansible.builtin.shell: |
|
|
ceph config set mon mon_allow_pool_delete true
|
|
ceph osd pool delete {{ rados_pool }} {{ rados_pool }} \
|
|
--yes-i-really-really-mean-it
|
|
ceph config set mon mon_allow_pool_delete false
|
|
args:
|
|
executable: /bin/bash
|
|
when:
|
|
- inventory_hostname in groups['ceph_bootstrap']
|
|
- rados_cleanup | bool
|
|
- rados_mode == 'write'
|
|
changed_when: true
|