mirror of https://github.com/immich-app/yucca.git synced 2026-09-30 21:37:50 +08:00

Files

T

Andy Molenda 63087f6850 feat(ceph): import yucca-ceph ansible + terraform infrastructure (#86 )

* feat(ceph): import yucca-ceph ansible + terraform infrastructure

Imports the yucca-ceph Ansible tree into ansible/ceph/ and adds the
Terraform stack at tf/ that drives it. Cuts over from ansible-vault
to the hybrid secrets architecture (TF as inventory authority, 1P
as secrets store, op-inject at deploy time) in one atomic move.
Source: internal yucca-ceph working tree; fresh subtree-style
import, history not preserved. Andy continues operating sietch +
painbox post-merge; yucca-team hosts the code and reviews changes.

What it adds:
  - sietch (3-node Austin, production Ceph S3 backend, untouched
    by this PR)
  - painbox (single-node Hetzner SX295 in Helsinki) as a second
    deployable cluster
  - Future clusters land by appending to clusters.auto.tfvars in
    the matching environment stack (tf/deployment/<env>/ceph/) —
    no per-cluster TF code required

How it works (full map: ansible/ceph/docs/architecture.md):
  - tf/shared/modules/ceph-cluster renders inventory.ini variants
    + secrets.yml.tpl per cluster from clusters.auto.tfvars
  - secrets.yml.tpl carries op:// refs; `op inject -f` resolves
    them at deploy time from the matching yucca_tf_<env> vault
  - State in OVH yucca-tf-state bucket (key ceph/<env>/<stack>/)
  - 11 ADRs capture the decisions: ansible/ceph/docs/adr/

Out of scope (intentional):
  - LUKS keys not yet in 1P (deferred until hybrid is stable)
  - tf/shared/modules/ceph-cluster/secrets.tf.disabled is dormant;
    today's 1P items via `op item create` per
    ansible/ceph/docs/adding-a-cluster.md
  - Talos K8s on sietch is a separate workstream

Atomicity + rollback: TF-rendered inventory + secrets-template
files are gitignored (TF generates them) and ansible-vault removal
is coupled to the op-inject path. Splitting this PR lands in a
non-bootable state — merge as one unit. The merge itself is
reversible via `git revert` until the post-merge `tf:apply` runs;
after apply, full rollback needs state restore or `tofu state mv`
(land + validate before applying).

Dev-env impact: adds opentofu + terragrunt to yucca root mise tools
plus a self-contained ansible/ceph/.mise.toml. No new commands or
prereqs for immich-side contributors who don't touch ceph or run
tf:* tasks.

Verification:
  - `mise run lint` (from ansible/ceph/): 130 files, 0 warnings
  - `mise run check`: 19 playbooks parse clean
  - `mise run tf:plan`: succeeds; 7 expected file path-rename
    replacements (3 painbox + 4 sietch). State drift from import,
    no cluster-side change.
  - painbox deployed 2026-04-26 on the new code path: Bookworm +
    Ceph Tentacle, 15 OSDs (14 HDD + 1 SSD) up + in, mon/mgr/rgw
    running. HEALTH_WARN is expected on a single-node cluster.

Post-merge: from the yucca root, `mise run tf:apply` flips the
bucket state to the new monorepo paths (the 7 renames above).

* fix(ceph): exempt ansible/ and tf/ subtrees from root prettier

The imported infrastructure subtrees enforce their own format
conventions (yamllint + ansible-lint inside ansible/ceph/; tofu fmt
inside tf/). Prettier on ansible YAML reflows long Jinja2 expressions
and shell command blocks in unwanted ways, so root prettier checks
are skipped for both subtrees.

Also reformat root README.md table column alignment to match prettier
conventions (only the imported subtrees are exempt; yucca-side files
including the root README still follow root prettier rules).

* fix(ceph): clean up secrets tmpfile after ansible-playbook exits

`ansible-play.sh` rendered the resolved secrets file via `op inject`
into a `mktemp` tmpfile, set up a `trap 'rm -f "$TMPFILE"' EXIT INT
TERM`, then `exec`'d ansible-playbook. The `exec` replaced the bash
shell entirely, so the EXIT trap never fired — every play left a
plaintext-secrets file in /tmp.

In practice this was masked because /tmp is tmpfs (RAM only on this
operator's setup), so files evaporate on reboot. But within an
operator session, files accumulated linearly with each playbook
invocation. Recent count on the import-PR session: 38 files.

Drop the `exec`. With `set -euo pipefail` already on, bash:

  - propagates ansible-playbook's exit code (set -e)
  - fires the EXIT trap before exiting (always)
  - cleans up the tmpfile on success, failure, or signal

Verified: `CEPH_ENV=... scripts/ansible-play.sh status.yml
--syntax-check` creates and removes the tmpfile within the same
invocation — /tmp is clean before and after.

`scripts/preflight.sh` uses the same trap pattern but does not
`exec`, so its tmpfile cleanup was already correct (and the suffix
differs: `-secrets-test.yml` vs `-secrets.yml`, confirming
ansible-play.sh as the sole offender).

2026-05-18 06:17:56 -07:00

3.1 KiB

Raw Blame History

Capacity Planning

Audience: Managers, procurement, budget planning. For hardware specs and disk layouts see hardware.md; this doc is the sizing math.

Current deployments

Cluster	Nodes	HDD × size	Raw HDD	EC-usable (8+3)	70%-full target
sietch (Austin)	3 × Dell R730xd	30 × 6 TB	~164 TiB	~119 TiB	~83 TiB
painbox (Hetzner Helsinki)	1 × SX295	14 × 22 TB	~280 TiB	~204 TiB	~143 TiB
Combined			~444 TiB	~323 TiB	~226 TiB

Each cluster also contributes ~1 TiB of SSD OSD on the boot SSDs (minor; ignored in the math above).

Sietch has ~6 empty bays across its three nodes (block.db LVs pre-created), worth +36 TB raw (~33 TiB) by populating them. No LVM or network changes needed — cheapest expansion path.

Sizing formulas

EC-usable          = raw × (k / (k + m))   = raw × 8/11 = raw × 0.727
Operational target = EC-usable × 0.70       (keep cluster below 70% full)
Raw needed         = target_data / 0.727 / 0.70

backfillfull triggers at 85% full (stops recovery); full triggers at 95% (stops writes). 70% is the conservative operational ceiling — substantial headroom for failures, rebalancing, and growth.

Ceph also consumes small amounts for index pools, RGW metadata, and the non-EC multipart-upload pool. All negligible relative to data (< 1% each at steady state).

Worked example

Target: 50 TiB of application data.

raw_needed = 50 / 0.727 / 0.70 = 98 TiB raw HDD
HDDs       = 98 TiB / 5.45 TiB = 18 drives (at 6 TB each)
Nodes      = 18 / 12 bays      = 2 nodes minimum (populated)

For 22 TB Hetzner drives the drive count is much lower (~5 drives) but you still need enough failure domains for EC — see below.

When to add drives vs. nodes

	Add drives	Add a node
When	Empty bays exist with pre-created block.db LVs	All bays populated; need more IOPS, network, or failure domains
Cost	~$30–50 per 6 TB HDD (used)	~$1,000–1,500 per fully-populated node (used)
Adds	+3.96 TiB EC-usable per drive	+47 TiB EC-usable per fully-populated R730xd
Impact	Backfill only	Backfill + CRUSH reshape + monitoring/SSH/cephadm host onboarding

Failure domain ceiling

EC 8+3 needs 11 failure domains. Austin currently uses failure_domain=osd (spreads across 30 OSDs across 3 nodes) — works today but a full-node loss degrades a large share of PGs. For host-level failure domain you need 11+ nodes minimum. Production (Yucca) will want this; dev can tolerate the weaker guarantee.

block.db sizing

Rule of thumb: block.db ≈ 4% of OSD data size.

Austin (6 TB HDDs): 240 GiB LV per HDD matches the rule; 6 LVs consume 1,440 GiB of each SSD's partition 5 (see hardware.md).
Hetzner (22 TB HDDs): 128 GiB LV is undersized against the 4% rule (would want 256–512 GiB). Acceptable for dev/benchmark use; production deployments with 22 TB HDDs should target larger block.db.

If block.db fills up, BlueStore spills metadata to the HDD data partition — OSD keeps working but small-object operations slow down. Fix: grow the block.db LV or reduce metadata density.

3.1 KiB Raw Blame History Unescape Escape