immich-app/yucca
1
0
Fork 0
mirror of https://github.com/immich-app/yucca.git synced 2026-09-30 13:33:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs(ceph): drop painbox references (#201)

Browse Source
This commit is contained in:
Andy Molenda
2026-06-26 08:19:40 -07:00
committed by GitHub
parent 97b4514644
commit e6335c9715
16 changed files with 78 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -56,7 +56,7 @@ library + per-service charts). See [`kubernetes/README.md`](./kubernetes/README.
| Start here | Path | Purpose |
| ------------------------------------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`ansible/ceph/README.md`](./ansible/ceph/README.md) | `ansible/ceph/` | Ansible automation for Ceph clusters (sietch, painbox). Deploys + operates via cephadm on bare-metal and Hetzner. |
| [`ansible/ceph/README.md`](./ansible/ceph/README.md) | `ansible/ceph/` | Ansible automation for Ceph clusters (sietch). Deploys + operates via cephadm on bare-metal and Hetzner. |
| [`ansible/talos/README.md`](./ansible/talos/README.md) | `ansible/talos/` | Talos K8s as libvirt VMs on the Ceph hypervisors. Ansible provisions the substrate + VMs; TF renders inventory and bootstraps the cluster. |
| [`tf/README.md`](./tf/README.md) | `tf/` | Terraform/OpenTofu authority for cluster identity, 1P secret items, rendered Ansible inventories. Terragrunt multi-env (`deployment/<env>/<stack>/`). |
| [`kubernetes/README.md`](./kubernetes/README.md) | `kubernetes/` | Flux GitOps surface (apps/components/flux/bootstrap) for the Talos K8s cluster. Per-app HelmReleases over the in-repo `charts/`; mirrored locally by Tilt. |
ansible/ceph/CONTRIBUTING.md
+6 -7
View File
@@ -23,20 +23,20 @@ manage.
### SSH setup
The `ansible-iac` SSH keys live in `yucca_tf_dev` as items
`SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` and `PAINBOX_CEPH_ANSIBLE_IAC_SSH_KEY`
The `ansible-iac` SSH keys live in `yucca_tf_dev` as items like
`SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY`
(see [ADR-010](docs/adr/010-ssh-keys-in-1password.md) for rationale).
**First-time workstation setup:**
```bash
cd yucca/ansible/ceph
scripts/install-ssh-keys.sh # op read → ~/.ssh/id_ed25519_{sietch,painbox}
scripts/install-ssh-keys.sh # op read → ~/.ssh/id_ed25519_sietch
```
The script is idempotent and refuses to overwrite an existing key whose
fingerprint doesn't match 1P. Keys land as `~/.ssh/id_ed25519_sietch` +
`~/.ssh/id_ed25519_painbox` (private + `.pub` both 0600/0644).
fingerprint doesn't match 1P. Keys land as `~/.ssh/id_ed25519_sietch`
(private + `.pub` both 0600/0644).
**Jump hosts / proxies** belong in your personal `~/.ssh/config`, not in
this repo.
@@ -84,7 +84,6 @@ It points to an **inventory file** (not a directory):
```bash
# Inline prefix — required for `mise run` invocations:
CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run preflight
CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run status
```
**`export CEPH_ENV=...` does NOT work with `mise run`.** mise's `[env]`
@@ -98,7 +97,7 @@ For multiple commands against the same cluster, set a local (non-exported)
shell variable and inline-prefix each invocation:
```bash
CE=inventories/painbox-ceph.dev.hel.htz/inventory.ini
CE=inventories/sietch-ceph.staging.austin.int/inventory.ini
CEPH_ENV=$CE mise run preflight
CEPH_ENV=$CE mise run status
CEPH_ENV=$CE mise run deploy
ansible/ceph/README.md
+2 -8
View File
@@ -8,7 +8,6 @@ scaffolding are provisioned from `yucca/tf/` (see `../../tf/`).
| Cluster | Domain | Location | Hardware | Nodes |
|---------|--------|----------|----------|-------|
| **sietch** | `dev.austin.int.futo.cloud` | Austin DC | Dell R730xd | 3 |
| **painbox** | `dev.hel.htz.futo.cloud` | Hetzner Helsinki | SX295 | 1 |
Clusters are declared in `yucca/tf/deployment/staging/ceph/clusters.auto.tfvars`;
`tofu apply` renders `inventories/<cluster>/inventory.ini` and
@@ -30,14 +29,9 @@ graph TB
S[samara<br/>MON+MGR+OSD+RGW]
end
subgraph "Hetzner Helsinki"
P[painbox-ceph-evelyn<br/>MON+MGR+OSD+RGW]
end
A -->|SSH| L
A -->|SSH| W
A -->|SSH| S
A -->|SSH| P
```
See [docs/architecture.md](docs/architecture.md) for role dependencies,
@@ -134,12 +128,12 @@ Inventory scaffolding + secret-item provisioning live in `yucca/tf/` — run
| [docs/adding-a-cluster.md](docs/adding-a-cluster.md) | Developers -- inventory setup, secrets |
| [docs/secrets.md](docs/secrets.md) | Developers/ops -- 1Password integration |
| [docs/naming.md](docs/naming.md) | Everyone -- hostname and inventory naming |
| [docs/hardware.md](docs/hardware.md) | Ops/procurement -- R730xd vs SX295 specs |
| [docs/hardware.md](docs/hardware.md) | Ops/procurement -- node specs and hardware shapes |
| [docs/s3-integration.md](docs/s3-integration.md) | App developers -- endpoints, boto3, certs |
| [docs/security-model.md](docs/security-model.md) | InfoSec -- encryption, users, firewall |
| [docs/capacity-planning.md](docs/capacity-planning.md) | Managers -- costs, formulas, growth |
| [docs/troubleshooting.md](docs/troubleshooting.md) | SRE/on-call -- symptom/diagnosis/fix |
| [docs/runbooks/](docs/runbooks/) | Ops -- add/replace node, replace disk, rotate certs/secrets/SSH/SA token, remote hands, painbox reprovision, bad-tofu-apply recovery, backup/restore |
| [docs/runbooks/](docs/runbooks/) | Ops -- add/replace node, replace disk, rotate certs/secrets/SSH/SA token, remote hands, bad-tofu-apply recovery, backup/restore |
| [docs/adr/](docs/adr/) | Everyone -- architecture decision records |
## Known Limitations
ansible/ceph/docs/adding-a-cluster.md
+7 -9
View File
@@ -36,7 +36,6 @@ of datacenter or environment.
Existing examples:
- `sietch-ceph.dev.austin.int/` — Austin DC, internal network, dev
- `painbox-ceph.dev.hel.htz/` — Hetzner Helsinki, dev
Future environments land as siblings: `*-ceph.staging.<dc>.<provider>/`,
`*-ceph.prod.<dc>.<provider>/`.
@@ -48,7 +47,7 @@ Future environments land as siblings: `*-ceph.staging.<dc>.<provider>/`,
The engineer adding the cluster picks the name. Conventions and constraints
live in [docs/naming.md](naming.md#cluster-naming). Quick summary:
- **Convention:** Dune-themed (existing: `sietch`, `painbox`). Not enforced.
- **Convention:** Dune-themed (existing: `sietch`). Not enforced.
- **Constraints:** lowercase, short (6–10 chars ideal), no dashes or dots,
unique within the `yucca_tf_*` item namespace, not already a key in
`clusters.auto.tfvars`.
@@ -66,7 +65,6 @@ Working example for a hypothetical `mesa` cluster at Hetzner Falkenstein:
```hcl
clusters = {
sietch = { ... }
painbox = { ... }
mesa = {
domain = "dev.fsn.htz.futo.cloud"
@@ -126,10 +124,11 @@ Hand-maintained, committed. Copy the closer existing analogue as a starting
point:
- **Bare-metal cluster:** copy from `sietch-ceph.dev.austin.int/group_vars/all/vars.yml`
- **Hetzner/single-NIC cluster:** copy from `painbox-ceph.dev.hel.htz/group_vars/all/vars.yml`
- **Hetzner/single-NIC cluster:** start from the sietch vars and adjust for
the NVMe-RAID shape (public /32, no bond/ProxyJump, installimage-owned LVM).
```bash
cp inventories/painbox-ceph.dev.hel.htz/group_vars/all/vars.yml \
cp inventories/sietch-ceph.dev.austin.int/group_vars/all/vars.yml \
inventories/mesa-ceph.dev.fsn.htz/group_vars/all/vars.yml
```
@@ -280,8 +279,8 @@ the template, SSH reachable, Python 3 on targets.
### 9. Deploy
For Hetzner installimage clusters, run the installimage flow first
(out-of-band; see [runbooks/painbox-reprovision.md](runbooks/painbox-reprovision.md)
for the pattern). For Austin bare-metal clusters, run `provision.yml`
(out-of-band: reboot into rescue mode, run `installimage/autosetup` plus
the op-injected `post-install.sh`). For Austin bare-metal clusters, run `provision.yml`
first (boot into the live image, then `scripts/ansible-play.sh
provision.yml -e confirm_wipe=true` with
`CEPH_ENV=.../inventory-provision.ini`). Then:
@@ -323,7 +322,7 @@ CEPH_ENV=inventories/mesa-ceph.dev.fsn.htz/
Default is set in `.mise.toml` (`sietch` in dev). Override per-command:
```bash
CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run status
CEPH_ENV=inventories/sietch-ceph.dev.austin.int/inventory.ini mise run status
```
`scripts/ansible-play.sh` derives the secrets template path from `CEPH_ENV`
@@ -375,4 +374,3 @@ trap-cleaned on exit.
- [scripts.md](scripts.md) — wrapper reference
- [ADR-009](adr/009-tf-first-op-inject-over-vault-password-sh.md) — why TF is authoritative
- [ADR-010](adr/010-ssh-keys-in-1password.md) — why SSH keys live in 1P
- [runbooks/painbox-reprovision.md](runbooks/painbox-reprovision.md) — Hetzner-specific reprovisioning pattern
ansible/ceph/docs/architecture.md
+18 -23
View File
@@ -19,13 +19,11 @@ flowchart LR
ONEP[("1Password org<br/>yucca_tf · yucca_tf_dev · ...")]
S3[("OVH S3<br/>yucca-tf-state bucket")]
SIETCH["Sietch · Austin DC<br/>3× Dell R730xd"]
PAINBOX["Painbox · Hetzner Helsinki<br/>1× SX295"]
OP -->|edits| YUCCA
OP -->|reads/writes secrets| ONEP
OP -->|TF state I/O| S3
OP -->|SSH ansible-iac| SIETCH
OP -->|SSH ansible-iac| PAINBOX
```
The yucca monorepo is the single source of truth for cluster identity and
@@ -40,8 +38,8 @@ External dependencies are minimal and explicit:
- **OVH S3** — `yucca-tf-state` bucket at `s3.eu-west-par.io.cloud.ovh.net`.
Project-keyed (`ceph/<env>/<stack>/terraform.tfstate`) so multiple stacks
share the bucket without collision.
- **Hardware** — Austin colo for sietch (Dell R730xd × 3, single 10G bond);
Hetzner Helsinki for painbox (SX295 × 1). Detail in [hardware.md](hardware.md).
- **Hardware** — Austin colo for sietch (Dell R730xd × 3, single 10G bond).
Detail in [hardware.md](hardware.md).
---
@@ -59,7 +57,7 @@ its environment from the same source — directory layout — so dev / staging
| Ansible inv | `inventories/<cluster>-ceph.dev.<dc>.<provider>/` | `inventories/<cluster>-ceph.staging.<dc>.<provider>/` | `inventories/<cluster>-ceph.prod.<dc>.<provider>/` |
| mise default | `CEPH_ENV=...sietch-ceph.dev.austin.int/inventory.ini` | overridden via env at invocation | overridden via env at invocation |
Today the only deployed environment is dev (sietch + painbox). Adding
Today the only deployed environment is dev (sietch). Adding
staging/prod is purely additive: create the matching `tf/deployment/<env>/ceph/`
directory, populate `clusters.auto.tfvars`, and the same module + Ansible
roles + mise tasks work unchanged. The state backend key path, 1P vault
@@ -196,8 +194,8 @@ Operator-declared names are excluded from the pool to prevent collisions
within a cluster. Adding hosts at the tail is safe — existing positions
keep their names across applies.
Painbox today demonstrates this: its single host has no `name`, so TF
auto-picked `evelyn` → hostname `painbox-ceph-evelyn`.
A host declared with no `name` demonstrates this: TF auto-picks a stable
word (e.g. `evelyn`) → hostname `<cluster>-ceph-evelyn`.
### Rendered artifacts (gitignored)
@@ -269,7 +267,7 @@ Rotation procedure: [docs/runbooks/rotate-sa-token.md](runbooks/rotate-sa-token.
### Item categories and naming
Per cluster, the following items live in the cluster's `vault` (currently
`yucca_tf_dev` for both sietch and painbox):
`yucca_tf_dev` for sietch):
| Category | Item title pattern | Field consumed |
|------------|---------------------------------------------------------|----------------------|
@@ -374,7 +372,7 @@ flowchart TB
P2["Phase 2 · bootstrap.yml<br/><i>cephadm bootstrap on first node</i>"]
P3["Phase 3 · join.yml<br/><i>ceph orch host add for remaining nodes</i>"]
P4["Phase 4 · placement.yml<br/><i>MON/MGR placement calculation</i>"]
P45["Phase 4.5 · lvm-setup.yml<br/><i>ensure block.db VGs/LVs exist (sietch-shape only;<br/>painbox skips — LVM owned by installimage post-install)</i>"]
P45["Phase 4.5 · lvm-setup.yml<br/><i>ensure block.db VGs/LVs exist (sietch-shape only;<br/>NVMe-RAID shape skips — LVM owned by installimage post-install)</i>"]
P5["Phase 5 · osds.yml<br/><i>render osd-spec.yml.j2 → ceph orch apply osd<br/>(cephadm provisions LUKS + LVM internally)</i>"]
P55["Phase 5.5 · crush-rules.yml<br/><i>replicated_hdd / replicated_ssd rules</i>"]
P575["Phase 5.75 · rgw.yml<br/><i>EC pools, realm/zone, TLS, S3 user</i>"]
@@ -403,16 +401,12 @@ inventories/
sietch-ceph-lawson.yml
sietch-ceph-samara.yml
installimage/ Hetzner installimage assets (sietch n/a)
painbox-ceph.dev.hel.htz/ Hetzner dev cluster
inventory.ini TF-generated, gitignored
inventory-destroy.ini TF-generated, gitignored
secrets.yml.tpl TF-generated, gitignored
group_vars/all/vars.yml committed
host_vars/painbox-ceph-evelyn.yml committed
installimage/ post-install.sh.tpl (op-injected)
```
Hetzner NVMe-RAID clusters follow the same layout, adding an
`installimage/` directory with a `post-install.sh.tpl` (op-injected) that
owns LVM setup.
`host_vars/*.yml` is committed because per-node hardware facts (bond_ip, SAS
expander paths, SSD PHY positions, HDD-to-block.db mappings) are stable
inventory truth — not operator preference. The `.local.yml` suffix is
@@ -500,7 +494,7 @@ Both are overridable per-invocation:
```bash
TF_STACK_DIR=tf/deployment/staging/ceph mise run tf:plan
CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run status
CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run status
```
---
@@ -639,7 +633,7 @@ sequenceDiagram
```mermaid
flowchart TB
SIETCH["Sietch prep:<br/>provision_host/disks.yml partitions SSDs<br/>then ceph_deploy/lvm-setup.yml<br/><i>creates VG + db-slot LVs on each SSD's partition 5</i>"]
PAINBOX["Painbox prep:<br/>installimage post-install.sh<br/><i>NVMe RAID-1 → vg0 → db-slot0..13 + ssd-osd LVs</i>"]
NVMERAID["NVMe-RAID prep:<br/>installimage post-install.sh<br/><i>NVMe RAID-1 → vg0 → db-slot0..13 + ssd-osd LVs</i>"]
SPEC["ceph_deploy/osds.yml renders<br/>templates/osd-spec.yml.j2 → /etc/ceph/osd-spec.yml<br/><i>one document per host; paths from host_vars</i>"]
APPLY["ceph orch apply osd -i /etc/ceph/osd-spec.yml<br/><i>cephadm: discover disks, LUKS-format, LVM, deploy daemons</i>"]
POLL["Wait for cephadm to provision<br/><i>poll num_osds until expected count reached</i>"]
@@ -648,7 +642,7 @@ flowchart TB
REWEIGHT["Safety net: fix any reweight=0 OSDs"]
SIETCH --> SPEC
PAINBOX --> SPEC
NVMERAID --> SPEC
SPEC --> APPLY --> POLL --> UP --> UNSET --> REWEIGHT
```
@@ -657,7 +651,8 @@ flowchart TB
Earlier versions of this role iterated `cephadm ceph-volume lvm create`
per disk and composed `/dev/disk/by-path/...` paths from host_vars
(`sas_path_prefix` + `path_phy`). That assumed sietch's SAS expander
topology and broke on painbox's PCI-ATA disks plus LV-backed SSD OSD.
topology and broke on the NVMe-RAID shape's PCI-ATA disks plus LV-backed
SSD OSD.
The current flow renders a cephadm OSD service spec from per-host data
and applies it via `ceph orch apply osd -i`. Cephadm handles device
@@ -675,7 +670,7 @@ with two shape branches:
- **Sietch** (`sas_path_prefix` defined): data path =
`/dev/disk/by-path/{{ sas_path_prefix }}-{{ path_phy }}-lun-0`; SSD
OSD = partition on the SAS-attached SSD via `path_phy + partition`.
- **Painbox** (`sas_path_prefix` undefined): data path =
- **NVMe-RAID shape** (`sas_path_prefix` undefined): data path =
`/dev/disk/by-path/{{ path_phy }}` (operator authors the full PCI-ATA
identifier in host_vars); SSD OSD = LV via the `lv` field
(`/dev/{{ lv }}`).
@@ -773,7 +768,7 @@ This prevents:
- Overwriting the marker with a stale `provisioned_at` timestamp
The marker filename (`ceph-provisioned.json`) is project-scoped, not
cluster-scoped — every Ceph cluster (sietch, painbox, future) writes the
cluster-scoped — every Ceph cluster (sietch, future) writes the
same filename. The marker's *contents* identify which cluster + host the
machine belongs to.
ansible/ceph/docs/capacity-planning.md
-2
View File
@@ -8,8 +8,6 @@ disk layouts see [hardware.md](hardware.md); this doc is the sizing math.
| Cluster | Nodes | HDD × size | Raw HDD | EC-usable (8+3) | 70%-full target |
|---|---|---|---|---|---|
| sietch (Austin) | 3 × Dell R730xd | 30 × 6 TB | ~164 TiB | ~119 TiB | ~83 TiB |
| painbox (Hetzner Helsinki) | 1 × SX295 | 14 × 22 TB | ~280 TiB | ~204 TiB | ~143 TiB |
| **Combined** | | | **~444 TiB** | **~323 TiB** | **~226 TiB** |
Each cluster also contributes ~1 TiB of SSD OSD on the boot SSDs (minor;
ignored in the math above).
ansible/ceph/docs/hardware.md
+17 -47
View File
@@ -9,14 +9,16 @@ For where this fits in the tool mesh, see [architecture.md](architecture.md).
## Network topology
Both clusters use a **single-network** design today — public and cluster
Clusters use a **single-network** design today — public and cluster
traffic share one subnet. Production will separate them; see
[security-model.md](security-model.md) for the threat-model implications.
| Cluster | Subnet | Connection |
|----------|-------------------|------------------------------------------------------------------------|
| sietch | `10.10.10.0/24` | 2× 10GbE bonded active-backup (eno1 + eno2) per node; private switch |
| painbox | public /32 | Single 1GbE, direct SSH (no bond, no ProxyJump) |
A Hetzner NVMe-RAID host would attach over a public /32 with a single NIC
and direct SSH (no bond, no ProxyJump).
Per-node connection IPs (`bond_ip`) are declared in
`tf/deployment/staging/ceph/clusters.auto.tfvars` and rendered by TF into the
@@ -52,34 +54,14 @@ resolution, dashboard URL construction).
| 5 | ~1.4 TB | Ceph block.db LVs (LVM VG) |
| 6 | ~2 TB | SSD OSD data (ceph-volume) |
## painbox -- Hetzner SX295 (Helsinki)
## Hetzner NVMe-RAID shape (e.g. SX295)
| Component | Spec |
|-----------|------|
| Chassis | Hetzner SX295 storage server |
| CPU | AMD EPYC 7502P (32C/64T) |
| RAM | 128 GB DDR4 ECC |
| Boot NVMe | 2x Samsung 7.68TB (installimage RAID-1, vg0) |
| HDD OSDs | 14x Seagate Exos X22 22TB SATA |
| SSD OSD | 1x ~4.4TB LV on vg0 (NVMe remainder) |
| Block.db | 14x 128GB LVs on vg0 |
| SATA | 3 onboard controllers (8+2+4 ports = 14 total) |
| Network | Single 1GbE, direct SSH (no bond, no ProxyJump) |
| Boot | BIOS (Hetzner standard) |
| OS | Debian 12 Bookworm (Hetzner installimage) |
| Provisioning | Rescue mode → `installimage/autosetup` + `post-install.sh` |
### NVMe layout (vg0 on md1)
| LV | Size | Use |
|----|------|-----|
| swap | 32 GB | Swap |
| root | 100 GB | / |
| var | 200 GB | /var |
| varlog | 50 GB | /var/log |
| db-slot0..13 | 14x 128 GB | Block.db per HDD OSD |
| ssd-osd | ~4.4 TB | SSD OSD data |
| (reserve) | ~512 GB | Future expansion |
The roles also support a Hetzner-style single-box shape for future
hosting-provider clusters: NVMe boot drives in installimage RAID-1
(`vg0`), HDD OSDs over onboard SATA with per-HDD block.db LVs on the NVMe
VG, and a single LV-backed SSD OSD carved from the NVMe remainder.
Provisioning is rescue mode → `installimage/autosetup` + `post-install.sh`,
booting Debian 12 Bookworm.
> **Why Bookworm and not Trixie:** upstream Ceph Tentacle's Debian
> repository at `download.ceph.com/debian-tentacle/dists/` publishes only
@@ -87,24 +69,12 @@ resolution, dashboard URL construction).
> autosetup `IMAGE` line MUST select a Bookworm tarball until upstream
> ships Trixie packages.
## Comparison
| | sietch (per node) | painbox (single node) |
|---|---|---|
| HDD OSD count | 8-12 | 14 |
| SSD OSD count | 2 | 1 |
| block.db per HDD | 240 GB | 128 GB |
| Total raw HDD | 48-72 TB | 308 TB |
| Device path format | `/dev/disk/by-path/sas-exp*-phy*-lun-0` | `/dev/disk/by-path/pci-*-ata-*` |
| EC profile | 8+3 (failure domain: OSD) | 8+3 (failure domain: OSD) |
| Replicated pool size | 2 (min_size 1) | 2 (min_size 1) |
## host_vars schema by hardware shape
The two clusters have fundamentally different storage topologies, which
shows up in their `host_vars/<host>.yml` schemas. When adding a new cluster,
operators must pick the schema matching the hardware — not just copy from
either existing cluster blindly.
Storage topologies differ across hardware shapes, which shows up in the
`host_vars/<host>.yml` schema. When adding a new cluster, operators must
pick the schema matching the hardware — not just copy from an existing
cluster blindly.
### sietch-shape (SAS expander + dual-SSD-VG)
@@ -130,7 +100,7 @@ The role composes full disk paths as
`-part<N>` suffix (SSD partitions). `roles/ceph_deploy/tasks/lvm-setup.yml`
runs this shape's VG/LV recovery path.
### painbox-shape (NVMe-RAID + single VG + LV-backed SSD OSD)
### NVMe-RAID shape (single VG + LV-backed SSD OSD)
```yaml
hostname_short: <cluster>-ceph-<name>
@@ -155,7 +125,7 @@ LVM is owned by the Hetzner installimage post-install script.
- **Has a SAS expander** (PERC HBA, mpt3sas, etc.) and **dedicated boot SSDs
partitioned for both block.db and OS** → sietch-shape.
- **Single VG covering boot + block.db + SSD OSD** (typical for
hosting-provider servers with NVMe RAID-1) → painbox-shape.
hosting-provider servers with NVMe RAID-1) → NVMe-RAID shape.
- **Other shapes** (e.g., dedicated NVMe block.db drives) require either
a new shape branch in `roles/ceph_deploy/tasks/osds.yml`'s template or
a fresh decision — see [ADR-011](adr/011-cephadm-osd-service-specs.md)
ansible/ceph/docs/naming.md
+9 -9
View File
@@ -32,14 +32,14 @@ OSD might set `role_in_hostname = "osd"` (yielding hostnames like
| Component | Example | Source (per-cluster tfvars field) |
|-----------|---------------------------------|--------------------------------------|
| cluster | `sietch`, `painbox` | top-level map key |
| cluster | `sietch` | top-level map key |
| role | `ceph` (small clusters), `osd` | `role_in_hostname` (default `ceph`) |
| name | `laurel`, `evelyn` | `hosts[].name`, or TF-picked |
| env | `dev`, `staging`, `prod` | `environment` |
| dc | `austin`, `hel`, `fsn` | `datacenter` |
| provider | `int`, `htz` | `provider_code` |
Current `role_in_hostname` values: both sietch and painbox use `ceph`
Current `role_in_hostname` values: sietch uses `ceph`
(mixed-role, all-nodes-are-everything). Dedicated-role hostnames (`osd`,
`mon`) are supported but not used today.
@@ -53,9 +53,8 @@ a deliberate, one-time decision.
bootstrap. Renaming after deployment is expensive (see "Cost of renaming"
below).
**Convention (unenforced):** Dune-themed. Existing clusters are `sietch`
(an underground Fremen community) and `painbox` (the Bene Gesserit
gom-jabbar test apparatus). Dune candidates not yet used, and that fit
**Convention (unenforced):** Dune-themed. The existing cluster is `sietch`
(an underground Fremen community). Dune candidates not yet used, and that fit
the hard constraints below: `arrakis`, `caladan`, `giedi`, `ixian`,
`kwisatz`, `muaddib`, `fremen`, `chani`, `leto`, `jessica`. Nothing in
the code enforces Dune specifically — mixed themes or theme breaks are
@@ -151,8 +150,9 @@ hosts = [
]
```
Painbox uses this path — `hosts[0]` has no name, and TF picked `evelyn`
on first apply, yielding `painbox-ceph-evelyn`.
A single-host cluster commonly uses this path — `hosts[0]` has no name,
so TF picks a stable word (e.g. `evelyn`) on first apply, yielding
`<cluster>-ceph-evelyn`.
Operator-declared names are excluded from the available pool to prevent
collisions within the cluster.
@@ -173,7 +173,7 @@ collisions within the cluster.
auto-picked host at `hosts[2]` could get a different name even if
nothing about its own entry changed. The safe patterns:
1. All-operator-declared within a cluster (sietch's model), or
2. All-auto-picked within a cluster (painbox's model).
2. All-auto-picked within a cluster.
Mixed works for initial setup but complicates later add/remove.
- **Converting between paths after deploy** (e.g., adding `name = "evelyn"`
to a host that previously auto-picked `evelyn`) **does not preserve the
@@ -233,7 +233,7 @@ The operator-side private key path is derived from the cluster name:
~/.ssh/id_ed25519_<cluster>
```
Examples: `~/.ssh/id_ed25519_sietch`, `~/.ssh/id_ed25519_painbox`. Per
Example: `~/.ssh/id_ed25519_sietch`. Per
[ADR-010](adr/010-ssh-keys-in-1password.md), the keypair lives in
`<CLUSTER>_CEPH_ANSIBLE_IAC_SSH_KEY` in 1P and is installed via
`scripts/install-ssh-keys.sh <cluster>`.
ansible/ceph/docs/patterns.md
+4 -4
View File
@@ -302,9 +302,9 @@ Ansible output is committed to `ansible.log` and displayed to operators
service in Ansible and run `cephadm` per item." This couples the role
tightly to per-host hardware shape (path composition, partition layout,
LV vs disk topology) and breaks on any new cluster shape — the original
sietch-shape `osds.yml` failed on painbox's PCI-ATA + LV-backed SSD OSD
topology because `sas_path_prefix` was hardcoded into the path
composition.
sietch-shape `osds.yml` failed on the NVMe-RAID shape's PCI-ATA +
LV-backed SSD OSD topology because `sas_path_prefix` was hardcoded into
the path composition.
**Pattern:** for any cephadm-managed surface (OSDs, RGW, MON/MGR
placement, monitoring), render a **declarative service spec** and apply
@@ -320,7 +320,7 @@ template's Jinja conditional, not the role logic.
- `templates/osd-spec.yml.j2` + `tasks/osds.yml`'s
`ceph orch apply osd -i` task — OSD service spec; per-host documents
in a multi-doc YAML; Jinja conditional handles sietch-shape vs
painbox-shape path composition.
NVMe-RAID-shape path composition.
**Template skeleton:**
ansible/ceph/docs/runbooks/recover-bad-tofu-apply.md
+1 -1
View File
@@ -79,7 +79,7 @@ that existing auto-name indices shifted.
```bash
mise run tf:plan
# Plan shows: ~inventory.ini content with hostname change from painbox-ceph-evelyn to painbox-ceph-<other>
# Plan shows: ~inventory.ini content with hostname change from <cluster>-ceph-evelyn to <cluster>-ceph-<other>
```
**Do NOT apply** — renaming a deployed host cascades into SSH known_hosts,
ansible/ceph/docs/runbooks/remote-hands-access.md
+1 -1
View File
@@ -41,7 +41,7 @@ least-privilege principle.
```bash
scripts/install-ssh-keys.sh
```
Lands `~/.ssh/id_ed25519_sietch` and `~/.ssh/id_ed25519_painbox` (0600).
Lands `~/.ssh/id_ed25519_sietch` (0600).
6. **Verify**:
```bash
mise run preflight # should pass; no desktop 1P required
ansible/ceph/docs/runbooks/replace-host.md
+2 -2
View File
@@ -52,8 +52,8 @@ CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory-provision.ini \
--limit sietch-ceph-<name>
```
**Hetzner (painbox)**: reboot into rescue, run installimage + post-install
scripts from `inventories/painbox-ceph.dev.hel.htz/installimage/`.
**Hetzner (NVMe-RAID host)**: reboot into rescue, run installimage +
post-install scripts from the cluster's `inventories/<cluster>/installimage/`.
### 4. Clear the old host's Ceph state
ansible/ceph/docs/runbooks/rotate-secrets.md
+1 -1
View File
@@ -26,7 +26,7 @@ deploy` picks up the new value via `op inject`.
| S3 service-user access | `<CLUSTER>_CEPH_S3_SVC_YUCCA_RESTIC_ACCESS_KEY` | `yucca_tf_dev` | RGW S3 user `yucca-restic` access key |
| S3 service-user secret | `<CLUSTER>_CEPH_S3_SVC_YUCCA_RESTIC_SECRET_KEY` | `yucca_tf_dev` | RGW S3 user `yucca-restic` secret key |
Replace `<CLUSTER>` with `SIETCH` or `PAINBOX`. The active vault name is
Replace `<CLUSTER>` with the cluster short name (e.g. `SIETCH`). The active vault name is
declared per-cluster in the `vault` field of the cluster's entry in
`tf/deployment/staging/ceph/clusters.auto.tfvars` — `yucca_tf_dev` for dev,
future `yucca_tf_staging` / `yucca_tf` for staging/prod.
ansible/ceph/docs/scripts.md
+6 -6
View File
@@ -22,13 +22,13 @@ it inline, never via `export`:**
```bash
# Correct — inline prefix, applies to one mise/script invocation
CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run preflight
CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run preflight
# WRONG — mise's [env] machinery silently strips shell-exported vars
# when launching tasks; CEPH_ENV reaches an empty environment and the
# wrapper exits with "CEPH_ENV must be set". Confusing because your shell
# clearly has it set.
export CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini
export CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini
mise run preflight # fails
```
@@ -145,7 +145,7 @@ scripts/install-ssh-keys.sh [cluster...]
```
If no cluster arguments are given, installs keys for every known cluster
(`sietch`, `painbox`).
(`sietch`).
### What it does
@@ -173,7 +173,7 @@ For each target cluster:
### Arguments
Zero or more cluster short names (`sietch`, `painbox`). With no args, all
Zero or more cluster short names (`sietch`). With no args, all
known clusters are installed.
### Exit codes
@@ -262,8 +262,8 @@ Warnings (non-blocking) are reported in the summary but don't affect exit.
# Via mise (recommended)
mise run preflight
# Direct, against painbox
CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini \
# Direct, against sietch
CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini \
scripts/preflight.sh
```
ansible/ceph/docs/secrets.md
-7
View File
@@ -47,9 +47,6 @@ playbook time):
| `SIETCH_CEPH_OPS_PASSWORD` | `password` | `vault_ops_password` |
| `SIETCH_CEPH_DASHBOARD_PASSWORD` | `password` | `vault_ceph_dashboard_password` |
| `SIETCH_CEPH_GRAFANA_PASSWORD` | `password` | `vault_grafana_admin_password` |
| `PAINBOX_CEPH_OPS_PASSWORD` | `password` | `vault_ops_password` |
| `PAINBOX_CEPH_DASHBOARD_PASSWORD` | `password` | `vault_ceph_dashboard_password` |
| `PAINBOX_CEPH_GRAFANA_PASSWORD` | `password` | `vault_grafana_admin_password` |
**SSH Key items** (category `SSH Key`, consumed via
`scripts/install-ssh-keys.sh` on operator workstations and
@@ -59,8 +56,6 @@ playbook time):
|---|---|---|
| `SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` | `private_key` | `~/.ssh/id_ed25519_sietch` on operator workstation |
| `SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` | `public_key` | sietch nodes' `ansible-iac@:~/.ssh/authorized_keys` |
| `PAINBOX_CEPH_ANSIBLE_IAC_SSH_KEY` | `private_key` | `~/.ssh/id_ed25519_painbox` on operator workstation |
| `PAINBOX_CEPH_ANSIBLE_IAC_SSH_KEY` | `public_key` | painbox nodes' `ansible-iac@:~/.ssh/authorized_keys` |
**S3 service-user items** (predetermined keys passed to
`radosgw-admin user create --access-key=X --secret-key=Y` at deploy
@@ -71,8 +66,6 @@ credentials without waiting for post-bootstrap capture):
|---|---|---|
| `SIETCH_CEPH_S3_SVC_YUCCA_RESTIC_ACCESS_KEY` | `password` | `vault_s3_restic_access_key` → `ceph_rgw_s3_user_access_key` |
| `SIETCH_CEPH_S3_SVC_YUCCA_RESTIC_SECRET_KEY` | `password` | `vault_s3_restic_secret_key` → `ceph_rgw_s3_user_secret_key` |
| `PAINBOX_CEPH_S3_SVC_YUCCA_RESTIC_ACCESS_KEY` | `password` | same, painbox |
| `PAINBOX_CEPH_S3_SVC_YUCCA_RESTIC_SECRET_KEY` | `password` | same, painbox |
**Disaster-recovery items** (populated by `mise run capture` after
deploy — stored in 1P for recovery if the bootstrap node's filesystem
ansible/ceph/docs/troubleshooting.md
+3 -4
View File
@@ -510,10 +510,9 @@ ceph_rgw_dns_name: s3.{{ cluster_domain }}
```
This derives the DNS name from `cluster_domain` (e.g.
`s3.dev.hel.htz.futo.cloud`). Sietch defines this explicitly; painbox
was missing it before its first deploy. Future clusters should include
it from the start — see [docs/adding-a-cluster.md](adding-a-cluster.md)
group_vars template.
`s3.dev.austin.int.futo.cloud`). Sietch defines this explicitly. New
clusters should include it from the start — see
[docs/adding-a-cluster.md](adding-a-cluster.md) group_vars template.
---