diff --git a/README.md b/README.md index 261db7b7..19f74d2b 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ library + per-service charts). See [`kubernetes/README.md`](./kubernetes/README. | Start here | Path | Purpose | | ------------------------------------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [`ansible/ceph/README.md`](./ansible/ceph/README.md) | `ansible/ceph/` | Ansible automation for Ceph clusters (sietch, painbox). Deploys + operates via cephadm on bare-metal and Hetzner. | +| [`ansible/ceph/README.md`](./ansible/ceph/README.md) | `ansible/ceph/` | Ansible automation for Ceph clusters (sietch). Deploys + operates via cephadm on bare-metal and Hetzner. | | [`ansible/talos/README.md`](./ansible/talos/README.md) | `ansible/talos/` | Talos K8s as libvirt VMs on the Ceph hypervisors. Ansible provisions the substrate + VMs; TF renders inventory and bootstraps the cluster. | | [`tf/README.md`](./tf/README.md) | `tf/` | Terraform/OpenTofu authority for cluster identity, 1P secret items, rendered Ansible inventories. Terragrunt multi-env (`deployment///`). | | [`kubernetes/README.md`](./kubernetes/README.md) | `kubernetes/` | Flux GitOps surface (apps/components/flux/bootstrap) for the Talos K8s cluster. Per-app HelmReleases over the in-repo `charts/`; mirrored locally by Tilt. | diff --git a/ansible/ceph/CONTRIBUTING.md b/ansible/ceph/CONTRIBUTING.md index 233af188..02d2dae7 100644 --- a/ansible/ceph/CONTRIBUTING.md +++ b/ansible/ceph/CONTRIBUTING.md @@ -23,20 +23,20 @@ manage. ### SSH setup -The `ansible-iac` SSH keys live in `yucca_tf_dev` as items -`SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` and `PAINBOX_CEPH_ANSIBLE_IAC_SSH_KEY` +The `ansible-iac` SSH keys live in `yucca_tf_dev` as items like +`SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` (see [ADR-010](docs/adr/010-ssh-keys-in-1password.md) for rationale). **First-time workstation setup:** ```bash cd yucca/ansible/ceph -scripts/install-ssh-keys.sh # op read → ~/.ssh/id_ed25519_{sietch,painbox} +scripts/install-ssh-keys.sh # op read → ~/.ssh/id_ed25519_sietch ``` The script is idempotent and refuses to overwrite an existing key whose -fingerprint doesn't match 1P. Keys land as `~/.ssh/id_ed25519_sietch` + -`~/.ssh/id_ed25519_painbox` (private + `.pub` both 0600/0644). +fingerprint doesn't match 1P. Keys land as `~/.ssh/id_ed25519_sietch` +(private + `.pub` both 0600/0644). **Jump hosts / proxies** belong in your personal `~/.ssh/config`, not in this repo. @@ -84,7 +84,6 @@ It points to an **inventory file** (not a directory): ```bash # Inline prefix — required for `mise run` invocations: CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run preflight -CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run status ``` **`export CEPH_ENV=...` does NOT work with `mise run`.** mise's `[env]` @@ -98,7 +97,7 @@ For multiple commands against the same cluster, set a local (non-exported) shell variable and inline-prefix each invocation: ```bash -CE=inventories/painbox-ceph.dev.hel.htz/inventory.ini +CE=inventories/sietch-ceph.staging.austin.int/inventory.ini CEPH_ENV=$CE mise run preflight CEPH_ENV=$CE mise run status CEPH_ENV=$CE mise run deploy diff --git a/ansible/ceph/README.md b/ansible/ceph/README.md index 20dd25c9..d2b5d10a 100644 --- a/ansible/ceph/README.md +++ b/ansible/ceph/README.md @@ -8,7 +8,6 @@ scaffolding are provisioned from `yucca/tf/` (see `../../tf/`). | Cluster | Domain | Location | Hardware | Nodes | |---------|--------|----------|----------|-------| | **sietch** | `dev.austin.int.futo.cloud` | Austin DC | Dell R730xd | 3 | -| **painbox** | `dev.hel.htz.futo.cloud` | Hetzner Helsinki | SX295 | 1 | Clusters are declared in `yucca/tf/deployment/staging/ceph/clusters.auto.tfvars`; `tofu apply` renders `inventories//inventory.ini` and @@ -30,14 +29,9 @@ graph TB S[samara
MON+MGR+OSD+RGW] end - subgraph "Hetzner Helsinki" - P[painbox-ceph-evelyn
MON+MGR+OSD+RGW] - end - A -->|SSH| L A -->|SSH| W A -->|SSH| S - A -->|SSH| P ``` See [docs/architecture.md](docs/architecture.md) for role dependencies, @@ -134,12 +128,12 @@ Inventory scaffolding + secret-item provisioning live in `yucca/tf/` — run | [docs/adding-a-cluster.md](docs/adding-a-cluster.md) | Developers -- inventory setup, secrets | | [docs/secrets.md](docs/secrets.md) | Developers/ops -- 1Password integration | | [docs/naming.md](docs/naming.md) | Everyone -- hostname and inventory naming | -| [docs/hardware.md](docs/hardware.md) | Ops/procurement -- R730xd vs SX295 specs | +| [docs/hardware.md](docs/hardware.md) | Ops/procurement -- node specs and hardware shapes | | [docs/s3-integration.md](docs/s3-integration.md) | App developers -- endpoints, boto3, certs | | [docs/security-model.md](docs/security-model.md) | InfoSec -- encryption, users, firewall | | [docs/capacity-planning.md](docs/capacity-planning.md) | Managers -- costs, formulas, growth | | [docs/troubleshooting.md](docs/troubleshooting.md) | SRE/on-call -- symptom/diagnosis/fix | -| [docs/runbooks/](docs/runbooks/) | Ops -- add/replace node, replace disk, rotate certs/secrets/SSH/SA token, remote hands, painbox reprovision, bad-tofu-apply recovery, backup/restore | +| [docs/runbooks/](docs/runbooks/) | Ops -- add/replace node, replace disk, rotate certs/secrets/SSH/SA token, remote hands, bad-tofu-apply recovery, backup/restore | | [docs/adr/](docs/adr/) | Everyone -- architecture decision records | ## Known Limitations diff --git a/ansible/ceph/docs/adding-a-cluster.md b/ansible/ceph/docs/adding-a-cluster.md index c8aa64ef..8c19c880 100644 --- a/ansible/ceph/docs/adding-a-cluster.md +++ b/ansible/ceph/docs/adding-a-cluster.md @@ -36,7 +36,6 @@ of datacenter or environment. Existing examples: - `sietch-ceph.dev.austin.int/` — Austin DC, internal network, dev -- `painbox-ceph.dev.hel.htz/` — Hetzner Helsinki, dev Future environments land as siblings: `*-ceph.staging../`, `*-ceph.prod../`. @@ -48,7 +47,7 @@ Future environments land as siblings: `*-ceph.staging../`, The engineer adding the cluster picks the name. Conventions and constraints live in [docs/naming.md](naming.md#cluster-naming). Quick summary: -- **Convention:** Dune-themed (existing: `sietch`, `painbox`). Not enforced. +- **Convention:** Dune-themed (existing: `sietch`). Not enforced. - **Constraints:** lowercase, short (6–10 chars ideal), no dashes or dots, unique within the `yucca_tf_*` item namespace, not already a key in `clusters.auto.tfvars`. @@ -66,7 +65,6 @@ Working example for a hypothetical `mesa` cluster at Hetzner Falkenstein: ```hcl clusters = { sietch = { ... } - painbox = { ... } mesa = { domain = "dev.fsn.htz.futo.cloud" @@ -126,10 +124,11 @@ Hand-maintained, committed. Copy the closer existing analogue as a starting point: - **Bare-metal cluster:** copy from `sietch-ceph.dev.austin.int/group_vars/all/vars.yml` -- **Hetzner/single-NIC cluster:** copy from `painbox-ceph.dev.hel.htz/group_vars/all/vars.yml` +- **Hetzner/single-NIC cluster:** start from the sietch vars and adjust for + the NVMe-RAID shape (public /32, no bond/ProxyJump, installimage-owned LVM). ```bash -cp inventories/painbox-ceph.dev.hel.htz/group_vars/all/vars.yml \ +cp inventories/sietch-ceph.dev.austin.int/group_vars/all/vars.yml \ inventories/mesa-ceph.dev.fsn.htz/group_vars/all/vars.yml ``` @@ -280,8 +279,8 @@ the template, SSH reachable, Python 3 on targets. ### 9. Deploy For Hetzner installimage clusters, run the installimage flow first -(out-of-band; see [runbooks/painbox-reprovision.md](runbooks/painbox-reprovision.md) -for the pattern). For Austin bare-metal clusters, run `provision.yml` +(out-of-band: reboot into rescue mode, run `installimage/autosetup` plus +the op-injected `post-install.sh`). For Austin bare-metal clusters, run `provision.yml` first (boot into the live image, then `scripts/ansible-play.sh provision.yml -e confirm_wipe=true` with `CEPH_ENV=.../inventory-provision.ini`). Then: @@ -323,7 +322,7 @@ CEPH_ENV=inventories/mesa-ceph.dev.fsn.htz/ Default is set in `.mise.toml` (`sietch` in dev). Override per-command: ```bash -CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run status +CEPH_ENV=inventories/sietch-ceph.dev.austin.int/inventory.ini mise run status ``` `scripts/ansible-play.sh` derives the secrets template path from `CEPH_ENV` @@ -375,4 +374,3 @@ trap-cleaned on exit. - [scripts.md](scripts.md) — wrapper reference - [ADR-009](adr/009-tf-first-op-inject-over-vault-password-sh.md) — why TF is authoritative - [ADR-010](adr/010-ssh-keys-in-1password.md) — why SSH keys live in 1P -- [runbooks/painbox-reprovision.md](runbooks/painbox-reprovision.md) — Hetzner-specific reprovisioning pattern diff --git a/ansible/ceph/docs/architecture.md b/ansible/ceph/docs/architecture.md index ba001018..e3377770 100644 --- a/ansible/ceph/docs/architecture.md +++ b/ansible/ceph/docs/architecture.md @@ -19,13 +19,11 @@ flowchart LR ONEP[("1Password org
yucca_tf · yucca_tf_dev · ...")] S3[("OVH S3
yucca-tf-state bucket")] SIETCH["Sietch · Austin DC
3× Dell R730xd"] - PAINBOX["Painbox · Hetzner Helsinki
1× SX295"] OP -->|edits| YUCCA OP -->|reads/writes secrets| ONEP OP -->|TF state I/O| S3 OP -->|SSH ansible-iac| SIETCH - OP -->|SSH ansible-iac| PAINBOX ``` The yucca monorepo is the single source of truth for cluster identity and @@ -40,8 +38,8 @@ External dependencies are minimal and explicit: - **OVH S3** — `yucca-tf-state` bucket at `s3.eu-west-par.io.cloud.ovh.net`. Project-keyed (`ceph///terraform.tfstate`) so multiple stacks share the bucket without collision. -- **Hardware** — Austin colo for sietch (Dell R730xd × 3, single 10G bond); - Hetzner Helsinki for painbox (SX295 × 1). Detail in [hardware.md](hardware.md). +- **Hardware** — Austin colo for sietch (Dell R730xd × 3, single 10G bond). + Detail in [hardware.md](hardware.md). --- @@ -59,7 +57,7 @@ its environment from the same source — directory layout — so dev / staging | Ansible inv | `inventories/-ceph.dev../` | `inventories/-ceph.staging../` | `inventories/-ceph.prod../` | | mise default | `CEPH_ENV=...sietch-ceph.dev.austin.int/inventory.ini` | overridden via env at invocation | overridden via env at invocation | -Today the only deployed environment is dev (sietch + painbox). Adding +Today the only deployed environment is dev (sietch). Adding staging/prod is purely additive: create the matching `tf/deployment//ceph/` directory, populate `clusters.auto.tfvars`, and the same module + Ansible roles + mise tasks work unchanged. The state backend key path, 1P vault @@ -196,8 +194,8 @@ Operator-declared names are excluded from the pool to prevent collisions within a cluster. Adding hosts at the tail is safe — existing positions keep their names across applies. -Painbox today demonstrates this: its single host has no `name`, so TF -auto-picked `evelyn` → hostname `painbox-ceph-evelyn`. +A host declared with no `name` demonstrates this: TF auto-picks a stable +word (e.g. `evelyn`) → hostname `-ceph-evelyn`. ### Rendered artifacts (gitignored) @@ -269,7 +267,7 @@ Rotation procedure: [docs/runbooks/rotate-sa-token.md](runbooks/rotate-sa-token. ### Item categories and naming Per cluster, the following items live in the cluster's `vault` (currently -`yucca_tf_dev` for both sietch and painbox): +`yucca_tf_dev` for sietch): | Category | Item title pattern | Field consumed | |------------|---------------------------------------------------------|----------------------| @@ -374,7 +372,7 @@ flowchart TB P2["Phase 2 · bootstrap.yml
cephadm bootstrap on first node"] P3["Phase 3 · join.yml
ceph orch host add for remaining nodes"] P4["Phase 4 · placement.yml
MON/MGR placement calculation"] - P45["Phase 4.5 · lvm-setup.yml
ensure block.db VGs/LVs exist (sietch-shape only;
painbox skips — LVM owned by installimage post-install)"] + P45["Phase 4.5 · lvm-setup.yml
ensure block.db VGs/LVs exist (sietch-shape only;
NVMe-RAID shape skips — LVM owned by installimage post-install)"] P5["Phase 5 · osds.yml
render osd-spec.yml.j2 → ceph orch apply osd
(cephadm provisions LUKS + LVM internally)"] P55["Phase 5.5 · crush-rules.yml
replicated_hdd / replicated_ssd rules"] P575["Phase 5.75 · rgw.yml
EC pools, realm/zone, TLS, S3 user"] @@ -403,16 +401,12 @@ inventories/ sietch-ceph-lawson.yml sietch-ceph-samara.yml installimage/ Hetzner installimage assets (sietch n/a) - - painbox-ceph.dev.hel.htz/ Hetzner dev cluster - inventory.ini TF-generated, gitignored - inventory-destroy.ini TF-generated, gitignored - secrets.yml.tpl TF-generated, gitignored - group_vars/all/vars.yml committed - host_vars/painbox-ceph-evelyn.yml committed - installimage/ post-install.sh.tpl (op-injected) ``` +Hetzner NVMe-RAID clusters follow the same layout, adding an +`installimage/` directory with a `post-install.sh.tpl` (op-injected) that +owns LVM setup. + `host_vars/*.yml` is committed because per-node hardware facts (bond_ip, SAS expander paths, SSD PHY positions, HDD-to-block.db mappings) are stable inventory truth — not operator preference. The `.local.yml` suffix is @@ -500,7 +494,7 @@ Both are overridable per-invocation: ```bash TF_STACK_DIR=tf/deployment/staging/ceph mise run tf:plan -CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run status +CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run status ``` --- @@ -639,7 +633,7 @@ sequenceDiagram ```mermaid flowchart TB SIETCH["Sietch prep:
provision_host/disks.yml partitions SSDs
then ceph_deploy/lvm-setup.yml
creates VG + db-slot LVs on each SSD's partition 5"] - PAINBOX["Painbox prep:
installimage post-install.sh
NVMe RAID-1 → vg0 → db-slot0..13 + ssd-osd LVs"] + NVMERAID["NVMe-RAID prep:
installimage post-install.sh
NVMe RAID-1 → vg0 → db-slot0..13 + ssd-osd LVs"] SPEC["ceph_deploy/osds.yml renders
templates/osd-spec.yml.j2 → /etc/ceph/osd-spec.yml
one document per host; paths from host_vars"] APPLY["ceph orch apply osd -i /etc/ceph/osd-spec.yml
cephadm: discover disks, LUKS-format, LVM, deploy daemons"] POLL["Wait for cephadm to provision
poll num_osds until expected count reached"] @@ -648,7 +642,7 @@ flowchart TB REWEIGHT["Safety net: fix any reweight=0 OSDs"] SIETCH --> SPEC - PAINBOX --> SPEC + NVMERAID --> SPEC SPEC --> APPLY --> POLL --> UP --> UNSET --> REWEIGHT ``` @@ -657,7 +651,8 @@ flowchart TB Earlier versions of this role iterated `cephadm ceph-volume lvm create` per disk and composed `/dev/disk/by-path/...` paths from host_vars (`sas_path_prefix` + `path_phy`). That assumed sietch's SAS expander -topology and broke on painbox's PCI-ATA disks plus LV-backed SSD OSD. +topology and broke on the NVMe-RAID shape's PCI-ATA disks plus LV-backed +SSD OSD. The current flow renders a cephadm OSD service spec from per-host data and applies it via `ceph orch apply osd -i`. Cephadm handles device @@ -675,7 +670,7 @@ with two shape branches: - **Sietch** (`sas_path_prefix` defined): data path = `/dev/disk/by-path/{{ sas_path_prefix }}-{{ path_phy }}-lun-0`; SSD OSD = partition on the SAS-attached SSD via `path_phy + partition`. -- **Painbox** (`sas_path_prefix` undefined): data path = +- **NVMe-RAID shape** (`sas_path_prefix` undefined): data path = `/dev/disk/by-path/{{ path_phy }}` (operator authors the full PCI-ATA identifier in host_vars); SSD OSD = LV via the `lv` field (`/dev/{{ lv }}`). @@ -773,7 +768,7 @@ This prevents: - Overwriting the marker with a stale `provisioned_at` timestamp The marker filename (`ceph-provisioned.json`) is project-scoped, not -cluster-scoped — every Ceph cluster (sietch, painbox, future) writes the +cluster-scoped — every Ceph cluster (sietch, future) writes the same filename. The marker's *contents* identify which cluster + host the machine belongs to. diff --git a/ansible/ceph/docs/capacity-planning.md b/ansible/ceph/docs/capacity-planning.md index f8160b32..4861256c 100644 --- a/ansible/ceph/docs/capacity-planning.md +++ b/ansible/ceph/docs/capacity-planning.md @@ -8,8 +8,6 @@ disk layouts see [hardware.md](hardware.md); this doc is the sizing math. | Cluster | Nodes | HDD × size | Raw HDD | EC-usable (8+3) | 70%-full target | |---|---|---|---|---|---| | sietch (Austin) | 3 × Dell R730xd | 30 × 6 TB | ~164 TiB | ~119 TiB | ~83 TiB | -| painbox (Hetzner Helsinki) | 1 × SX295 | 14 × 22 TB | ~280 TiB | ~204 TiB | ~143 TiB | -| **Combined** | | | **~444 TiB** | **~323 TiB** | **~226 TiB** | Each cluster also contributes ~1 TiB of SSD OSD on the boot SSDs (minor; ignored in the math above). diff --git a/ansible/ceph/docs/hardware.md b/ansible/ceph/docs/hardware.md index c3ba08c0..667377dc 100644 --- a/ansible/ceph/docs/hardware.md +++ b/ansible/ceph/docs/hardware.md @@ -9,14 +9,16 @@ For where this fits in the tool mesh, see [architecture.md](architecture.md). ## Network topology -Both clusters use a **single-network** design today — public and cluster +Clusters use a **single-network** design today — public and cluster traffic share one subnet. Production will separate them; see [security-model.md](security-model.md) for the threat-model implications. | Cluster | Subnet | Connection | |----------|-------------------|------------------------------------------------------------------------| | sietch | `10.10.10.0/24` | 2× 10GbE bonded active-backup (eno1 + eno2) per node; private switch | -| painbox | public /32 | Single 1GbE, direct SSH (no bond, no ProxyJump) | + +A Hetzner NVMe-RAID host would attach over a public /32 with a single NIC +and direct SSH (no bond, no ProxyJump). Per-node connection IPs (`bond_ip`) are declared in `tf/deployment/staging/ceph/clusters.auto.tfvars` and rendered by TF into the @@ -52,34 +54,14 @@ resolution, dashboard URL construction). | 5 | ~1.4 TB | Ceph block.db LVs (LVM VG) | | 6 | ~2 TB | SSD OSD data (ceph-volume) | -## painbox -- Hetzner SX295 (Helsinki) +## Hetzner NVMe-RAID shape (e.g. SX295) -| Component | Spec | -|-----------|------| -| Chassis | Hetzner SX295 storage server | -| CPU | AMD EPYC 7502P (32C/64T) | -| RAM | 128 GB DDR4 ECC | -| Boot NVMe | 2x Samsung 7.68TB (installimage RAID-1, vg0) | -| HDD OSDs | 14x Seagate Exos X22 22TB SATA | -| SSD OSD | 1x ~4.4TB LV on vg0 (NVMe remainder) | -| Block.db | 14x 128GB LVs on vg0 | -| SATA | 3 onboard controllers (8+2+4 ports = 14 total) | -| Network | Single 1GbE, direct SSH (no bond, no ProxyJump) | -| Boot | BIOS (Hetzner standard) | -| OS | Debian 12 Bookworm (Hetzner installimage) | -| Provisioning | Rescue mode → `installimage/autosetup` + `post-install.sh` | - -### NVMe layout (vg0 on md1) - -| LV | Size | Use | -|----|------|-----| -| swap | 32 GB | Swap | -| root | 100 GB | / | -| var | 200 GB | /var | -| varlog | 50 GB | /var/log | -| db-slot0..13 | 14x 128 GB | Block.db per HDD OSD | -| ssd-osd | ~4.4 TB | SSD OSD data | -| (reserve) | ~512 GB | Future expansion | +The roles also support a Hetzner-style single-box shape for future +hosting-provider clusters: NVMe boot drives in installimage RAID-1 +(`vg0`), HDD OSDs over onboard SATA with per-HDD block.db LVs on the NVMe +VG, and a single LV-backed SSD OSD carved from the NVMe remainder. +Provisioning is rescue mode → `installimage/autosetup` + `post-install.sh`, +booting Debian 12 Bookworm. > **Why Bookworm and not Trixie:** upstream Ceph Tentacle's Debian > repository at `download.ceph.com/debian-tentacle/dists/` publishes only @@ -87,24 +69,12 @@ resolution, dashboard URL construction). > autosetup `IMAGE` line MUST select a Bookworm tarball until upstream > ships Trixie packages. -## Comparison - -| | sietch (per node) | painbox (single node) | -|---|---|---| -| HDD OSD count | 8-12 | 14 | -| SSD OSD count | 2 | 1 | -| block.db per HDD | 240 GB | 128 GB | -| Total raw HDD | 48-72 TB | 308 TB | -| Device path format | `/dev/disk/by-path/sas-exp*-phy*-lun-0` | `/dev/disk/by-path/pci-*-ata-*` | -| EC profile | 8+3 (failure domain: OSD) | 8+3 (failure domain: OSD) | -| Replicated pool size | 2 (min_size 1) | 2 (min_size 1) | - ## host_vars schema by hardware shape -The two clusters have fundamentally different storage topologies, which -shows up in their `host_vars/.yml` schemas. When adding a new cluster, -operators must pick the schema matching the hardware — not just copy from -either existing cluster blindly. +Storage topologies differ across hardware shapes, which shows up in the +`host_vars/.yml` schema. When adding a new cluster, operators must +pick the schema matching the hardware — not just copy from an existing +cluster blindly. ### sietch-shape (SAS expander + dual-SSD-VG) @@ -130,7 +100,7 @@ The role composes full disk paths as `-part` suffix (SSD partitions). `roles/ceph_deploy/tasks/lvm-setup.yml` runs this shape's VG/LV recovery path. -### painbox-shape (NVMe-RAID + single VG + LV-backed SSD OSD) +### NVMe-RAID shape (single VG + LV-backed SSD OSD) ```yaml hostname_short: -ceph- @@ -155,7 +125,7 @@ LVM is owned by the Hetzner installimage post-install script. - **Has a SAS expander** (PERC HBA, mpt3sas, etc.) and **dedicated boot SSDs partitioned for both block.db and OS** → sietch-shape. - **Single VG covering boot + block.db + SSD OSD** (typical for - hosting-provider servers with NVMe RAID-1) → painbox-shape. + hosting-provider servers with NVMe RAID-1) → NVMe-RAID shape. - **Other shapes** (e.g., dedicated NVMe block.db drives) require either a new shape branch in `roles/ceph_deploy/tasks/osds.yml`'s template or a fresh decision — see [ADR-011](adr/011-cephadm-osd-service-specs.md) diff --git a/ansible/ceph/docs/naming.md b/ansible/ceph/docs/naming.md index 2f6a0e83..796aa219 100644 --- a/ansible/ceph/docs/naming.md +++ b/ansible/ceph/docs/naming.md @@ -32,14 +32,14 @@ OSD might set `role_in_hostname = "osd"` (yielding hostnames like | Component | Example | Source (per-cluster tfvars field) | |-----------|---------------------------------|--------------------------------------| -| cluster | `sietch`, `painbox` | top-level map key | +| cluster | `sietch` | top-level map key | | role | `ceph` (small clusters), `osd` | `role_in_hostname` (default `ceph`) | | name | `laurel`, `evelyn` | `hosts[].name`, or TF-picked | | env | `dev`, `staging`, `prod` | `environment` | | dc | `austin`, `hel`, `fsn` | `datacenter` | | provider | `int`, `htz` | `provider_code` | -Current `role_in_hostname` values: both sietch and painbox use `ceph` +Current `role_in_hostname` values: sietch uses `ceph` (mixed-role, all-nodes-are-everything). Dedicated-role hostnames (`osd`, `mon`) are supported but not used today. @@ -53,9 +53,8 @@ a deliberate, one-time decision. bootstrap. Renaming after deployment is expensive (see "Cost of renaming" below). -**Convention (unenforced):** Dune-themed. Existing clusters are `sietch` -(an underground Fremen community) and `painbox` (the Bene Gesserit -gom-jabbar test apparatus). Dune candidates not yet used, and that fit +**Convention (unenforced):** Dune-themed. The existing cluster is `sietch` +(an underground Fremen community). Dune candidates not yet used, and that fit the hard constraints below: `arrakis`, `caladan`, `giedi`, `ixian`, `kwisatz`, `muaddib`, `fremen`, `chani`, `leto`, `jessica`. Nothing in the code enforces Dune specifically — mixed themes or theme breaks are @@ -151,8 +150,9 @@ hosts = [ ] ``` -Painbox uses this path — `hosts[0]` has no name, and TF picked `evelyn` -on first apply, yielding `painbox-ceph-evelyn`. +A single-host cluster commonly uses this path — `hosts[0]` has no name, +so TF picks a stable word (e.g. `evelyn`) on first apply, yielding +`-ceph-evelyn`. Operator-declared names are excluded from the available pool to prevent collisions within the cluster. @@ -173,7 +173,7 @@ collisions within the cluster. auto-picked host at `hosts[2]` could get a different name even if nothing about its own entry changed. The safe patterns: 1. All-operator-declared within a cluster (sietch's model), or - 2. All-auto-picked within a cluster (painbox's model). + 2. All-auto-picked within a cluster. Mixed works for initial setup but complicates later add/remove. - **Converting between paths after deploy** (e.g., adding `name = "evelyn"` to a host that previously auto-picked `evelyn`) **does not preserve the @@ -233,7 +233,7 @@ The operator-side private key path is derived from the cluster name: ~/.ssh/id_ed25519_ ``` -Examples: `~/.ssh/id_ed25519_sietch`, `~/.ssh/id_ed25519_painbox`. Per +Example: `~/.ssh/id_ed25519_sietch`. Per [ADR-010](adr/010-ssh-keys-in-1password.md), the keypair lives in `_CEPH_ANSIBLE_IAC_SSH_KEY` in 1P and is installed via `scripts/install-ssh-keys.sh `. diff --git a/ansible/ceph/docs/patterns.md b/ansible/ceph/docs/patterns.md index 8e92a4cb..0b78d60c 100644 --- a/ansible/ceph/docs/patterns.md +++ b/ansible/ceph/docs/patterns.md @@ -302,9 +302,9 @@ Ansible output is committed to `ansible.log` and displayed to operators service in Ansible and run `cephadm` per item." This couples the role tightly to per-host hardware shape (path composition, partition layout, LV vs disk topology) and breaks on any new cluster shape — the original -sietch-shape `osds.yml` failed on painbox's PCI-ATA + LV-backed SSD OSD -topology because `sas_path_prefix` was hardcoded into the path -composition. +sietch-shape `osds.yml` failed on the NVMe-RAID shape's PCI-ATA + +LV-backed SSD OSD topology because `sas_path_prefix` was hardcoded into +the path composition. **Pattern:** for any cephadm-managed surface (OSDs, RGW, MON/MGR placement, monitoring), render a **declarative service spec** and apply @@ -320,7 +320,7 @@ template's Jinja conditional, not the role logic. - `templates/osd-spec.yml.j2` + `tasks/osds.yml`'s `ceph orch apply osd -i` task — OSD service spec; per-host documents in a multi-doc YAML; Jinja conditional handles sietch-shape vs - painbox-shape path composition. + NVMe-RAID-shape path composition. **Template skeleton:** diff --git a/ansible/ceph/docs/runbooks/recover-bad-tofu-apply.md b/ansible/ceph/docs/runbooks/recover-bad-tofu-apply.md index ee6699dc..99d9614a 100644 --- a/ansible/ceph/docs/runbooks/recover-bad-tofu-apply.md +++ b/ansible/ceph/docs/runbooks/recover-bad-tofu-apply.md @@ -79,7 +79,7 @@ that existing auto-name indices shifted. ```bash mise run tf:plan -# Plan shows: ~inventory.ini content with hostname change from painbox-ceph-evelyn to painbox-ceph- +# Plan shows: ~inventory.ini content with hostname change from -ceph-evelyn to -ceph- ``` **Do NOT apply** — renaming a deployed host cascades into SSH known_hosts, diff --git a/ansible/ceph/docs/runbooks/remote-hands-access.md b/ansible/ceph/docs/runbooks/remote-hands-access.md index 5987a920..dab77c54 100644 --- a/ansible/ceph/docs/runbooks/remote-hands-access.md +++ b/ansible/ceph/docs/runbooks/remote-hands-access.md @@ -41,7 +41,7 @@ least-privilege principle. ```bash scripts/install-ssh-keys.sh ``` - Lands `~/.ssh/id_ed25519_sietch` and `~/.ssh/id_ed25519_painbox` (0600). + Lands `~/.ssh/id_ed25519_sietch` (0600). 6. **Verify**: ```bash mise run preflight # should pass; no desktop 1P required diff --git a/ansible/ceph/docs/runbooks/replace-host.md b/ansible/ceph/docs/runbooks/replace-host.md index d8c96e07..19301f4d 100644 --- a/ansible/ceph/docs/runbooks/replace-host.md +++ b/ansible/ceph/docs/runbooks/replace-host.md @@ -52,8 +52,8 @@ CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory-provision.ini \ --limit sietch-ceph- ``` -**Hetzner (painbox)**: reboot into rescue, run installimage + post-install -scripts from `inventories/painbox-ceph.dev.hel.htz/installimage/`. +**Hetzner (NVMe-RAID host)**: reboot into rescue, run installimage + +post-install scripts from the cluster's `inventories//installimage/`. ### 4. Clear the old host's Ceph state diff --git a/ansible/ceph/docs/runbooks/rotate-secrets.md b/ansible/ceph/docs/runbooks/rotate-secrets.md index d4332eab..23797089 100644 --- a/ansible/ceph/docs/runbooks/rotate-secrets.md +++ b/ansible/ceph/docs/runbooks/rotate-secrets.md @@ -26,7 +26,7 @@ deploy` picks up the new value via `op inject`. | S3 service-user access | `_CEPH_S3_SVC_YUCCA_RESTIC_ACCESS_KEY` | `yucca_tf_dev` | RGW S3 user `yucca-restic` access key | | S3 service-user secret | `_CEPH_S3_SVC_YUCCA_RESTIC_SECRET_KEY` | `yucca_tf_dev` | RGW S3 user `yucca-restic` secret key | -Replace `` with `SIETCH` or `PAINBOX`. The active vault name is +Replace `` with the cluster short name (e.g. `SIETCH`). The active vault name is declared per-cluster in the `vault` field of the cluster's entry in `tf/deployment/staging/ceph/clusters.auto.tfvars` — `yucca_tf_dev` for dev, future `yucca_tf_staging` / `yucca_tf` for staging/prod. diff --git a/ansible/ceph/docs/scripts.md b/ansible/ceph/docs/scripts.md index 6febed53..7d1abe41 100644 --- a/ansible/ceph/docs/scripts.md +++ b/ansible/ceph/docs/scripts.md @@ -22,13 +22,13 @@ it inline, never via `export`:** ```bash # Correct — inline prefix, applies to one mise/script invocation -CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini mise run preflight +CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run preflight # WRONG — mise's [env] machinery silently strips shell-exported vars # when launching tasks; CEPH_ENV reaches an empty environment and the # wrapper exits with "CEPH_ENV must be set". Confusing because your shell # clearly has it set. -export CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini +export CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini mise run preflight # fails ``` @@ -145,7 +145,7 @@ scripts/install-ssh-keys.sh [cluster...] ``` If no cluster arguments are given, installs keys for every known cluster -(`sietch`, `painbox`). +(`sietch`). ### What it does @@ -173,7 +173,7 @@ For each target cluster: ### Arguments -Zero or more cluster short names (`sietch`, `painbox`). With no args, all +Zero or more cluster short names (`sietch`). With no args, all known clusters are installed. ### Exit codes @@ -262,8 +262,8 @@ Warnings (non-blocking) are reported in the summary but don't affect exit. # Via mise (recommended) mise run preflight -# Direct, against painbox -CEPH_ENV=inventories/painbox-ceph.dev.hel.htz/inventory.ini \ +# Direct, against sietch +CEPH_ENV=inventories/sietch-ceph.staging.austin.int/inventory.ini \ scripts/preflight.sh ``` diff --git a/ansible/ceph/docs/secrets.md b/ansible/ceph/docs/secrets.md index c10e9735..0bb21aca 100644 --- a/ansible/ceph/docs/secrets.md +++ b/ansible/ceph/docs/secrets.md @@ -47,9 +47,6 @@ playbook time): | `SIETCH_CEPH_OPS_PASSWORD` | `password` | `vault_ops_password` | | `SIETCH_CEPH_DASHBOARD_PASSWORD` | `password` | `vault_ceph_dashboard_password` | | `SIETCH_CEPH_GRAFANA_PASSWORD` | `password` | `vault_grafana_admin_password` | -| `PAINBOX_CEPH_OPS_PASSWORD` | `password` | `vault_ops_password` | -| `PAINBOX_CEPH_DASHBOARD_PASSWORD` | `password` | `vault_ceph_dashboard_password` | -| `PAINBOX_CEPH_GRAFANA_PASSWORD` | `password` | `vault_grafana_admin_password` | **SSH Key items** (category `SSH Key`, consumed via `scripts/install-ssh-keys.sh` on operator workstations and @@ -59,8 +56,6 @@ playbook time): |---|---|---| | `SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` | `private_key` | `~/.ssh/id_ed25519_sietch` on operator workstation | | `SIETCH_CEPH_ANSIBLE_IAC_SSH_KEY` | `public_key` | sietch nodes' `ansible-iac@:~/.ssh/authorized_keys` | -| `PAINBOX_CEPH_ANSIBLE_IAC_SSH_KEY` | `private_key` | `~/.ssh/id_ed25519_painbox` on operator workstation | -| `PAINBOX_CEPH_ANSIBLE_IAC_SSH_KEY` | `public_key` | painbox nodes' `ansible-iac@:~/.ssh/authorized_keys` | **S3 service-user items** (predetermined keys passed to `radosgw-admin user create --access-key=X --secret-key=Y` at deploy @@ -71,8 +66,6 @@ credentials without waiting for post-bootstrap capture): |---|---|---| | `SIETCH_CEPH_S3_SVC_YUCCA_RESTIC_ACCESS_KEY` | `password` | `vault_s3_restic_access_key` → `ceph_rgw_s3_user_access_key` | | `SIETCH_CEPH_S3_SVC_YUCCA_RESTIC_SECRET_KEY` | `password` | `vault_s3_restic_secret_key` → `ceph_rgw_s3_user_secret_key` | -| `PAINBOX_CEPH_S3_SVC_YUCCA_RESTIC_ACCESS_KEY` | `password` | same, painbox | -| `PAINBOX_CEPH_S3_SVC_YUCCA_RESTIC_SECRET_KEY` | `password` | same, painbox | **Disaster-recovery items** (populated by `mise run capture` after deploy — stored in 1P for recovery if the bootstrap node's filesystem diff --git a/ansible/ceph/docs/troubleshooting.md b/ansible/ceph/docs/troubleshooting.md index de9f09b3..6a18fc8c 100644 --- a/ansible/ceph/docs/troubleshooting.md +++ b/ansible/ceph/docs/troubleshooting.md @@ -510,10 +510,9 @@ ceph_rgw_dns_name: s3.{{ cluster_domain }} ``` This derives the DNS name from `cluster_domain` (e.g. -`s3.dev.hel.htz.futo.cloud`). Sietch defines this explicitly; painbox -was missing it before its first deploy. Future clusters should include -it from the start — see [docs/adding-a-cluster.md](adding-a-cluster.md) -group_vars template. +`s3.dev.austin.int.futo.cloud`). Sietch defines this explicitly. New +clusters should include it from the start — see +[docs/adding-a-cluster.md](adding-a-cluster.md) group_vars template. ---