yucca/ansible/ceph/docs/runbooks/replace-host.md

Runbook: Replace a Host (Hardware Swap, Preserving Name)

When: chassis failure, motherboard swap, or preventative hardware refresh. You want the new box to assume the old host's identity — same hostname, same IP, same Ceph OSD identities.

Time estimate: 1-2 hours (bare-metal) / 30 min (Hetzner).

---

Preserving name = preserving trust

Keeping the old hostname avoids:

• Re-issuing the ansible-iac SSH key to a new hostname
• Rotating the host's Ceph auth keys
• Updating monitoring dashboards / alert rules with new labels
• Updating any external systems that refer to the hostname

The TF inventory declaration stays unchanged — cluster name, host name, bond IP all match. Only the physical box changes.

Steps

1. Mark OSDs out, wait for backfill

# SSH to bootstrap node
sudo ceph osd out $(sudo ceph osd ls-tree <hostname-being-replaced>)
sudo ceph -w   # watch for HEALTH_OK or acceptable degraded state

Depending on cluster size, full backfill can take hours. For time-critical replacements, skip this step and accept degraded state during rebuild — but confirm you have headroom.

2. Power down old host, rack new one in same slot

Preserve the physical network cabling and IPMI address. The new box gets the same bond IP via the same DHCP reservation / static config.

3. Re-run provisioning against the new host

Bare-metal (sietch):

# Boot the new box from the Debian 12 live image (same procedure as first
# provision — see docs/runbooks/add-node.md for iDRAC steps).

CEPH_ENV=inventories/staging-austin/sietch/inventory-provision.ini \
  scripts/ansible-play.sh provision.yml \
  -e confirm_wipe=true \
  --limit sietch-ceph-<name>

Hetzner (NVMe-RAID host): reboot into rescue, run installimage + post-install scripts from the cluster's inventories/<cluster>/installimage/.

4. Clear the old host's Ceph state

The cluster still has the old host's OSDs, CRUSH entries, and cephadm host record. Clean those up BEFORE the new box rejoins:

# On bootstrap node:
for osd in $(sudo ceph osd ls-tree <hostname>); do
  sudo ceph osd purge $osd --yes-i-really-mean-it
done
sudo ceph osd crush remove <hostname>
sudo ceph orch host rm <hostname> --force

5. Re-run baseline + deploy for that host

scripts/ansible-play.sh baseline.yml --limit <hostname>
scripts/ansible-play.sh deploy-ceph.yml \
  --limit <hostname>,<bootstrap-hostname>

Note bootstrap host must be in --limit — join/placement/OSD activation all run from bootstrap.

6. Verify

# All OSDs for the replaced host are up+in
sudo ceph osd tree | grep <hostname>

# Cluster is rebalancing or healthy
sudo ceph -s

Gotchas

• Hardware topology changed: new chassis may have different PCI paths for SAS controllers / NVMe. Update host_vars/<hostname>.yml (sas_path_prefix, ceph_hdd_osds) before step 5.
• Different serial numbers: acceptable — by-path is used for OSD slot identity, not by-id.
• Backfill thundering herd: if you skipped step 1, the new OSDs enter the cluster and backfill aggressively. Throttle with ceph_osd_recovery_max_active in vars.yml if you see client-IO impact.

References

• docs/runbooks/add-node.md — related but for net-new hosts
• docs/runbooks/replace-disk.md — single-disk replacement within a host