yucca/tf

yucca/tf

Terraform/OpenTofu authority for cluster identity, 1P secret items, and the inventory artifacts Ansible consumes. Multi-partition / multi-region via terragrunt.

The partition / region model

Everything is keyed on a single first-class topology:

partition → region → { exactly one K8s cluster, one-or-more Ceph clusters }

• Partition = prod | staging | dev (formerly env).
• Region = a site: htz-fsn1, austin, local (formerly site/datacenter). Plus a reserved global pseudo-region for partition-wide stacks (DNS, the account-wide NetBird layer) — global is not a physical site, so its role/FQDN metadata is null.
• The human form is partition@region (prod@htz-fsn1); the canonical slug is partition-region (prod-htz-fsn1), which derives names on every surface.

Every stack sits at three path segments: deployment/<partition>/<region>/<stack>. The account-wide prod layer is prod/global/netbird (a stack under the global region), so there is no two-segment special case.

Layout

tf/
├── .env                              ← op:// references (committed; no literal secrets)
├── op-run.sh                         ← op-run wrapper used by the mise tf:* tasks
├── shared/
│   └── modules/
│       ├── ceph-cluster/             ← per-cluster ceph orchestration module
│       │   ├── main.tf, variables.tf, outputs.tf, rendering.tf
│       │   ├── wordlist.txt          ← 923 words for auto-picked hostnames
│       │   └── templates/
│       │       ├── inventory.ini.tftpl
│       │       ├── inventory-destroy.ini.tftpl
│       │       ├── inventory-provision-debian-live.ini.tftpl
│       │       └── secrets.yml.tpl.tftpl
│       └── talos-baremetal/         ← Talos on bare-metal nodes already in maintenance mode
│           ├── main.tf, variables.tf, outputs.tf
│           └── firewall.tf              ← Talos host ingress firewall (default-deny + allow-lists)
└── deployment/
    ├── terragrunt.hcl                ← root: state backend + partition/region/stack from path
    ├── staging/
    │   ├── austin/
    │   │   ├── region.hcl            ← role + FQDN parts for staging@austin
    │   │   ├── ceph/                 ← sietch ceph cluster (clusters.auto.tfvars)
    │   │   └── talos/                ← bare-metal Talos cluster (3× CP, Cilium CNI)
    │   └── global/
    │       ├── region.hcl            ← role=null (pseudo-region)
    │       ├── dns/                  ← Cloudflare records (futo.cloud)
    │       └── netbird/              ← NetBird Cloud access control (staging)
    ├── dev/
    │   ├── local/
    │   │   ├── region.hcl
    │   │   └── talos/                ← Talos VMs on the sietch hypervisors
    │   └── global/
    │       ├── region.hcl
    │       └── dns/
    └── prod/
        ├── htz-fsn1/
        │   ├── region.hcl           ← role + FQDN parts; site_id=40
        │   ├── mgmt-hosts.yaml      ← mgmt-host roster (region root; fabric + render read it)
        │   ├── fabric/              ← Junos switch fabric + NetBox + mgmt reprovision
        │   └── netbird/             ← htz-fsn1 site NetBird layer (routed htz-fsn1 network)
        └── global/
            ├── region.hcl
            ├── terragrunt.hcl       ← account-wide prod NetBird (two-segment region-root stack)
            └── netbird.tf

Add a region: create deployment/<partition>/<region>/region.hcl + stacks under it. Add a stack: a new sibling dir under a region (<region>/monitoring/ …). NetBird Cloud access control lives in staging/global/netbird/, and for prod is layered: prod/global/ (account-wide) above per-region prod/<region>/netbird/ (e.g. prod/htz-fsn1/netbird/). See "The netbird-env module" below.

The dns stack manages infrastructure names in the futo.cloud Cloudflare zone (today: the Sietch RGW S3 endpoint + virtual-hosted wildcard). Records are declarative in records.auto.tfvars; the API token resolves from op://yucca_tf_manual/CLOUDFLARE_API_TOKEN via tf/.env.

The talos stack is documented in ansible/talos/README.md and ansible/talos/docs/runbooks/cluster-bring-up.md (the TF + Ansible flow is interleaved — TF renders the inventory Ansible consumes, then bootstraps the VMs Ansible created).

Conventions

Partition, region, and stack are derived from the directory path

deployment/terragrunt.hcl parses the child's relative path: partition = segs[0], region = segs[1], stack = join(segs[2:]), and slug = "<partition>-<region>".

deployment/staging/austin/ceph     → partition=staging, region=austin,   stack=ceph
deployment/staging/global/dns      → partition=staging, region=global,   stack=dns
deployment/prod/htz-fsn1/fabric    → partition=prod,    region=htz-fsn1, stack=fabric
deployment/prod/htz-fsn1/netbird   → partition=prod,    region=htz-fsn1, stack=netbird
deployment/prod/global/netbird     → partition=prod,    region=global,   stack=netbird

The state backend key is derived from these: yucca/${partition}/${region}/${stack}/terraform.tfstate in the shared yucca-tf-state S3 bucket. (The legacy ceph/ project prefix is dropped — talos/dns/netbird/fabric all share the bucket now.) Every stack is three segments — the account-wide prod layer is prod/global/netbird (not a bare prod/global), so region.hcl at prod/global/ is found by find_in_parent_folders from the stack dir, exactly like staging/global. The n==2 branch in terragrunt.hcl is now dead and can be removed.

Per-region metadata + role (region.hcl)

Each region dir carries a deployment/<partition>/<region>/region.hcl holding role (primary | secondary), site_id, datacenter, provider_code, and domain. The root terragrunt finds it via find_in_parent_folders("region.hcl", "") (with a not-found guard) and merges its locals into every stack's inputs — so every stack in a region inherits the metadata without per-tfvars duplication. global pseudo-regions set role = null (and null FQDN parts). Each stack declares matching variable blocks with null defaults.

role encodes the product invariant: when a partition spans multiple regions, yucca-api + the database run only in the primary region; secondary regions run the storage-local subset. It is authoritative here in TF state (discovery.role) and consumed downstream (Flux role components, yuctl).

The op run --env-file=tf/.env -- pattern

tf/.env holds 1Password op:// references — not literal secrets:

export OP_SERVICE_ACCOUNT_TOKEN="op://yucca_tf_dev/yucca_futo_1pass_superuser_service_account/password"

Wrap every terragrunt invocation with op run --env-file=tf/.env -- (the mise tf:* tasks do this automatically). The op CLI resolves the op:// reference and injects the actual token as OP_SERVICE_ACCOUNT_TOKEN into the child process's environment. The 1P Terraform provider picks it up from the env var and authenticates.

The same pattern is used in immich-app/devtools and is the Futo-wide convention for TF secret injection.

Committed .env is safe because it's just pointers

Yucca's root .gitignore normally excludes .env files — we add an explicit !tf/.env exception. This file contains only op:// URIs; no secret ever transits the repo. It's a committed manifest of "which 1P items this TF depends on."

Stack override via TF_STACK_DIR

The default mise run tf:* tasks target tf/deployment/staging/austin/ceph. Point them at another stack via the TF_STACK_DIR env var:

TF_STACK_DIR=tf/deployment/staging/austin/ceph  mise run tf:plan
TF_STACK_DIR=tf/deployment/staging/austin/talos mise run tf:apply

Running TF

One-shot (preferred for now)

mise run tf:init      # first time in a stack
mise run tf:plan      # dry run
mise run tf:apply     # render artifacts + (future) create 1P items

These wrap: op run --env-file=tf/.env -- terragrunt --working-dir <stack> <cmd>.

In CI (staging stacks)

.github/workflows/infra.yml runs the staging stacks from GitHub Actions:

• Plan on every PR touching tf/**; apply on merge to main, gated behind the staging-infra Environment (required reviewers).
• Applies staging/austin/talos (cluster + Flux + secrets) then staging/global/dns.
• The Talos stack reaches the 10.10.10.0/24 nodes by joining the tailnet (tailscale/github-action, --accept-routes) — the cluster firewall already trusts the Tailscale CIDRs. The DNS stack is pure Cloudflare API, no tailnet.
• Secrets come from the same op run --env-file=tf/.env path; CI just supplies the per-env 1P service-account token (the rest resolves from 1P). The token is injected as OP_SERVICE_ACCOUNT_TOKEN from the environment-specific secret — OP_TF_YUCCA_STAGING_ENV here (dev/prod workflows use OP_TF_YUCCA_DEV_ENV / OP_TF_YUCCA_PROD_ENV) — replacing a shared superuser SA with a scoped one.

Prerequisites (out-of-band): repo secret OP_TF_YUCCA_STAGING_ENV — a staging 1P service account mirroring the dev/prod ones, i.e. granted shared_tf, shared_tf_staging, yucca_tf (read) + yucca_tf_staging (read/write for the JWT-keypair item). Note: a copy of the dev SA token won't work — it can't read yucca_tf_staging. Also: TS_OAUTH_CLIENT_ID, TS_OAUTH_SECRET; a Tailscale subnet router advertising 10.10.10.0/24 with tag:project-yucca approved for it; and the staging-infra Environment with required reviewers.

State backend

Remote: shared yucca-tf-state S3 bucket at OVH Paris (https://s3.eu-west-par.io.cloud.ovh.net/). Key path: yucca/${partition}/${region}/${stack}/terraform.tfstate (region-root stacks: yucca/${partition}/${region}/...). All yucca stacks live under the yucca/ prefix in the shared bucket.

Credentials are AWS-compatible env vars (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY), injected via op run --env-file=tf/.env from the TF_STATE_S3_* items in the yucca_tf vault. OVH-specific config (skip AWS validation, path-style URLs) is in deployment/terragrunt.hcl.

State locking is not enabled. OVH has no DynamoDB equivalent; OpenTofu's use_lockfile = true option would handle single-bucket locking but expects the lockfile object to already exist — terragrunt init against a fresh backend fails with 404 before it can create one. Enable it once the concern is concurrent applies (multiple operators working the same stack simultaneously). Single operator today → low risk.

Discovery outputs contract

Every stack emits a single non-sensitive top-level discovery output (plus discovery_schema_version). It is the machine-readable description of the topology that yuctl consumes — it reads the state object straight from S3 and parses .outputs.discovery.value, so no checkout, init, or provider is needed.

Secrets are always op:// references, never values. The in-stack sensitive kubeconfig/talosconfig outputs stay for in-stack use; discovery only carries the reference (op://<vault>/<title>/password).

Common envelope (every stack):

{
  schema_version, partition, region, slug, role, stack, stack_type,
  region_meta: { site_id, datacenter, provider_code, domain }
}

Per stack_type payload:

stack_type	stacks	payload key + fields
region-k8s	*/talos	kubernetes: { cluster_name, api_endpoint, operator_endpoint, cp_node_ips, kubeconfig_ref, talosconfig_ref }
ceph	*/ceph	ceph_clusters: { <name> => { cluster_name, fqdn, rgw_s3_endpoint, health_cred_ref, s3_admin_cred_refs, secret_item_titles, bootstrap_host } }
dns	*/dns	dns: { provider, zone, record_fqdns, api_token_ref }
netbird	*/netbird, prod/global	netbird: { name_prefix, vault, group_ids, policy_ids, network_ids, setup_key_item_titles }
fabric	prod/htz-fsn1/fabric	fabric: { site_id, kube_cidr, mgmt_cidr, cluster_cidrs }

The talos stacks persist their kube/talosconfig into 1Password (onepassword_item, mirroring the JWT-keypair / netbird-setup-key pattern) so the *_ref fields resolve — staging writes YUCCA_STAGING_{KUBE,TALOS}CONFIG; dev writes YUCCA_DEV_<CLUSTER>_{KUBE,TALOS}CONFIG.

Migrating an existing stack to the new layout

The dir moves + terragrunt rewrite change the state key (ceph/<env>/... → yucca/<partition>/<region>/...), so a live stack's remote state must be re-pointed — reversibly, no destroy/recreate. Per stack (single-operator window; no state locking):

1. Pre-flight terragrunt plan on the OLD layout → confirm no-op; back up the old state object.
2. Land the terragrunt.hcl rewrite + dir moves + renames (no apply yet).
3. Server-side S3 copy old key → new yucca/... key (old object stays as a rollback anchor).
4. Delete the stale generated backend.tf; terragrunt init in the new dir (decline the state-copy prompt; -reconfigure if needed).
5. terragrunt plan → must be no-op (the gate; a non-empty plan = a rename/source-path mismatch, not a state problem — STOP + roll back).
6. terragrunt output discovery → confirm the contract resolves.
7. After all stacks are green, aws s3 rm the legacy ceph/* objects.

Order: staging first, then prod (global → region → fabric — prod/global before prod/htz-fsn1/netbird, which has a real terragrunt dependency on it). dev is dir-move only (no remote state). The NetBird plans must show no resource renames — the rendered names are byte-identical across the rename.

The ceph-cluster module

Declarative input in clusters.auto.tfvars:

clusters = {
  sietch = {
    domain            = "staging.austin.int.futo.cloud"
    partition         = "staging"
    region            = "austin"
    provider_code     = "int"
    role_in_hostname  = "ceph"
    ansible_ssh_user  = "ansible-iac"
    ansible_ssh_key   = "~/.ssh/id_ed25519_sietch"
    vault             = "yucca_tf_staging"
    provision_profile = "debian-live"   # null for Hetzner-installimage clusters
    hosts = [
      { name = "laurel", bond_ip = "10.10.10.90", bootstrap = true },
      { name = "lawson", bond_ip = "10.10.10.91" },
      { name = "samara", bond_ip = "10.10.10.92" },
    ]
  }
}

On apply, the module:

1. Picks wordlist names for hosts[].name == null (stable across applies; seeded per-cluster; operator-declared names excluded from the pool to prevent collisions).
2. Renders inventory.ini (normal ops), inventory-destroy.ini (explicit destroy flag), secrets.yml.tpl (op:// references to yucca_tf_staging/<CLUSTER>_CEPH_*_PASSWORD/password). Optionally renders inventory-provision.ini when provision_profile != null.
3. (Not yet TF-managed) onepassword_item resources for cluster secrets are dormant — items are created via op CLI today and read by Ansible at play time. See ansible/ceph/docs/secrets.md for the re-enable plan.

The talos-baremetal module (staging/austin/talos)

Brings up Talos on bare-metal nodes already running in maintenance mode at known addresses — no Ansible, no hypervisors, no VLANs (the earlier VM-oriented talos-cluster module was removed unused). It dials each node's maintenance IP, applies machine config (which installs to disk + reboots), bootstraps one CP, then emits kube/talosconfig and gates on cluster health.

Declarative input in deployment/staging/austin/talos/clusters.auto.tfvars:

clusters = {
  yucca-staging = {
    talos_version      = "1.13.4"
    kubernetes_version = "v1.36.1"
    install_disk       = "/dev/sda"        # WIPED — the 240GB DELLBOSS; NVMe left raw
    cluster_vip        = "10.10.10.15"     # L2 VIP, etcd-elected across CPs
    gateway            = "10.10.10.1"
    subnet_cidr        = "10.10.10.0/24"
    cni                = "cilium"           # cni:none in Talos + Cilium via Helm
    disable_kube_proxy = true               # Cilium kube-proxy replacement (KubePrism)
    cilium_version     = "1.19.5"
    hubble             = true
    bond = { interfaces = ["eno1np0", "eno2np1"], mode = "active-backup" } # flip to 802.3ad after the switch is LACP'd
    nodes = [
      { name = "staging-cp1", address = "10.10.10.47" },
      { name = "staging-cp2", address = "10.10.10.242" },
      { name = "staging-cp3", address = "10.10.10.117" },
    ]
  }
}

Notes:

• Static IP = maintenance IP. Each node's address is pinned as the static IP on bond0, so TF stays reachable across the install reboot.
• bond comes up active-backup (no switch config needed). Migrate to 802.3ad later, node-by-node, after converting the switch ports to LACP port-channels — LACP needs both ends configured at once, so a big-bang flip drops connectivity until both sides agree.
• Ingress firewall (firewall.tf): default-deny + per-service allow-lists scoped to the subnet (+ pod CIDR on kubelet). apid + apiserver also trust the Tailscale ranges (trust_tailscale). ⚠️ The host running tf apply must have a source IP inside an allowed range or apid (50000) is blocked and bootstrap hangs — add operator/jump subnets to trusted_cidrs.
• CNI is installed in the same apply. With cni:none the nodes are NotReady until Cilium lands, so the module's health gate runs skip_kubernetes_checks; helm.tf installs Cilium, then a second (full) health gate enforces Ready.
• One cluster per stack. The helm provider binds to a single cluster (one(...)); add more clusters in their own stack.

Run it (see "Running TF" below — needs 1Password unlocked + an on-LAN apply host):

TF_STACK_DIR=tf/deployment/staging/austin/talos mise run tf:init
TF_STACK_DIR=tf/deployment/staging/austin/talos mise run tf:plan
TF_STACK_DIR=tf/deployment/staging/austin/talos mise run tf:apply   # WIPES /dev/sda, installs Talos

The netbird-env module (NetBird Cloud access control)

Manages one layer's NetBird Cloud footprint: groups, access policies, device auth (setup) keys, and routed networks. One NetBird Cloud account (api.netbird.io) backs everything — the module namespaces every object <name_prefix>_<key> (all underscores) so all envs/sites coexist.

Group model

Per env (and per prod site), the baseline groups are:

group	who	rendered (staging / prod htz-fsn1)
ci	ephemeral CI runners	yucca-staging-ci / yucca-prod-htz-fsn1-ci
mgmt	management nodes (configured via Ansible; also the route peers)	yucca-staging-mgmt / …
talos	Talos cluster nodes	yucca-staging-talos / …
k8s_operator	in-cluster kubernetes operator	yucca-staging-k8s-operator / …

Logical keys (the tfvars map keys, e.g. ci) stay lowercase; rendered NetBird names are UPPER_SNAKE (uppercased, hyphens → underscores). CI is per-env (ci, reaching only that env's groups) — no cross-env CI plane. k8s is split into talos (the nodes) and k8s_operator (the operator identity) so they can carry different policies.

Stacks & layering

stack	partition / scope	state key
deployment/staging/global/netbird	staging (global region)	yucca/staging/global/netbird/…
deployment/prod/global/netbird	prod, account-wide (cross-region)	yucca/prod/global/netbird/…
deployment/prod/htz-fsn1/netbird	prod, region htz-fsn1	yucca/prod/htz-fsn1/netbird/…

Staging is single-layer. Prod is layered: a global layer (reserved for account-wide / cross-region groups + policies) above per-region layers. The global layer is empty today — each region owns its own resource group and the yucca → resources policy is module-generated per layer (see below). Region groups are region-scoped (yucca-prod-<region>-<role>) so a network router's peers are unambiguously that region's mgmt nodes. A region layer can still consume a global group via a terragrunt dependency on prod/global → the module's external_groups input, but none do today. The root terragrunt derives stack from the full sub-path, so prod/htz-fsn1/netbird gets its own state key. (Now that the fabric stack lives in its own prod/htz-fsn1/fabric/ sub-stack, the region root carries no terragrunt.hcl, so prod/htz-fsn1/netbird uses a normal find_in_parent_folders include — the old direct-root-include workaround is gone.) Rendered NetBird object names and 1P item titles are unchanged by the rename (YUCCA_STAGING_*, NETBIRD_YUCCA_PROD_HTZ_FSN1_*): the env→partition / site→region swap keeps the same string values.

The yucca → yucca-tags access model

yucca members reach everything tagged as a yucca tag (NetBird object names are lowercase-kebab — e.g. yucca-prod-htz-fsn1-mgmt; the 1Password setup-key item titles stay UPPER_SNAKE, since CI/ansible/talos read them by op:// string):

• yucca — the existing users group (people). External (looked up by its actual name yucca); never managed here.
• yucca tags — any group flagged resource = true in a layer's groups. This marks the group yucca-reachable; it applies to peer/node groups (so a yucca member can SSH yucca-prod-htz-fsn1-mgmt, the mgmt nodes) as well as routed-subnet tags (yucca-prod-htz-fsn1-resources, which the site's netbird_network_resources are tagged into). Today every group a layer owns is flagged.

For each layer that owns ≥1 yucca tag, the netbird-env module auto-generates a <prefix>-yucca-to-resources policy (bidirectional = false) whose destinations are all of that layer's flagged groups — so flagging a group grants yucca users access to it (its peers and any tagged resources) with no policy to edit. The source is var.yucca_users_group (default yucca, looked up by name; set null to opt a layer out). bidirectional = false means yucca users only initiate — this policy never makes a tagged group a source. The union of these per-layer policies is the account-wide "yucca reaches every yucca tag we create" guarantee.

The former single shared yucca_resource tag in prod/global (one account-wide yucca → yucca_resource policy, consumed by sites via a terragrunt dependency) was retired in favour of this per-layer model — no cross-stack group reference, and the destination list is derived, not hand-maintained.

Staging additionally grants its ci group access to the existing Liberty Park infra groups (where the staging nodes live today) — those are external groups resolved by name in staging/global/netbird/main.tf.

Declarative input (netbird.auto.tfvars)

Groups, setup keys, policies and networks reference groups by logical key, never opaque NetBird IDs:

# resource = true ⇒ yucca-reachable ("yucca tag"); here every group is flagged,
# so yucca users reach all of them (SSH the nodes + the routed subnets)
groups = { ci = { resource = true }, mgmt = { resource = true },
           talos = { resource = true }, k8s_operator = { resource = true },
           resources = { resource = true } }

setup_keys = {
  ci           = { type = "reusable", ephemeral = true, auto_groups = ["ci"] }
  mgmt         = { type = "reusable", auto_groups = ["mgmt"] }
  talos        = { type = "reusable", auto_groups = ["talos"] }
  k8s_operator = { type = "reusable", auto_groups = ["k8s_operator"] }
}

policies = {
  ci-to-all = {                       # CI reaches every node group in this env
    rules = [{ name = "ci-to-all", protocol = "all"
               sources = ["ci"], destinations = ["mgmt", "talos", "k8s_operator"] }]
  }
}

NetBird is default-deny — a peer gets only the access its groups' policies grant; an empty policies map means total isolation. The yucca → resource policy is not declared here: the module generates it from every group flagged resource = true (see the access model above).

Networks (prod htz-fsn1) — CIDRs propagated, not hardcoded

The htz-fsn1 site layer exposes a NetBird Network named htz-fsn1: the mgmt group are the routing peers, and each routed subnet is a netbird_network_resource. The CIDRs are derived from the same fabric-addressing module the fabric stack uses (re-instantiated in the layer's addressing.tf — a pure, stateless module, so no duplication and no cross-stack coupling). Every resource is tagged into the site's own resources group, so access is the module-generated yucca-prod-htz-fsn1-yucca-to-resources policy. The only per-site input is the site id (the CIDRs flow from it):

site_id = 40   # mirrors prod/htz-fsn1; feeds fabric-addressing → the routed CIDRs
               #   mgmt 10.40.5.0/24 · api 10.40.10.0/24
               #   cls1_public 10.40.20.0/23 · cls1_private 10.40.22.0/23

Setup-key plaintext → 1Password. Each setup key's secret key is written to the per-env vault (yucca_tf_<env>) as item NETBIRD_<UPPERCASED_NAMESPACED_NAME>_SETUP_KEY (onepassword_item, same "TF mints secrets into 1P" pattern as the JWT keypair). The namespaced title keeps multiple prod sites writing to the one yucca_tf_prod vault from colliding.

Auth. Two providers, both fed by op run --env-file=tf/.env[.prod]:

• netbird — admin PAT from NB_PAT (op://shared_tf/NETBIRD_TF_PAT, shared across all envs; management_url defaults to NetBird Cloud).
• onepassword — OP_SERVICE_ACCOUNT_TOKEN (same session), writes the keys.

Run it (pure cloud API — no tailnet, no node contact):

TF_STACK_DIR=tf/deployment/staging/global/netbird mise run tf:init   # then tf:plan / tf:apply
# prod — global layer first, then each region layer (uses the prod env file + SA):
OP_ENV_FILE=tf/.env.prod TF_STACK_DIR=tf/deployment/prod/global/netbird   mise run tf:apply
OP_ENV_FILE=tf/.env.prod TF_STACK_DIR=tf/deployment/prod/htz-fsn1/netbird mise run tf:apply

CI (.github/workflows/infra.yml) applies staging/global/netbird in the staging matrix, and the prod layers (prod/global then prod/htz-fsn1/netbird) as gated prod-infra jobs on the prod 1P SA / tf/.env.prod. Prod CI needs the OP_TF_YUCCA_PROD_ENV[_WRITE] repo secrets + a prod-infra Environment — see the workflow header.

CI connects over NetBird

CI reaches the staging 10.10.10.0/24 nodes over the NetBird overlay (this replaced the Tailscale subnet-router path). The .github/actions/netbird-connect composite action installs the client and runs netbird up with the ci setup key read from 1P (op://yucca_tf_staging/NETBIRD_YUCCA_STAGING_CI_SETUP_KEY); the runner joins as a ci peer and the existing staging route advertises the LAN. The apply job applies staging/global/netbird first (minting that key) before connecting, so a fresh bootstrap is self-contained. The prod fabric workflow (fabric.yml) still uses Tailscale — 10.40.5.0/24 isn't on NetBird yet.

Where secrets actually live

• yucca_tf_dev (team-shared): live values consumed by Ansible at play time. Password items per cluster (ops, dashboard, grafana, S3 svc-user access + secret), SSH Key items per cluster (ansible-iac keypairs), and DR-capture items per cluster (RGW TLS cert + key, client.admin keyring — populated by mise run capture).
• yucca_tf_dev_manual (team-shared): placeholders for human-fillable secrets (API tokens, OAuth client secrets). Not yet used by ceph-cluster.

Service accounts themselves are in yucca_tf_dev as two items:

SA	Purpose	Consumed by
yucca_futo_1pass_superuser_service_account	Read + write all yucca_tf_* vaults	TF (via tf/.env)
yucca_futo_1pass_service_account	Read-only on yucca_tf and yucca_tf_dev	Ansible runtime / CI

Both are shared with other Futo consumers (o11y, base Yucca infra). Rotation affects all of them — see ansible/ceph/docs/runbooks/rotate-sa-token.md for the coordination procedure.

Adding a new cluster

1. Add an entry to clusters.auto.tfvars.
2. Create the 1P items in yucca_tf_dev:
   • Password items: <CLUSTER>_CEPH_{OPS,DASHBOARD,GRAFANA}_PASSWORD, plus <CLUSTER>_CEPH_S3_SVC_YUCCA_RESTIC_{ACCESS,SECRET}_KEY. Use op item create --generate-password for each.
   • SSH Key item: <CLUSTER>_CEPH_ANSIBLE_IAC_SSH_KEY via op item create --category "SSH Key" --ssh-generate-key=ed25519.
3. mise run tf:apply — renders inventory + secrets template.
4. Create per-node host_vars/*.yml files in the new inventory dir (hardware topology; not TF-rendered yet).
5. On operator workstation: scripts/install-ssh-keys.sh <cluster> to pull the private key from 1P.
6. After first successful deploy: mise run capture to snapshot the RGW TLS material + admin keyring to 1P for DR.
7. See ansible/ceph/docs/adding-a-cluster.md for the full walk-through.

Related

• TF-first + op inject model: ansible/ceph/docs/secrets.md
• Immich devtools (upstream pattern): https://github.com/immich-app/devtools/tree/main/tf