docs: describe the netbird mesh topology and refresh stale sections (#107)

Signed-off-by: Devin Buhl <devin@buhl.casa>

README.md

@@ -20,9 +20,10 @@ Built for geographic resilience with single-cluster operational simplicity: thre
 ```text
 deployment/modules/
 ├── ovh/account/          # cloud project, vRack, private network, CPs, workers, IPLB, DNS
-├── netbird/cluster/      # per-env mesh: node group, setup key, vRack network route, access policy
+├── netbird/cluster/      # per-env mesh: vRack route, mesh-gateway VIP + DNS, pod egress, policies
+├── netbox/cluster/       # IPAM registration of the ranges the other modules allocate
 ├── talos/cluster/        # machine secrets, CP + worker configs, bootstrap, ingress firewall
-└── kubernetes/helm/      # Flux Operator + Instance, env-scoped secrets
+└── kubernetes/helm/      # CoreDNS, Flux Operator + Instance, bootstrap-settings, env secrets
 
 kubernetes/
 ├── apps/
@@ -31,7 +32,7 @@ kubernetes/
 └── clusters/
     └── <env>/
         ├── apps.yaml             # cluster-apps entry point (the Flux Instance points here)
-        └── cluster-settings.yaml # per-env ConfigMap: APP_DOMAIN, CLUSTER_NAME
+        └── cluster-settings.yaml # per-env CLUSTER_* ConfigMap (BOOTSTRAP_* comes from Terraform)
 ```
 
 State lives in S3 under `yucca/o11y/v3/<module>/<env>`. Secrets and OVH/NetBird tokens come from 1Password via `op run` and `deployment/.env`.

docs/01-bootstrap-guide.md

@@ -31,34 +31,40 @@ export TF_VAR_env=staging
    mise run tg run --working-dir deployment/modules/ovh/account apply
    ```
 
-2. **NetBird** — the per-environment mesh objects: the Talos node group, a reusable setup key, the vRack network route (Talos nodes as routing peers), and the `yucca → resource` access policy. The Talos module consumes the setup key from here, so apply NetBird first.
+2. **NetBird** — the per-environment mesh objects (all named `o11y-<env>-*`): groups and setup keys for the Talos nodes and the in-cluster routing peers, the vRack network route, the mesh-gateway VIP resource and DNS zone, the pod-egress network, and the access policies (`yucca → resource`, `yucca → gateway`, `talos → bootstrap opc`). The Talos module consumes the node setup key and mesh zone from here, so apply NetBird first.
 
    ```bash
    mise run tg run --working-dir deployment/modules/netbird/cluster apply
    ```
 
-3. **Talos (bootstrap)** — initial bring-up over public IPs, because the NetBird extension isn't running yet.
+3. **NetBox** — registers the environment's ranges (vRack, gateway ServiceCIDR, pod-egress CIDR) in IPAM, from the same values the other modules allocate.
+
+   ```bash
+   mise run tg run --working-dir deployment/modules/netbox/cluster apply
+   ```
+
+4. **Talos (bootstrap)** — initial bring-up over public IPs, because the NetBird extension isn't running yet.
 
    ```bash
    TF_VAR_use_public_endpoints=true mise run tg run --working-dir deployment/modules/talos/cluster apply
    ```
 
-4. **Verify** the cluster is up and operator-side NetBird routing works. Pull the configs (see [Cluster access](#cluster-access)) and hit the APIs over the NetBird network:
+5. **Verify** the cluster is up and operator-side NetBird routing works. Pull the configs (see [Cluster access](#cluster-access)) and hit the APIs over the NetBird network — use the **direct** kubeconfig context during bring-up, since the default context targets the mesh gateway, which only exists once Flux has reconciled:
 
    ```bash
    mise run talos:kubeconfig && mise run talos:talosconfig
-   kubectl --kubeconfig .private/$ENVIRONMENT/kubeconfig get nodes -o wide
+   kubectl --kubeconfig .private/$ENVIRONMENT/kubeconfig --context o11y-$ENVIRONMENT-direct get nodes -o wide
    talosctl --talosconfig .private/$ENVIRONMENT/talosconfig -n 10.150.200.10 get members
    ```
 
-5. **Talos (steady state)** — drop the public-endpoints override now that NetBird routes work; the host firewall closes the public NIC (everything except `:30443` on workers).
+6. **Talos (steady state)** — drop the public-endpoints override now that NetBird routes work; the host firewall closes the public NIC (everything except `:30443` on workers).
 
    ```bash
    unset TF_VAR_use_public_endpoints
    mise run tg run --working-dir deployment/modules/talos/cluster apply
    ```
 
-6. **Kubernetes/Helm** — install the Flux Operator + Instance and create bootstrap secrets (cert-manager, OVH DNS credentials, external-secrets 1Password token). After this, Flux owns cluster state.
+7. **Kubernetes/Helm** — install CoreDNS (Terraform-seeded — Flux needs cluster DNS from its first reconcile; Talos's copy is disabled), the Flux Operator + Instance, the `bootstrap-settings` ConfigMap, and the bootstrap secrets (cert-manager OVH DNS credentials, the 1Password Connect token for external-secrets). After this, Flux owns cluster state.
 
    ```bash
    mise run tg run --working-dir deployment/modules/kubernetes/helm apply

docs/02-infrastructure-architecture-guide.md

@@ -42,11 +42,18 @@ The worker host firewall scopes `:30443` to OVH's IPLB NAT range (`10.108.0.0/14
 
 Because the farm targets the workers' public IPs (not the vRack), three things are required and are handled in the cluster config: NodePorts must answer on the public NIC, exactly one Envoy must run per worker, and Envoy must parse PROXY protocol. See the cluster architecture guide for those details.
 
-## Operator access (NetBird)
+## NetBird mesh
 
-NetBird runs as a Talos system extension on **every** node, so operators reach `talosctl` and `kubectl` over the NetBird network without exposing those APIs publicly. The vRack subnet is published as a NetBird network route with the Talos nodes as routing peers — any node can route, so it's HA — and operator traffic is masqueraded to the routing peer's vRack IP, which the host firewall already trusts. A per-environment access policy lets the shared `yucca` operator group reach this environment's routed subnet on the management ports only (apid `50000`, kube-apiserver `6443`); the groups and policy are environment-scoped (`O11Y_STAGING_*` vs `O11Y_PRODUCTION_*`), so staging operators can't pivot into production.
+NetBird connects operators, other FUTO clusters, and the bootstrap cluster to this environment without exposing anything publicly. Everything is Terraform-managed (`netbird/cluster`, objects named `o11y-<env>-*`) and environment-scoped — separate groups, networks, and policies per environment, so staging access can't pivot into production. Four building blocks:
 
-Operators point `kubectl`/`talosctl` at a specific control plane's static private IP — not the floating VIP, since cross-DC ARP for the VIP over the NetBird network route is unreliable. The VIP remains the in-cluster apiserver endpoint used by kubelet and other in-cluster components.
+* **Node mesh (operator access).** NetBird runs as a Talos system extension on every node, and the vRack subnet is advertised as a network route with the Talos nodes as routing peers — any node can route, so it's HA. Operator traffic arrives masqueraded to the routing peer's vRack IP, which the host firewall already trusts. A policy grants the shared `yucca` operator group the management ports only (apid `50000`, kube-apiserver `6443`). This is the path `talosctl` and the Terraform providers use.
+* **Workload ingress (mesh gateway).** In-cluster `netbird-router` pods are the routing peers for a pinned Envoy gateway VIP — a ClusterIP from a dedicated secondary ServiceCIDR, advertised as a `/32` resource. The pods exist because only pod-level routing can advertise a ClusterIP (kube-proxy's DNAT runs in the host netns). A NetBird DNS zone resolves `*.<mesh-domain>` to the VIP for mesh peers; the `yucca` group is allowed `:443` (mesh-facing HTTPRoutes) and `:6443` — the HA kube-apiserver endpoint `kube.<mesh-domain>`, which `kubectl` uses by default: it load-balances across every apiserver and never hairpins through a routing peer.
+* **Pod egress (Multus).** Pods can't normally originate mesh traffic — NetBird only masquerades traffic sourced from ranges a peer advertises, and the flannel pod CIDR isn't one. Pods that need the mesh (today: the external-secrets controller, reaching the bootstrap cluster's 1Password Connect at `opc.o11y.futo.network`) opt in via a Multus `NetworkAttachmentDefinition`: a second interface in an egress range the nodes advertise, with a route scoped to just the opc VIP. The node's own NetBird carries it out; everything else stays on flannel.
+* **Mesh DNS.** The router pods also serve NetBird DNS to the cluster: CoreDNS forwards the `futo.network` zone to a pinned `netbird-dns` Service in front of them, so any pod resolves mesh names (opc, mesh gateways) through ordinary cluster DNS with zero per-pod configuration.
+
+All the ranges involved — the vRack, the gateway ServiceCIDR, and the egress CIDR — are registered in NetBox by the `netbox/cluster` module from the same Terraform values that allocate them.
+
+Operators point `talosctl` at a control plane's static private IP over the node mesh (not the floating VIP — cross-DC ARP for the VIP is unreliable over the route; it remains the in-cluster apiserver endpoint). `kubectl` defaults to `kube.<mesh-domain>` through the mesh gateway, with a direct-CP break-glass context in the same kubeconfig for bootstrap/DR.
 
 ## Cost
 
@@ -69,5 +76,8 @@ Staging + production run-rate ≈ **$955/mo** plus the one-time **$221** product
 | Workers | 3× `SYS-2` (`24sys022`) | 3× `Rise-2` (`24rise02-v1`) |
 | IPLB | 1 zone (`gra`) | 3 zones (`gra` + `rbx` + `sbg`), anycast |
 | Private CIDR | `10.150.200.0/24` | `10.150.100.0/24` |
-| NetBird objects | `O11Y_STAGING_*` | `O11Y_PRODUCTION_*` |
+| Mesh domain | `staging.o11y.futo.network` | `o11y.futo.network` |
+| Gateway ServiceCIDR (VIP `.10`) | `10.69.1.0/24` | `10.69.0.0/24` |
+| Pod egress CIDR | `10.69.3.0/24` | `10.69.2.0/24` |
+| NetBird objects | `o11y-staging-*` | `o11y-production-*` |
 | Flux source | `staging` overlay | `production` overlay |

docs/03-cluster-architecture-guide.md

@@ -4,7 +4,7 @@ How the cluster itself is built: the Talos operating system, the Kubernetes laye
 
 ## Talos
 
-Talos Linux on every node, with flannel CNI and kube-proxy in nftables mode.
+Talos Linux on every node, with flannel CNI (wrapped by Multus for opt-in secondary pod interfaces) and kube-proxy in nftables mode.
 
 ### Image schematics
 
@@ -19,7 +19,9 @@ Control planes and workers use **different** Talos Factory schematics, on purpos
 
 ### Machine configuration
 
-* **Control-plane endpoint** is the floating VIP (`10.150.200.5`); the apiserver cert SANs include the VIP and every CP private IP, so operators can reach the API on any individual CP when the VIP doesn't ARP across DCs.
+* **Control-plane endpoint** is the floating VIP (`10.150.200.5`); the apiserver cert SANs include the VIP, every CP private IP (the direct/break-glass path), and `kube.<mesh-domain>` — the HA endpoint the mesh gateway fronts via TLS passthrough, which `kubectl` uses by default.
+* **CoreDNS is Terraform-seeded**, not Flux-managed: Flux itself needs cluster DNS from its first reconcile, so a fresh bootstrap would deadlock. Talos's own CoreDNS is disabled and the chart (installed by `kubernetes/helm`) owns the `kube-dns` Service; kubelet's `clusterDNS` pins its IP. The Corefile adds a `futo.network` zone forwarded to the NetBird mesh DNS (see the infrastructure guide).
+* **kubelet's node IP is pinned to the vRack subnet** — otherwise kubelet auto-detects, and a lower-sorting host address (such as the Multus egress bridge) steals the node's InternalIP and breaks apiserver→kubelet traffic.
 * **Component metrics** for kube-controller-manager, kube-scheduler, and etcd bind to all interfaces rather than localhost, so VMAgent (running on a worker) can scrape them. The host firewall keeps these ports private. The controller-manager and scheduler endpoints are authenticated HTTPS; etcd's is plain HTTP, so the firewall is its only protection.
 * **kube-proxy** is told to answer NodePorts on every interface (not just the node's primary vRack IP), so the IPLB can reach Envoy on the workers' public NIC; its own metrics endpoint is likewise bound for scraping. These settings are generated into the cluster-wide kube-proxy DaemonSet.
 * **Flannel's VXLAN endpoint** is pinned to the vRack interface; otherwise it defaults to the public NIC and the host firewall drops east-west pod traffic.
@@ -53,7 +55,7 @@ Operator `talosctl`/`kubectl` traffic needs no rule of its own: it arrives over
 
 ## Kubernetes
 
-Kubernetes with flannel CNI and kube-proxy in nftables mode. Spegel runs as a peer-to-peer image registry mirror so each node's containerd pulls layers from its peers before the upstream registry (this requires `discard_unpacked_layers = false` in the worker containerd config).
+Kubernetes with flannel CNI and kube-proxy in nftables mode. **Multus** runs as a meta-CNI wrapping the flannel config: pods annotated with `k8s.v1.cni.cncf.io/networks` get extra interfaces from `NetworkAttachmentDefinition`s (today just `netbird-egress`, the mesh egress leg — see the infrastructure guide); unannotated pods are untouched. Spegel runs as a peer-to-peer image registry mirror so each node's containerd pulls layers from its peers before the upstream registry (this requires `discard_unpacked_layers = false` in the worker containerd config).
 
 **Control-plane scraping** is wired end to end: the component metrics endpoints are bound off localhost (above), the host firewall scopes them to the vRack and pod CIDR, and VMAgent scrapes kube-controller-manager, kube-scheduler, etcd, kube-proxy, and the node-exporter DaemonSet.
 
@@ -67,15 +69,13 @@ Everything above the OS is managed by Flux v2. Manifests are organized as reusab
 
 ### Version pinning
 
-Chart (and the CloudNativePG Postgres image) versions are pinned per environment in the overlay Kustomization patches, so a version can be promoted in staging and soaked before production moves. Versions are renovate-managed in those patches; component versions are not documented here because they change continuously — the manifests are the source of truth.
+Chart (and the CloudNativePG Postgres image) versions are pinned per environment in the overlay Kustomization patches, so a version can be promoted in staging and soaked before production moves. Staging rides `base/` directly; production pins via patches. OCI chart refs pin a **tag and its digest** — Flux gives the digest precedence, so the production patches must carry both or a base digest would silently override the env pin; a renovate custom manager keeps each tag+digest pair in lockstep, and the built-in flux manager maintains the pairs in `base/`. Component versions are not documented here because they change continuously — the manifests are the source of truth.
 
 ### Configuration substitution
 
-Per-environment values are **not** hardcoded in `base/` and **not** patched into each overlay. Instead, `cluster-apps` carries a single patch that targets every child Kustomization and injects a `postBuild.substituteFrom` pointing at the `cluster-settings` ConfigMap. Flux's envsubst then resolves placeholders at apply time. The variables:
+Per-environment values are **not** hardcoded in `base/` and **not** patched into each overlay. Instead, `cluster-apps` carries a single patch that targets every child Kustomization and injects `postBuild.substituteFrom` pointing at **two** ConfigMaps, resolved by Flux's envsubst at apply time — the prefix tells you who owns the value:
 
-| Variable | Staging | Consumed by |
+* **`cluster-settings`** (`CLUSTER_*`, committed in `kubernetes/clusters/<env>/`) — git-owned values: `CLUSTER_APP_DOMAIN`, `CLUSTER_NAME`, the 1Password vault names, VictoriaMetrics retention/storage class, and the bootstrap Connect VIP.
-|----------|---------|-------------|
+* **`bootstrap-settings`** (`BOOTSTRAP_*`, created in-cluster by the `kubernetes/helm` Terraform module) — Terraform-owned values that must never drift from the infrastructure: the mesh DNS zone and the NetBird gateway VIP, ServiceCIDR, and egress CIDR/gateway. The entry is `optional` because offline renderers (flate CI) can't see an in-cluster-only ConfigMap; consumers still fail loudly at apply if it's genuinely missing.
-| `APP_DOMAIN` | `staging.futostatus.com` | cert `dnsNames`; HTTPRoute hostnames (Grafana, vmauth, echo) |
-| `CLUSTER_NAME` | `o11y-staging` | VictoriaMetrics `externalLabels.cluster` |
 
-Because of this, values that vary by environment (domains, cluster name) live once in `base/` with a placeholder rather than being duplicated across overlays — only versions are still patched per environment. A rendered object can be checked exactly as Flux will produce it using the `flate` CLI.
+Because of this, values that vary by environment live exactly once — in git or in Terraform — rather than being duplicated across overlays; only versions are still patched per environment. A rendered object can be checked exactly as Flux will produce it using the `flate` CLI.

docs/04-application-architecture-guide.md

@@ -8,18 +8,22 @@ The workloads running on the cluster — the ingress edge, the observability sta
 
 Envoy Gateway is the only external ingress. It runs one replica per worker with a hostname topology-spread constraint, so every IPLB backend has a local endpoint under `externalTrafficPolicy: Local`. The OVH load balancer does TCP passthrough to Envoy's NodePort; TLS terminates at Envoy. A client traffic policy parses PROXY protocol v2 (which the IPLB prepends) as optional, so the LB's bare-TCP health probe isn't reset while real client connections still surface the true source IP. Platform services attach to the gateway through HTTPRoutes (Grafana, vmauth, the echo test app).
 
+### Mesh gateway
+
+A second Envoy Gateway (`mesh`, in `envoy-system`) serves NetBird peers instead of the public internet; it hangs off a pinned VIP Service the mesh advertises (see the infrastructure guide's NetBird section). It terminates TLS with a `*.<mesh-domain>` wildcard from the same cert-manager pipeline for mesh-facing HTTPRoutes — an unauthenticated vmauth at `vmauth.<mesh-domain>` gives other FUTO clusters a remote-write path that never leaves the mesh — and carries a TLS-**passthrough** listener on `:6443` fronting the kube-apiserver as `kube.<mesh-domain>`: SNI-routed to the `kubernetes` Service, apiserver's own certificate end-to-end, load-balanced across all three control planes.
+
 ### TLS certificates
 
-cert-manager issues short-lived ECDSA P-256 wildcard certificates with always-rotate, using Let's Encrypt with the OVH DNS-01 challenge webhook. The certificate `dnsNames` are defined once in the base manifests using the `APP_DOMAIN` placeholders and resolve per environment from `cluster-settings` — staging gets `*.staging.futostatus.com`, production the bare-domain wildcards.
+cert-manager issues short-lived ECDSA P-256 wildcard certificates with always-rotate, using Let's Encrypt with the OVH DNS-01 challenge webhook. The certificate `dnsNames` are defined once in the base manifests using the `CLUSTER_APP_DOMAIN` placeholder and resolve per environment from `cluster-settings` — staging gets `*.staging.futostatus.com`, production the bare-domain wildcards.
 
 ## VictoriaMetrics — the central metrics store
 
 This cluster's VictoriaMetrics is the **central metrics store for all FUTO clusters**. Other Kubernetes clusters each run their own `vmagent` and remote-write into this cluster; it is the ingestion target plus the query and alerting brain for everyone.
 
-* **Storage** — VMCluster mode with `replicationFactor=2`, `vmstorage` spread one-per-worker across the three DCs on `openebs-spare-disk` with 90-day retention. The `vmstorage`, `vminsert`, and `vmselect` tiers scale independently.
+* **Storage** — VMCluster mode with `replicationFactor=2`, `vmstorage` spread one-per-worker across the three DCs on `openebs-spare-disk`; retention is set per environment via `CLUSTER_VMETRICS_RETENTION` (30d staging, 120d production). The `vmstorage`, `vminsert`, and `vmselect` tiers scale independently.
 * **Local collection** — a `vmagent` (with a persistent disk buffer) scrapes this cluster and remote-writes to the local `vminsert`. It tags series with the cluster's identity.
 * **Alerting** — `vmalert` evaluates rules; notifications are blackholed for now (no Alertmanager yet), so rules still evaluate and recording rules still write.
-* **Ingestion gateway** — a locked-down `vmauth` (no anonymous access, run as an HA pair) fronts `vminsert` and is exposed publicly at `vmauth.<APP_DOMAIN>` through the Envoy Gateway and IPLB with cert-manager TLS.
+* **Ingestion gateway** — a locked-down `vmauth` (no anonymous access, run as an HA pair) fronts `vminsert` and is exposed publicly at `vmauth.<CLUSTER_APP_DOMAIN>` through the Envoy Gateway and IPLB with cert-manager TLS.
 
 ### Tenancy and auth
 
@@ -37,9 +41,13 @@ Nothing changes on the central side. On the remote cluster: pull the shared toke
 
 The central store is a single point of failure for all observability, mitigated by the per-remote disk buffers, the RF=2 / three-DC resilience, and meta-monitoring that must live **outside** this cluster (it can't watch itself). Total load scales with the sum of each cluster's active series, so grow the storage and insert/select tiers as clusters onboard and add cardinality guardrails so one misbehaving remote can't overwhelm the store.
 
+## VictoriaLogs
+
+Logs follow the same shape as metrics: a **VictoriaLogs cluster** (`victoria-logs-cluster` chart) with three `vlstorage` replicas spread one-per-worker on `CLUSTER_VMLOGS_STORAGE_CLASS` (`openebs-spare-disk` staging, `openebs-spare-disk-2` production), retention per environment via `CLUSTER_VMLOGS_RETENTION` (30d staging, 120d production). A `victoria-logs-collector` DaemonSet tails this cluster's pod logs and writes them in, tagged with the cluster identity.
+
 ## Grafana
 
-Grafana runs as a 3-replica HA deployment managed by the Grafana operator, with non-blocking rolling updates (zero surge, one unavailable) and one replica per worker via a topology-spread constraint. Pod storage is ephemeral — **all state lives in Postgres** — and the replicas share a single security secret key (from 1Password) so signed cookies and sessions validate on any replica. Grafana connects to its Postgres over TLS; the password is supplied as an environment variable rather than written into config. It is reached at `grafana.<APP_DOMAIN>` and the alternate domain.
+Grafana runs as a 3-replica HA deployment managed by the Grafana operator, with non-blocking rolling updates (zero surge, one unavailable) and one replica per worker via a topology-spread constraint. Pod storage is ephemeral — **all state lives in Postgres** — and the replicas share a single security secret key (from 1Password) so signed cookies and sessions validate on any replica. Grafana connects to its Postgres over TLS; the password is supplied as an environment variable rather than written into config. It is reached at `grafana.<CLUSTER_APP_DOMAIN>` and the alternate domain.
 
 ## CloudNativePG
 
@@ -47,7 +55,8 @@ Grafana's database is a CloudNativePG cluster. The operator runs in the `cnpg-sy
 
 ## Supporting components
 
-* **external-secrets + 1Password Connect** — sync 1Password items into Kubernetes Secrets through a cluster secret store; nearly every app above gets its credentials this way.
+* **external-secrets** — syncs 1Password items into Kubernetes Secrets through cluster secret stores backed by the **bootstrap cluster's** 1Password Connect (`opc.o11y.futo.network`), reached over the NetBird mesh: the controller pod carries a Multus egress interface and resolves the endpoint via mesh DNS. Nearly every app above gets its credentials this way; the auth token is Terraform-seeded.
+* **Multus** — meta-CNI providing opt-in secondary pod interfaces; today only the `netbird-egress` attachment used by external-secrets.
 * **grafana-operator** — manages the Grafana instance plus dashboard and datasource resources, which VictoriaMetrics' chart provisions.
 * **prometheus-operator CRDs** — the ServiceMonitor/PrometheusRule CRDs the VM stack consumes.
 * **OpenEBS** — the local-hostpath provisioner backing the `openebs-system-disk` and `openebs-spare-disk` StorageClasses.