feat(netbird-ansible): better subnet routers (#217)

ansible/mgmt/README.md

@@ -26,9 +26,12 @@ The inventory is **TF-generated** at run time (see "Generated inventory").
    `tf/shared/modules/identity` (see "Generated inventory" below).
 3. **security** — nftables firewall, SSH hardening (no password auth,
    `PermitRootLogin prohibit-password`), unattended-upgrades.
-4. **networkd** — systemd-networkd VLAN sub-interfaces on the 25G fabric NIC.
-   **Gated on `mgmt_networkd_enabled` (default false)** — see the 25G caveat
-   below.
+4. **networkd** — the host L3 that lets these nodes *forward* the NetBird-routed
+   site subnets: the OOB 1G NIC on `10.40.5.0/24` (the switch vme) + the 25G
+   fabric VLAN sub-interfaces (public/private/api). Enabled by default
+   (`mgmt_networkd_enabled: true`); only the OOB + fabric NICs are matched, so the
+   primary public NIC is untouched. The OOB path is reliable; the 25G VLANs are
+   written but stay carrier-down until that link is up — see the 25G caveat below.
 5. **netbird** — install NetBird, `netbird up` with the `mgmt` setup key, enable
    IP forwarding (the mgmt nodes are the NetBird route peers for the site subnets;
    the routed network itself is declared in TF).
@@ -104,27 +107,28 @@ automatically. This needs the control node on the overlay too — the CI
 For the very first run after a reinstall, pass `-e mgmt_bootstrap=true` to force
 the public IP (skips any stale peer entry for the host).
 
-## 25G fabric caveat
+## Host L3 / routing & the 25G caveat
 
-The 25G fabric link (Intel E810, "ice" driver) is **currently physically
-unreliable**. The `networkd` role that configures its VLAN sub-interfaces is
-gated off by default (`mgmt_networkd_enabled: false`), so a normal `site.yml`
-run is a no-op for networking. Once the fabric links:
+These nodes are the NetBird route peers for the site, so they need L3 paths to
+the routed subnets in order to **forward** to them. The `networkd` role
+(enabled by default) configures, per the TF-rendered host_vars:
 
-1. Confirm the NIC name on each host with `ip link` (prior name:
-   `enp33s0f0np0`) and correct `mgmt_fabric_nic` in the host_vars if it
-   differs.
-2. Set `mgmt_networkd_enabled: true` (e.g. `--extra-vars` or group_vars).
+| Path | Network | mgmt-1 | mgmt-2 | NIC |
+|------|---------|--------|--------|-----|
+| OOB management (switch vme) | `10.40.5.0/24` | `10.40.5.50` | `10.40.5.51` | `enp37s0` (1G) |
+| VLAN 120 (cluster public) | `10.40.20.0/23` | `10.40.20.2` | `10.40.20.3` | `enp33s0f0np0` (25G) |
+| VLAN 122 (cluster private) | `10.40.22.0/23` | `10.40.22.2` | `10.40.22.3` | `enp33s0f0np0` (25G) |
+| VLAN 10 (api) | `10.40.10.0/24` | `10.40.10.2` | `10.40.10.3` | `enp33s0f0np0` (25G) |
 
-VLAN layout (gateways are `.1` on the leaf IRB):
+The **OOB path is 1G and reliable** — that's what carries the switch traffic.
+The **25G fabric link (Intel E810, "ice") is currently physically unreliable**;
+its VLAN sub-interfaces are written but stay carrier-down (`RequiredForOnline=no`,
+so they never block boot) until the link is up — harmless until then.
 
-| VLAN | Network | mgmt-1 | mgmt-2 |
-|------|---------|--------|--------|
-| 20 (cluster public) | `10.40.20.0/23` | `10.40.20.2` | `10.40.20.3` |
-| 22 (cluster private) | `10.40.22.0/23` | `10.40.22.2` | `10.40.22.3` |
-
-The primary public NIC keeps Hetzner's DHCP default — this tree does not touch
-it.
+After a reinstall, confirm both NIC names with `ip link` and correct
+`oob_nic` / `fabric_nic` in `tf/deployment/prod/htz-fsn1/mgmt-hosts.yaml` if they
+differ (predictable names can change). Only these NICs are matched; the primary
+public NIC keeps Hetzner's DHCP default — this tree does not touch it.
 
 ## Setup
 

ansible/mgmt/roles/networkd/defaults/main.yml

@@ -1,14 +1,20 @@
 ---
-# systemd-networkd VLAN sub-interfaces on the 25G fabric NIC.
+# systemd-networkd L3 so the node forwards the NetBird-routed site subnets:
+#   - OOB 1G NIC on the management LAN (mgmt_oob: 10.40.5.0/24 — switch vme)
+#   - tagged VLAN sub-interfaces on the 25G fabric NIC (mgmt_fabric_vlans)
 #
-# Prerequisites:
-#   - systemd-networkd present (Debian 13 base)
-#   - mgmt_fabric_nic, mgmt_fabric_vlans defined in host_vars
+# Prerequisites (from host_vars, TF-rendered): mgmt_oob, mgmt_fabric_nic,
+# mgmt_fabric_vlans. networkd only Matches these NICs — the primary public NIC
+# (Hetzner's default) is never matched, so it stays under its own management.
 #
-# Master toggle. False by default because the 25G fabric link is currently
-# unreliable — flip to true (and confirm mgmt_fabric_nic via `ip link`) once
-# the fabric is up.
-mgmt_networkd_enabled: false
+# Enabled by default: the OOB path (1G, reliable) is what lets the node reach the
+# switches. The 25G VLANs are written too but stay carrier-down until that link
+# is up (RequiredForOnline=no, so they never block) — harmless until then.
+# NICs are verified per host in mgmt-hosts.yaml; correct there if `ip link` differs.
+mgmt_networkd_enabled: true
+
+# OOB management interface { nic, address } — empty disables just the OOB part.
+mgmt_oob: {}
 
 # --- Paths ---
 mgmt_networkd_config_dir: /etc/systemd/network

ansible/mgmt/roles/networkd/tasks/deploy.yml

@@ -1,7 +1,26 @@
 ---
-# Write systemd-networkd config files for the 25G fabric VLANs.
-# Safe to re-run.
+# Write systemd-networkd config for the OOB management NIC + the 25G fabric
+# VLANs, and make sure networkd is running. Safe to re-run. Only the NICs matched
+# below are managed; the primary public NIC is left to its own manager.
 
+- name: Ensure systemd-networkd is enabled and running
+  ansible.builtin.systemd:
+    name: systemd-networkd
+    enabled: true
+    state: started
+
+# ── OOB management LAN (1G; the switch vme lives here) ──────────────────────
+- name: Deploy OOB management .network
+  ansible.builtin.template:
+    src: oob.network.j2
+    dest: "{{ mgmt_networkd_config_dir }}/10-{{ mgmt_oob.nic }}.network"
+    owner: root
+    group: root
+    mode: '0644'
+  when: mgmt_oob.nic is defined
+  notify: Reload systemd for networkd
+
+# ── 25G fabric VLAN sub-interfaces ──────────────────────────────────────────
 - name: Deploy fabric parent .network (declares VLANs, no L3 of its own)
   ansible.builtin.template:
     src: fabric.network.j2

ansible/mgmt/roles/networkd/tasks/main.yml

@@ -1,16 +1,12 @@
 ---
-# systemd-networkd VLAN sub-interfaces on the 25G fabric NIC.
+# systemd-networkd L3 so the node forwards the NetBird-routed site subnets:
+#   - OOB 1G NIC on the management LAN (mgmt_oob: 10.40.5.0/24 — switch vme)
+#   - tagged VLAN sub-interfaces on the 25G fabric NIC (public/private/api)
 #
-# NOTE: applies once the 25G fabric links — the 25G link is currently
-# physically unreliable (see notes). Gated on mgmt_networkd_enabled
-# (default: false) so a normal site.yml run is a no-op until the fabric
-# is up and the operator opts in.
-#
-#   1. write parent .network (DHCP off on fabric, declares VLANs)
-#   2. write per-VLAN .netdev + .network (static addresses)
-#   3. reload networkd
-#
-# The primary public NIC keeps Hetzner's DHCP default — untouched here.
+# Gated on mgmt_networkd_enabled (default true). The OOB path is 1G/reliable;
+# the 25G VLANs are written but stay carrier-down (RequiredForOnline=no) until
+# that link is up — harmless until then. Only the OOB + fabric NICs are Matched,
+# so the primary public NIC (Hetzner's default) is left untouched.
 
 - name: Deploy networkd configs
   ansible.builtin.import_tasks: deploy.yml

ansible/mgmt/roles/networkd/templates/oob.network.j2

@@ -0,0 +1,16 @@
+# {{ ansible_managed }}
+# OOB management LAN: {{ mgmt_oob.nic }} -> {{ mgmt_oob.address }}.
+# The switch vme (e.g. 10.40.5.115/125) lives on this L2. No gateway here — the
+# default route stays on the primary public NIC; this is a directly-attached
+# subnet. IP forwarding (enabled by the netbird role) lets this node route the
+# NetBird overlay's traffic to the switches.
+
+[Match]
+Name={{ mgmt_oob.nic }}
+
+[Network]
+Address={{ mgmt_oob.address }}
+IPv6AcceptRA=no
+
+[Link]
+RequiredForOnline=no