mirror of
https://github.com/immich-app/yucca.git
synced 2026-09-30 13:33:00 +08:00
feat(netbird-ansible): better subnet routers (#217)
This commit is contained in:
Antoine Lecompte
+24
-20
@@ -26,9 +26,12 @@ The inventory is **TF-generated** at run time (see "Generated inventory").
|
||||
`tf/shared/modules/identity` (see "Generated inventory" below).
|
||||
3. **security** — nftables firewall, SSH hardening (no password auth,
|
||||
`PermitRootLogin prohibit-password`), unattended-upgrades.
|
||||
4. **networkd** — systemd-networkd VLAN sub-interfaces on the 25G fabric NIC.
|
||||
**Gated on `mgmt_networkd_enabled` (default false)** — see the 25G caveat
|
||||
below.
|
||||
4. **networkd** — the host L3 that lets these nodes *forward* the NetBird-routed
|
||||
site subnets: the OOB 1G NIC on `10.40.5.0/24` (the switch vme) + the 25G
|
||||
fabric VLAN sub-interfaces (public/private/api). Enabled by default
|
||||
(`mgmt_networkd_enabled: true`); only the OOB + fabric NICs are matched, so the
|
||||
primary public NIC is untouched. The OOB path is reliable; the 25G VLANs are
|
||||
written but stay carrier-down until that link is up — see the 25G caveat below.
|
||||
5. **netbird** — install NetBird, `netbird up` with the `mgmt` setup key, enable
|
||||
IP forwarding (the mgmt nodes are the NetBird route peers for the site subnets;
|
||||
the routed network itself is declared in TF).
|
||||
@@ -104,27 +107,28 @@ automatically. This needs the control node on the overlay too — the CI
|
||||
For the very first run after a reinstall, pass `-e mgmt_bootstrap=true` to force
|
||||
the public IP (skips any stale peer entry for the host).
|
||||
|
||||
## 25G fabric caveat
|
||||
## Host L3 / routing & the 25G caveat
|
||||
|
||||
The 25G fabric link (Intel E810, "ice" driver) is **currently physically
|
||||
unreliable**. The `networkd` role that configures its VLAN sub-interfaces is
|
||||
gated off by default (`mgmt_networkd_enabled: false`), so a normal `site.yml`
|
||||
run is a no-op for networking. Once the fabric links:
|
||||
These nodes are the NetBird route peers for the site, so they need L3 paths to
|
||||
the routed subnets in order to **forward** to them. The `networkd` role
|
||||
(enabled by default) configures, per the TF-rendered host_vars:
|
||||
|
||||
1. Confirm the NIC name on each host with `ip link` (prior name:
|
||||
`enp33s0f0np0`) and correct `mgmt_fabric_nic` in the host_vars if it
|
||||
differs.
|
||||
2. Set `mgmt_networkd_enabled: true` (e.g. `--extra-vars` or group_vars).
|
||||
| Path | Network | mgmt-1 | mgmt-2 | NIC |
|
||||
|------|---------|--------|--------|-----|
|
||||
| OOB management (switch vme) | `10.40.5.0/24` | `10.40.5.50` | `10.40.5.51` | `enp37s0` (1G) |
|
||||
| VLAN 120 (cluster public) | `10.40.20.0/23` | `10.40.20.2` | `10.40.20.3` | `enp33s0f0np0` (25G) |
|
||||
| VLAN 122 (cluster private) | `10.40.22.0/23` | `10.40.22.2` | `10.40.22.3` | `enp33s0f0np0` (25G) |
|
||||
| VLAN 10 (api) | `10.40.10.0/24` | `10.40.10.2` | `10.40.10.3` | `enp33s0f0np0` (25G) |
|
||||
|
||||
VLAN layout (gateways are `.1` on the leaf IRB):
|
||||
The **OOB path is 1G and reliable** — that's what carries the switch traffic.
|
||||
The **25G fabric link (Intel E810, "ice") is currently physically unreliable**;
|
||||
its VLAN sub-interfaces are written but stay carrier-down (`RequiredForOnline=no`,
|
||||
so they never block boot) until the link is up — harmless until then.
|
||||
|
||||
| VLAN | Network | mgmt-1 | mgmt-2 |
|
||||
|------|---------|--------|--------|
|
||||
| 20 (cluster public) | `10.40.20.0/23` | `10.40.20.2` | `10.40.20.3` |
|
||||
| 22 (cluster private) | `10.40.22.0/23` | `10.40.22.2` | `10.40.22.3` |
|
||||
|
||||
The primary public NIC keeps Hetzner's DHCP default — this tree does not touch
|
||||
it.
|
||||
After a reinstall, confirm both NIC names with `ip link` and correct
|
||||
`oob_nic` / `fabric_nic` in `tf/deployment/prod/htz-fsn1/mgmt-hosts.yaml` if they
|
||||
differ (predictable names can change). Only these NICs are matched; the primary
|
||||
public NIC keeps Hetzner's DHCP default — this tree does not touch it.
|
||||
|
||||
## Setup
|
||||
|
||||
|
||||
@@ -1,14 +1,20 @@
|
||||
---
|
||||
# systemd-networkd VLAN sub-interfaces on the 25G fabric NIC.
|
||||
# systemd-networkd L3 so the node forwards the NetBird-routed site subnets:
|
||||
# - OOB 1G NIC on the management LAN (mgmt_oob: 10.40.5.0/24 — switch vme)
|
||||
# - tagged VLAN sub-interfaces on the 25G fabric NIC (mgmt_fabric_vlans)
|
||||
#
|
||||
# Prerequisites:
|
||||
# - systemd-networkd present (Debian 13 base)
|
||||
# - mgmt_fabric_nic, mgmt_fabric_vlans defined in host_vars
|
||||
# Prerequisites (from host_vars, TF-rendered): mgmt_oob, mgmt_fabric_nic,
|
||||
# mgmt_fabric_vlans. networkd only Matches these NICs — the primary public NIC
|
||||
# (Hetzner's default) is never matched, so it stays under its own management.
|
||||
#
|
||||
# Master toggle. False by default because the 25G fabric link is currently
|
||||
# unreliable — flip to true (and confirm mgmt_fabric_nic via `ip link`) once
|
||||
# the fabric is up.
|
||||
mgmt_networkd_enabled: false
|
||||
# Enabled by default: the OOB path (1G, reliable) is what lets the node reach the
|
||||
# switches. The 25G VLANs are written too but stay carrier-down until that link
|
||||
# is up (RequiredForOnline=no, so they never block) — harmless until then.
|
||||
# NICs are verified per host in mgmt-hosts.yaml; correct there if `ip link` differs.
|
||||
mgmt_networkd_enabled: true
|
||||
|
||||
# OOB management interface { nic, address } — empty disables just the OOB part.
|
||||
mgmt_oob: {}
|
||||
|
||||
# --- Paths ---
|
||||
mgmt_networkd_config_dir: /etc/systemd/network
|
||||
|
||||
@@ -1,7 +1,26 @@
|
||||
---
|
||||
# Write systemd-networkd config files for the 25G fabric VLANs.
|
||||
# Safe to re-run.
|
||||
# Write systemd-networkd config for the OOB management NIC + the 25G fabric
|
||||
# VLANs, and make sure networkd is running. Safe to re-run. Only the NICs matched
|
||||
# below are managed; the primary public NIC is left to its own manager.
|
||||
|
||||
- name: Ensure systemd-networkd is enabled and running
|
||||
ansible.builtin.systemd:
|
||||
name: systemd-networkd
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
# ── OOB management LAN (1G; the switch vme lives here) ──────────────────────
|
||||
- name: Deploy OOB management .network
|
||||
ansible.builtin.template:
|
||||
src: oob.network.j2
|
||||
dest: "{{ mgmt_networkd_config_dir }}/10-{{ mgmt_oob.nic }}.network"
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
when: mgmt_oob.nic is defined
|
||||
notify: Reload systemd for networkd
|
||||
|
||||
# ── 25G fabric VLAN sub-interfaces ──────────────────────────────────────────
|
||||
- name: Deploy fabric parent .network (declares VLANs, no L3 of its own)
|
||||
ansible.builtin.template:
|
||||
src: fabric.network.j2
|
||||
|
||||
@@ -1,16 +1,12 @@
|
||||
---
|
||||
# systemd-networkd VLAN sub-interfaces on the 25G fabric NIC.
|
||||
# systemd-networkd L3 so the node forwards the NetBird-routed site subnets:
|
||||
# - OOB 1G NIC on the management LAN (mgmt_oob: 10.40.5.0/24 — switch vme)
|
||||
# - tagged VLAN sub-interfaces on the 25G fabric NIC (public/private/api)
|
||||
#
|
||||
# NOTE: applies once the 25G fabric links — the 25G link is currently
|
||||
# physically unreliable (see notes). Gated on mgmt_networkd_enabled
|
||||
# (default: false) so a normal site.yml run is a no-op until the fabric
|
||||
# is up and the operator opts in.
|
||||
#
|
||||
# 1. write parent .network (DHCP off on fabric, declares VLANs)
|
||||
# 2. write per-VLAN .netdev + .network (static addresses)
|
||||
# 3. reload networkd
|
||||
#
|
||||
# The primary public NIC keeps Hetzner's DHCP default — untouched here.
|
||||
# Gated on mgmt_networkd_enabled (default true). The OOB path is 1G/reliable;
|
||||
# the 25G VLANs are written but stay carrier-down (RequiredForOnline=no) until
|
||||
# that link is up — harmless until then. Only the OOB + fabric NICs are Matched,
|
||||
# so the primary public NIC (Hetzner's default) is left untouched.
|
||||
|
||||
- name: Deploy networkd configs
|
||||
ansible.builtin.import_tasks: deploy.yml
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
# {{ ansible_managed }}
|
||||
# OOB management LAN: {{ mgmt_oob.nic }} -> {{ mgmt_oob.address }}.
|
||||
# The switch vme (e.g. 10.40.5.115/125) lives on this L2. No gateway here — the
|
||||
# default route stays on the primary public NIC; this is a directly-attached
|
||||
# subnet. IP forwarding (enabled by the netbird role) lets this node route the
|
||||
# NetBird overlay's traffic to the switches.
|
||||
|
||||
[Match]
|
||||
Name={{ mgmt_oob.nic }}
|
||||
|
||||
[Network]
|
||||
Address={{ mgmt_oob.address }}
|
||||
IPv6AcceptRA=no
|
||||
|
||||
[Link]
|
||||
RequiredForOnline=no
|
||||
@@ -2,13 +2,19 @@
|
||||
# BOTH the reprovision stack (mgmt.tf, uses server_number) and the ansible
|
||||
# inventory render (tf/render/ansible-mgmt, uses everything else).
|
||||
#
|
||||
# site_id — drives addressing (must match the stack's var.site_id).
|
||||
# cluster_id — the cluster whose public/private VLANs these hosts sit on.
|
||||
# host_index — host's offset within each VLAN /23 (gateway is .1 on the leaf);
|
||||
# .2/.3 here -> 10.40.20.2/.3 (public), 10.40.22.2/.3 (private).
|
||||
# fabric_nic — 25G NIC carrying the tagged VLAN sub-interfaces (verify after
|
||||
# reprovision; predictable name may differ on fresh Debian 13).
|
||||
# subnet_router — advertises the mgmt /24 over Tailscale (exactly one host).
|
||||
# site_id — drives addressing (must match the stack's var.site_id).
|
||||
# cluster_id — the cluster whose public/private VLANs these hosts sit on.
|
||||
# host_index — host's offset within each fabric VLAN (gateway is .1 on the leaf);
|
||||
# .2/.3 here -> 10.40.20.2/.3 (public), 10.40.22.2/.3 (private),
|
||||
# 10.40.10.2/.3 (api).
|
||||
# fabric_nic — 25G NIC carrying the tagged cluster/api VLAN sub-interfaces.
|
||||
# oob_nic — 1G NIC on the OOB management LAN (10.40.5.0/24, the switch vme).
|
||||
# oob_host — host octet on the OOB LAN (mgmt-1 .50, mgmt-2 .51).
|
||||
#
|
||||
# These hosts are the NetBird route peers for the site (see prod/htz-fsn1/netbird);
|
||||
# the L3 paths configured from this roster (ansible networkd role) are what let
|
||||
# them actually forward to the routed subnets. VERIFY both NIC names with
|
||||
# `ip link` after a reinstall — predictable names can differ on fresh Debian 13.
|
||||
site_id: 40
|
||||
cluster_id: 1
|
||||
|
||||
@@ -18,10 +24,12 @@ hosts:
|
||||
public_ip: 178.63.124.40
|
||||
host_index: 2
|
||||
fabric_nic: enp33s0f0np0
|
||||
subnet_router: true
|
||||
oob_nic: enp37s0
|
||||
oob_host: 50
|
||||
htz-fsn-mgmt-2:
|
||||
server_number: 3008209
|
||||
public_ip: 178.63.124.41
|
||||
host_index: 3
|
||||
fabric_nic: enp33s0f0np0
|
||||
subnet_router: false
|
||||
oob_nic: enp37s0
|
||||
oob_host: 51
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Render the ansible/mgmt inventory for a site from Terraform's sources of truth:
|
||||
# • fabric-addressing — VLAN ids + per-host VLAN addresses
|
||||
# • identity — server login users (+ keys + sudo)
|
||||
# • mgmt-hosts.yaml — the host roster (IPs, NIC, host index, subnet router)
|
||||
# • mgmt-hosts.yaml — the host roster (IPs, NICs, host index, OOB octet)
|
||||
#
|
||||
# Lightweight by design (local provider only, no backend/secrets), so it can run
|
||||
# just-in-time in the ansible CI job (and locally) via `mise run mgmt:ansible`.
|
||||
@@ -14,6 +14,8 @@ locals {
|
||||
|
||||
pub_mask = split("/", module.addressing.public_cidr)[1]
|
||||
priv_mask = split("/", module.addressing.private_cidr)[1]
|
||||
api_mask = split("/", module.addressing.api_cidr)[1]
|
||||
mgmt_mask = split("/", module.addressing.mgmt_cidr)[1]
|
||||
|
||||
header = "# GENERATED by `mise run mgmt:render-inventory` (tf/render/ansible-mgmt).\n# Do not edit — edit the Terraform sources (mgmt-hosts.yaml, fabric-addressing, identity).\n"
|
||||
}
|
||||
@@ -42,20 +44,25 @@ resource "local_file" "hosts" {
|
||||
})}"
|
||||
}
|
||||
|
||||
# Per-host: 25G NIC + the tagged VLAN sub-interfaces (ids + addresses from the
|
||||
# addressing module) + the subnet-router's advertised route.
|
||||
# Per-host L3 so the node can forward the NetBird-routed site subnets:
|
||||
# • OOB 1G NIC on the management LAN (10.40.5.0/24 — the switch vme)
|
||||
# • tagged VLAN sub-interfaces on the 25G NIC (public/private/api)
|
||||
# Addresses/ids come from the addressing module; bootstrap is the public IP
|
||||
# (site.yml reconnects over NetBird once the host joins).
|
||||
resource "local_file" "host_vars" {
|
||||
for_each = local.hosts
|
||||
filename = "${local.inv_dir}/host_vars/${each.key}.yml"
|
||||
content = "${local.header}${yamlencode({
|
||||
# Bootstrap address — site.yml reconnects over NetBird once the host joins.
|
||||
# (NetBird routes the site subnets via the mgmt peer group; that's declared in
|
||||
# tf/deployment/prod/<site>/netbird, so there's no per-host advertise flag.)
|
||||
mgmt_public_ip = each.value.public_ip
|
||||
mgmt_public_ip = each.value.public_ip
|
||||
mgmt_oob = {
|
||||
nic = each.value.oob_nic
|
||||
address = "${cidrhost(module.addressing.mgmt_cidr, each.value.oob_host)}/${local.mgmt_mask}"
|
||||
}
|
||||
mgmt_fabric_nic = each.value.fabric_nic
|
||||
mgmt_fabric_vlans = [
|
||||
{ id = module.addressing.public_vlan_id, address = "${cidrhost(module.addressing.public_cidr, each.value.host_index)}/${local.pub_mask}" },
|
||||
{ id = module.addressing.private_vlan_id, address = "${cidrhost(module.addressing.private_cidr, each.value.host_index)}/${local.priv_mask}" },
|
||||
{ id = module.addressing.api_vlan_id, address = "${cidrhost(module.addressing.api_cidr, each.value.host_index)}/${local.api_mask}" },
|
||||
]
|
||||
})}"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user