refactor(o11y): group tenant bundles under base/tenants (#320)

Signed-off-by: Devin Buhl <devin@buhl.casa>

docs/05-shipping-metrics-guide.md

@@ -38,7 +38,7 @@ The metrics remote-write path is the same on every host:
 /insert/0/prometheus/api/v1/write
 ```
 
-(`0` is the VictoriaMetrics tenant; everything lands in a single tenant, see Labels below.)
+(The `0` in the path is nominal: the central `vmauth` rewrites it onto the multitenant insert endpoint and the store assigns the real tenant from the `project` and `cluster` labels, see Tenants below.)
 
 ## Prerequisites (both paths)
 

docs/06-dashboards-and-alerts-guide.md

@@ -26,7 +26,7 @@ A folder files a dashboard under exactly one project; a **tag** is the orthogona
 
 This is how yucca ships (immich-app/yucca#315, see that repo's `o11y/README.md`). The project's CI renders each dashboard into a self-contained `GrafanaDashboard` CR (JSON embedded as `spec.gzipJson`) plus any `GrafanaAlertRuleGroup` CRs and a `GrafanaFolder`, pushes them as **one signed OCI artifact** (`flux push artifact` + cosign keyless), and o11y consumes the whole thing with a single Flux `OCIRepository` + `Kustomization`. New dashboards/alerts flow automatically on the next artifact.
 
-o11y's consumer side lives once in `kubernetes/apps/base/yucca-o11y/` and is pulled into each env's `o11y` overlay:
+o11y's consumer side lives once in `kubernetes/apps/base/tenants/yucca/bundle.yaml`, one file per tenant listed by `base/tenants/kustomization.yaml`, which each env's `o11y` overlay pulls in:
 
 ```yaml
 apiVersion: source.toolkit.fluxcd.io/v1
@@ -81,7 +81,7 @@ Two additions, then:
 - **An ExternalSecret rendering it as a `dockerconfigjson`**, in `flux-system`,
   because that is where the `OCIRepository` lives and a `secretRef` resolves in
   its own namespace. It pulls the two items by name and templates the docker
-  config around them. See `base/fmeet-o11y/externalsecret.yaml`.
+  config around them. See the `ExternalSecret` in `base/tenants/fmeet/bundle.yaml`.
 
 Both items go in the global `o11y_tf` vault, read through the `onepassword`
 store. Not `shared_tf` — that is for credentials more than one project
@@ -108,7 +108,7 @@ anonymous reads).
 `verify:` is also omitted for such a bundle unless the publisher signs with a
 key pair, as harbor-o11y does: its CI signs the digest with a key its
 infrastructure terraform mints, the public half is committed in that repo as
-`cosign.pub`, and `base/harbor-o11y` carries it as the `harbor-o11y-cosign`
+`cosign.pub`, and `base/tenants/harbor` carries it as the `harbor-o11y-cosign`
 Secret referenced from `verify.secretRef`. Keyless cosign mints its certificate from public Fulcio against the
 CI's OIDC identity, and Fulcio accepts `gitlab.com` but not a self-hosted
 forge — so the keyless block above cannot simply be copied across.

docs/runbooks/vm-tenant-migration/README.md

@@ -8,13 +8,15 @@ Commands are fish. The `vmctl` image tag in the Job must match the store's Victo
 set -x D docs/runbooks/vm-tenant-migration
 set -x VMQ https://vmetrics.staging.o11y.futo.network
 function apply_ks
-    flux -n flux-system build kustomization $argv[1] --path ./kubernetes/apps/base/$argv[1]/app | kubectl apply --server-side --force-conflicts -f -
+    flux -n flux-system build kustomization $argv[1] --path ./kubernetes/apps/base/$argv[1]/app | kubectl apply --server-side --field-manager=kustomize-controller --force-conflicts -f -
 end
 ```
 
+`apply_ks` applies as Flux's own field manager on purpose. A server-side apply under any other manager can add and change fields but cannot remove a field Flux owns, so a value the branch deletes (the chart's `defaultRules` params, for example) would survive live and show up as drift in `flux diff` later.
+
 ## Part A: staging rehearsal
 
-Run from the repo root on the branch with the staging kube context selected. Flux stays suspended for these four Kustomizations until Part B merges, which also holds any Renovate change to them; keep the window short.
+Run from the repo root on the branch with the staging kube context selected. Flux stays suspended for the root and these four app Kustomizations until Part B merges, which also holds any Renovate change to them; keep the window short.
 
 ### A0. Settings keys first
 
@@ -39,7 +41,7 @@ flux -n flux-system suspend kustomization victoria-metrics victoria-metrics-user
 Must land before any writer moves, or `o11y cluster stopped reporting` fires for a day. Only the Fleet datasource is applied here; the default datasource changes in A3 together with the vmauth that serves it.
 
 ```fish
-flux -n flux-system build kustomization grafana --path ./kubernetes/apps/base/grafana/app | yq 'select(.metadata.name == "victoria-metrics-fleet")' | kubectl apply --server-side --force-conflicts -f -
+flux -n flux-system build kustomization grafana --path ./kubernetes/apps/base/grafana/app | yq 'select(.metadata.name == "victoria-metrics-fleet")' | kubectl apply --server-side --field-manager=kustomize-controller --force-conflicts -f -
 curl -s $VMQ/select/multitenant/prometheus/api/v1/query --data-urlencode 'query=count by (cluster) (up)' | jq -c '.data.result[] | [.metric.cluster, .value[1]]'
 ```
 
@@ -146,6 +148,8 @@ Expected: staleness of a few seconds for every remote, the same per-cluster coun
 
 ## Part B: commit as two PRs
 
+Done on 2026-09-12 as #318 (plumbing) and #319 (cutover); kept here as the procedure.
+
 The branch splits into a behaviour-preserving PR and the cutover PR, so production never has the fleet view and the writers change in the same reconcile. Production's grafana Kustomization does not depend on victoria-metrics, so a single merge would leave their order to chance.
 
 **PR 1, multitenant plumbing, a no-op while everything is in tenant 0.** The `CLUSTER_VMETRICS_*` and `QUOTE` settings keys in both environments, so the live ConfigMap carries them before any render needs them (a child Kustomization can reconcile a new revision before the root has updated the ConfigMap, which renders empty IDs once); the Fleet datasource and the vmui redirect onto `/select/multitenant`; the shared-token VMUser and the mesh-unauth VMAuth rewrites of `/insert/0/` and `/select/0/` onto multitenant; the vmagent and vmalert `remoteWrite` URLs onto `/insert/multitenant/`. Without registry rules the multitenant path stores everything in tenant 0 and multitenant reads return it, so nothing observable changes.
@@ -170,6 +174,8 @@ flux -n flux-system resume kustomization victoria-metrics victoria-metrics-users
 
 ## Part C: production cutover
 
+Run on 2026-09-12; production o11y moved to `1:1` at 23:34:48Z when #319 reconciled.
+
 Merging PR 2 is the production cutover: Flux moves o11y's writers, registry and readers together. Only o11y is backfilled on production, and only 30 days: it is the one cluster with tenant-scoped readers (the default Grafana datasource, vmalert's remoteRead and MCP, all pinned to `1:1`), and a month covers what operational dashboards look at. Older o11y history stays in tenant 0, visible through the Fleet datasource, until the 120-day retention expires it. Staging measured this at about four hours for 30 days of o11y; run it off-peak.
 
 ```fish

kubernetes/apps/base/fip-o11y/kustomization.yaml

@@ -1,5 +0,0 @@
----
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - ./fip-o11y.yaml

kubernetes/apps/base/fmeet-o11y/externalsecret.yaml

@@ -1,25 +0,0 @@
----
-# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/external-secrets.io/externalsecret_v1.json
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: fmeet-o11y-registry
-  namespace: flux-system
-spec:
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: onepassword
-  target:
-    name: fmeet-o11y-registry
-    template:
-      type: kubernetes.io/dockerconfigjson
-      data:
-        .dockerconfigjson: >-
-          {"auths":{"gitlab.futo.org:5050":{"username":"{{ .username }}","password":"{{ .password }}","auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}
-  data:
-    - secretKey: username
-      remoteRef:
-        key: FMEET_O11Y_REGISTRY_DEPLOY_TOKEN_USERNAME
-    - secretKey: password
-      remoteRef:
-        key: FMEET_O11Y_REGISTRY_DEPLOY_TOKEN_PASSWORD

kubernetes/apps/base/fmeet-o11y/kustomization.yaml

@@ -1,6 +0,0 @@
----
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - ./externalsecret.yaml
-  - ./fmeet-o11y.yaml

kubernetes/apps/base/harbor-o11y/kustomization.yaml

@@ -1,5 +0,0 @@
----
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - ./harbor-o11y.yaml

kubernetes/apps/base/tenants/fmeet/bundle.yaml

@@ -41,3 +41,28 @@ spec:
   prune: true
   dependsOn:
     - name: grafana-operator
+---
+# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/external-secrets.io/externalsecret_v1.json
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: fmeet-o11y-registry
+  namespace: flux-system
+spec:
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword
+  target:
+    name: fmeet-o11y-registry
+    template:
+      type: kubernetes.io/dockerconfigjson
+      data:
+        .dockerconfigjson: >-
+          {"auths":{"gitlab.futo.org:5050":{"username":"{{ .username }}","password":"{{ .password }}","auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}
+  data:
+    - secretKey: username
+      remoteRef:
+        key: FMEET_O11Y_REGISTRY_DEPLOY_TOKEN_USERNAME
+    - secretKey: password
+      remoteRef:
+        key: FMEET_O11Y_REGISTRY_DEPLOY_TOKEN_PASSWORD

kubernetes/apps/base/tenants/kustomization.yaml

@@ -0,0 +1,8 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - ./yucca/bundle.yaml
+  - ./fmeet/bundle.yaml
+  - ./harbor/bundle.yaml
+  - ./fip/bundle.yaml

kubernetes/apps/base/yucca-o11y/kustomization.yaml

@@ -1,5 +0,0 @@
----
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - ./yucca-o11y.yaml

kubernetes/apps/production/o11y/kustomization.yaml

@@ -3,10 +3,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - ../../base/yucca-o11y
-  - ../../base/fmeet-o11y
-  - ../../base/harbor-o11y
-  - ../../base/fip-o11y
+  - ../../base/tenants
   - ../../base/prometheus-operator-crds
   - ../../base/grafana-operator
   - ../../base/grafana-postgres

kubernetes/apps/staging/o11y/kustomization.yaml

@@ -3,10 +3,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - ../../base/yucca-o11y
-  - ../../base/fmeet-o11y
-  - ../../base/harbor-o11y
-  - ../../base/fip-o11y
+  - ../../base/tenants
   - ../../base/prometheus-operator-crds
   - ../../base/grafana-operator
   - ../../base/grafana-postgres