mxm/homelab
1
0
Fork 0
Code Issues 3 Pull requests 1 Projects Releases Packages Wiki Activity Actions

discussion: convert single villa Proxmox nodes to bare NixOS service hosts (clan + disko + nixos-anywhere) #5

New issue
Open
opened 2026-06-01 06:52:21 +00:00 by mxm · 0 comments
mxm commented 2026-06-01 06:52:21 +00:00
Owner
Copy link

Context & goal

Companion to docs/architecture/nixos-container-substrate-map.md and the clan.lol evaluation. The substrate map already concludes the migration is "convert a Proxmox node to a bare NixOS host (running nspawn and Incus), node by node… start at villa (3 PVE nodes → convert one while two carry load)" and that networking is the hardest part, not deploying the services.

This issue is a focused discussion + runbook for the host-conversion mechanics of doing that on one villa node, specifically around clan, disko, a boot image, and nixos-anywhere in-place ("from within").

Status: exploration / decision — no migration committed. Consistent with the substrate map.

Villa facts: 3 PVE nodes villa-pve-01/02/03 at 10.1.10.11/.12/.13 (Mgmt VLAN 10); services on Server VLAN 40 (10.1.40.x, gw 10.1.40.1). Console reality: no out-of-band IPMI/iKVM. The only console is Proxmox noVNC, which lives on the other nodes — the node being converted loses its own console the instant it kexecs away from Proxmox.

The decision in one paragraph

We want one villa node to stop being a Proxmox hypervisor and instead be a bare NixOS host that carries services directly (host-intrinsic infra as native systemd; cattle as nspawn with their own VLAN-40 IP; optional pets as Incus-LXC). Getting NixOS onto the bare metal is the new step. Three mechanisms exist — nixos-anywhere in-place (kexec), boot-image/USB then install, and clan (which wraps nixos-anywhere+disko). All three end at the same place; they differ in how much we bet on the node coming back without a console. Given no IPMI, the recommendation is drain-then-convert (evacuate services to the two peers first so the node is disposable) and use a boot image for the first conversion, keeping in-place kexec for later nodes once the disko layout and a booting closure are proven.

Options for getting NixOS onto a villa node

A. nixos-anywhere in-place (kexec "from within") B. Boot image / USB → install C. clan machines install
Mechanism SSH to running Proxmox host → kexec into a RAM installer → disko wipes the boot disk → install → reboot Build a NixOS installer ISO/USB (nixos-generators), physically boot the node, run disko-install / nixos-anywhere locally Same as A under the hood (clan bundles nixos-anywhere + disko + sops-nix + nixos-generators)
Console needed? No — until it fails. If kexec/network/boot fails, no console = bricked until physical access Yes (physical) to boot the media — but that is the recovery surface Same as A
Disk wipe disko, destructive, irreversible disko, destructive, irreversible disko, destructive, irreversible
Risk w/ no IPMI High Low (you're already at the machine) High (same as A)
Extra value lowest-touch, fully remote safest first run adds inventory + vars secret generators + uniform CLI
Fit here later nodes, once proven first node optional management layer on top

All three require: a correct disko layout for the node's real disk, a NixOS closure that boots on this hardware, and root SSH (A/C) or physical access (B).

Feasibility of in-place "from within" given our console reality

Mechanically: yes — this is exactly what nixos-anywhere is designed for. Default phases: kexec (boot a minimal NixOS installer that runs entirely from RAM, so the old OS is no longer mounted) → disko (destroy + create + mount the target disk — the same disk the OS booted from, now free because we're in RAM) → install (build/copy closure, nixos-install) → reboot. Budget ~1.5–2.5 GB RAM for the kexec installer (the 1 GB README floor OOMs in practice without zram); villa nodes easily clear this. Build the closure on the deployer or a builder, not on the target.

Operationally: high-risk here, because we have no out-of-band console. Failure modes that brick the node until someone walks to it:

  • kexec tears down the SSH session; if the installer's network/SSH doesn't return, the node is unreachable.
  • post-install reboot into a non-booting NixOS (bad bootloader/initrd, wrong NIC name, no DHCP/static) — no console to fix it.
  • wrong device path in the disko config → wipes the wrong disk.
  • single-disk wipe is irreversible; there is no in-place rollback.

What makes it acceptable: drain-then-convert. Evacuate the node's guests to the two peer PVE nodes first (live-migrate or restore from vzdump), so the node carries no load and is fully disposable. Then a failed in-place attempt costs a drive to the rack, not an outage. De-risk further with nixos-anywhere --vm-test (boot the config in a local VM first) and --generate-hardware-config (capture the real NIC/disk).

Recommendation: first node → boot image (Option B); later nodes → in-place kexec (Option A) once the disko layout + booting closure are proven. Always stage a boot-USB as fallback regardless.

Full runbook — converting one drained villa node

⚠️ Every code block below is illustrative and NOT YET APPLIED. Device paths, NIC names, and disk topology must be read off the real node (lsblk, ip link) before running anything. disko destroys the disk.

Step 0 — Drain the node

  • Live-migrate / vzdump + restore the node's CTs/VMs onto the other two villa PVE nodes. Confirm every service is healthy on its new home (Traefik route, VLAN-40 reachability) before touching the node. The node must carry zero load.
  • Snapshot any node-local state to the NAS (→ Storage Box). After this, the node is disposable.

Step 1 — disko layout (read the real disk first!)

# disko-config.nix — NOT YET APPLIED. Verify the device with `lsblk -o NAME,SIZE,TYPE` on the node.
{
  disko.devices.disk.main = {
    type = "disk";
    device = "/dev/disk/by-id/REPLACE-with-stable-id";  # NEVER /dev/sda on a wipe
    content = {
      type = "gpt";
      partitions = {
        ESP  = { size = "1G"; type = "EF00"; content = { type = "filesystem"; format = "vfat"; mountpoint = "/boot"; }; };
        root = { size = "100%";              content = { type = "filesystem"; format = "ext4"; mountpoint = "/"; }; };
      };
    };
  };
}

Step 2 — host config skeleton (plain NixOS first; clan optional)

A bare service host needs: the disko module, a bootloader, the systemd-networkd config (Step 4), sops-nix, and the service feature modules (native systemd + containers.<name> nspawn, optionally virtualisation.incus.enable).

Option C add-on — wrap with clan (only if we adopt clan for this tier):

# flake.nix — NOT YET APPLIED. Exact wrapper symbol tracks the clan release; see the
# "Convert Existing NixOS Configuration" guide. Conceptually:
{
  inputs.clan-core.url = "git+https://git.clan.lol/clan/clan-core";
  # wrap existing machines with the clan builder; per machine:
  #   clan.core.networking.targetHost = "root@10.1.10.11";
}

clan machines install villa-pve-01 == nixos-anywhere + disko (Option A/C). clan machines update == nixos-rebuild switch --target-host (identical to today's homelab apply). clan's real win here is vars (secret generators on top of sops-nix); its mesh-VPN feature is redundant with our UniFi WireGuard and should stay disabled.

Step 3 — install

# Option A/C (in-place, from within) — NOT YET APPLIED:
nixos-anywhere --flake .#villa-pve-01 --target-host root@10.1.10.11 \
  --generate-hardware-config nixos-generate-config ./hosts/villa-pve-01/hardware.nix
  # (nixos-facter is the more modern backend for unattended installs)
# de-risk first:  nixos-anywhere --flake .#villa-pve-01 --vm-test
# pick the disk:  ls -l /dev/disk/by-id   # use the ata-/nvme- symlink, verify before wiping

# Option B (boot image): build + write USB, boot the node from it, then:
#   nixos-generate -f install-iso -c ./installer.nix   # bake in root SSH key
#   disko-install --flake .#villa-pve-01 --disk main /dev/disk/by-id/...

Step 4 — networking rebuild (the hard part)

Reproduce Proxmox VLAN-aware vmbr0.10/.40 in systemd-networkd. The repo already has the VLAN-on-NIC pattern in modules/hosts/villa-router-01.nix and modules/features/gateway.nix — extend it with VLAN 40 and a bridge+veth for guests:

# NOT YET APPLIED. NIC name (ens18 on VMs; enpXsY on bare metal) must be confirmed.
systemd.network = {
  netdevs."10-vlan40" = { netdevConfig = { Kind = "vlan"; Name = "vlan40"; }; vlanConfig.Id = 40; };
  networks."10-uplink" = {
    matchConfig.Name = "enp1s0";            # confirm real NIC
    networkConfig.VLAN = [ "vlan10" "vlan40" ];
  };
  networks."20-vlan10" = { matchConfig.Name = "vlan10"; networkConfig = { Address = "10.1.10.11/24"; Gateway = "10.1.10.254"; }; };
  networks."20-vlan40" = { matchConfig.Name = "vlan40"; networkConfig.Address = "10.1.40.11/24"; };
};

For nspawn/Incus guests that need their own 10.1.40.NN identity: a VLAN-aware bridge (VLANFiltering=yes) with veth pairs tagged into VLAN 40 (verify bridgeVLANs/PVID syntax against systemd.network(5)). This bridge+veth-per-guest is the genuinely new networking work the substrate map flags. Keep dnsmasq/router roles unchanged.

Step 5 — bring services back

  • Host-intrinsic infra → native systemd (services.<name>.enable).
  • Cattle → containers.<name> (nspawn) with /data bind-mounted from NAS, veth on VLAN 40, Traefik route + DNS/tunnel updated.
  • Pets → Incus-LXC (incus launch from the lxc-container.nix image), restore /data; adopt clan vars/update here if desired.

Step 6 — validation

  • Node reachable on Mgmt (10.1.10.11) and each service on VLAN 40 (10.1.40.x).
  • Traefik route + Cloudflare tunnel serve each moved service; wow↔villa WireGuard reachability intact.
  • Secrets decrypt (sops-nix), /data mounted, nixos-rebuild switch clean.

Step 7 — rollback

  • Services already live on the two peers (Step 0) → no service outage from a failed convert.
  • To restore the node itself: boot Proxmox install media (or a peer's vzdump) and reinstall PVE. There is no in-place rollback of the disko wipe — the boot-USB is the recovery path.

Where clan helps vs where it doesn't

Tier on the bare host Substrate Management
Host-intrinsic (router/dnsmasq, edge, NAS, monitoring agents) native systemd clan fits (clanServices = role modules on the host)
Cattle needing own VLAN-40 IP nspawn plain NixOS + sops-nix (+ homelab CLI); clan does NOT see nspawn sub-containers
Stateful pets Incus-LXC clan shines (machine = guest; vars + update)

clan fits everywhere except nspawn. For the nspawn tier, plain NixOS + sops-nix remains the honest substrate; clan would only manage the host layer. clan's standalone win regardless of substrate is vars (cherry-pickable onto our existing sops-nix without full adoption).

Recommendation

  1. Drain-then-convert, starting at one villa node (substrate map already endorses villa-first).
  2. First node via boot image (Option B) given no IPMI; prove the disko layout + a booting closure + the VLAN bridge/veth networking.
  3. Later nodes via in-place nixos-anywhere kexec (Option A) once proven, always with a boot-USB fallback.
  4. clan = optional, cherry-picked: adopt vars for secret generation; consider clan as the management layer for the Incus-pet tier only; do not enable clan's mesh VPN; keep homelab apply for nspawn hosts.

Open questions

  • Per-node disk topology (single vs dual disk) and real NIC names — must be read off each node.
  • VLAN-aware bridge in systemd-networkd: full VLANFiltering + per-veth bridgeVLANs, or simpler per-VLAN netdevs + macvlan for guests?
  • Do we want the Incus-pet tier at all on villa, or nspawn-cattle + one residual Proxmox-for-Windows to start?
  • Adopt clan now (for vars) or just replicate the vars pattern on sops-nix?

Generated as an exploration/decision artifact. No infrastructure changed. Grounded in nixos-anywhere/disko/clan docs and the repo's existing villa-router-01.nix + gateway.nix systemd-networkd patterns.

## Context & goal Companion to [`docs/architecture/nixos-container-substrate-map.md`](https://git.miskam.xyz/mxm/homelab/src/branch/main/docs/architecture/nixos-container-substrate-map.md) and the [clan.lol evaluation](https://git.miskam.xyz/mxm/homelab/src/branch/main/docs/logs/2026-05-30-clan-lol-evaluation.md). The substrate map already concludes the migration is *"convert a Proxmox node to a bare NixOS host (running nspawn **and** Incus), node by node… start at villa (3 PVE nodes → convert one while two carry load)"* and that **networking is the hardest part, not deploying the services**. This issue is a focused **discussion + runbook** for the *host-conversion mechanics* of doing that on **one** villa node, specifically around **clan**, **disko**, a **boot image**, and **nixos-anywhere in-place ("from within")**. > Status: **exploration / decision** — no migration committed. Consistent with the substrate map. **Villa facts:** 3 PVE nodes `villa-pve-01/02/03` at `10.1.10.11/.12/.13` (Mgmt VLAN 10); services on Server VLAN 40 (`10.1.40.x`, gw `10.1.40.1`). **Console reality: no out-of-band IPMI/iKVM.** The only console is Proxmox noVNC, which lives on the *other* nodes — the node being converted loses its own console the instant it `kexec`s away from Proxmox. ## The decision in one paragraph We want one villa node to stop being a Proxmox hypervisor and instead be a **bare NixOS host that carries services directly** (host-intrinsic infra as native systemd; cattle as nspawn with their own VLAN-40 IP; optional pets as Incus-LXC). Getting NixOS *onto* the bare metal is the new step. Three mechanisms exist — **nixos-anywhere in-place (kexec)**, **boot-image/USB then install**, and **clan (which wraps nixos-anywhere+disko)**. All three end at the same place; they differ in *how much we bet on the node coming back without a console*. Given no IPMI, the recommendation is **drain-then-convert** (evacuate services to the two peers first so the node is disposable) and use a **boot image for the first conversion**, keeping in-place kexec for later nodes once the disko layout and a booting closure are proven. ## Options for getting NixOS onto a villa node | | **A. nixos-anywhere in-place (kexec "from within")** | **B. Boot image / USB → install** | **C. clan `machines install`** | |---|---|---|---| | Mechanism | SSH to running Proxmox host → kexec into a RAM installer → disko wipes the boot disk → install → reboot | Build a NixOS installer ISO/USB (nixos-generators), physically boot the node, run `disko-install` / `nixos-anywhere` locally | Same as A under the hood (clan bundles nixos-anywhere + disko + sops-nix + nixos-generators) | | Console needed? | **No — until it fails.** If kexec/network/boot fails, **no console = bricked** until physical access | **Yes (physical)** to boot the media — but that *is* the recovery surface | Same as A | | Disk wipe | disko, destructive, irreversible | disko, destructive, irreversible | disko, destructive, irreversible | | Risk w/ no IPMI | **High** | **Low** (you're already at the machine) | High (same as A) | | Extra value | lowest-touch, fully remote | safest first run | adds inventory + `vars` secret generators + uniform CLI | | Fit here | later nodes, once proven | **first node** | optional management layer on top | All three require: a correct disko layout for the node's real disk, a NixOS closure that boots on this hardware, and root SSH (A/C) or physical access (B). ## Feasibility of in-place "from within" given our console reality **Mechanically: yes — this is exactly what nixos-anywhere is designed for.** Default phases: `kexec` (boot a minimal NixOS installer that runs **entirely from RAM**, so the old OS is no longer mounted) → `disko` (destroy + create + mount the target disk — the *same* disk the OS booted from, now free because we're in RAM) → `install` (build/copy closure, `nixos-install`) → `reboot`. Budget **~1.5–2.5 GB RAM** for the kexec installer (the 1 GB README floor OOMs in practice without zram); villa nodes easily clear this. Build the closure on the deployer or a builder, not on the target. **Operationally: high-risk here, because we have no out-of-band console.** Failure modes that brick the node until someone walks to it: - kexec tears down the SSH session; if the installer's network/SSH doesn't return, the node is unreachable. - post-install reboot into a non-booting NixOS (bad bootloader/initrd, wrong NIC name, no DHCP/static) — no console to fix it. - wrong `device` path in the disko config → wipes the wrong disk. - single-disk wipe is irreversible; there is no in-place rollback. **What makes it acceptable: drain-then-convert.** Evacuate the node's guests to the two peer PVE nodes first (live-migrate or restore from vzdump), so the node carries no load and is fully disposable. Then a failed in-place attempt costs a drive to the rack, not an outage. De-risk further with `nixos-anywhere --vm-test` (boot the config in a local VM first) and `--generate-hardware-config` (capture the real NIC/disk). > **Recommendation:** first node → **boot image (Option B)**; later nodes → **in-place kexec (Option A)** once the disko layout + booting closure are proven. Always stage a boot-USB as fallback regardless. ## Full runbook — converting one drained villa node > ⚠️ Every code block below is **illustrative and NOT YET APPLIED**. Device paths, NIC names, and disk topology **must** be read off the real node (`lsblk`, `ip link`) before running anything. disko **destroys the disk**. ### Step 0 — Drain the node - Live-migrate / `vzdump` + restore the node's CTs/VMs onto the other two villa PVE nodes. Confirm every service is healthy on its new home (Traefik route, VLAN-40 reachability) **before** touching the node. The node must carry zero load. - Snapshot any node-local state to the NAS (→ Storage Box). After this, the node is disposable. ### Step 1 — disko layout (read the real disk first!) ```nix # disko-config.nix — NOT YET APPLIED. Verify the device with `lsblk -o NAME,SIZE,TYPE` on the node. { disko.devices.disk.main = { type = "disk"; device = "/dev/disk/by-id/REPLACE-with-stable-id"; # NEVER /dev/sda on a wipe content = { type = "gpt"; partitions = { ESP = { size = "1G"; type = "EF00"; content = { type = "filesystem"; format = "vfat"; mountpoint = "/boot"; }; }; root = { size = "100%"; content = { type = "filesystem"; format = "ext4"; mountpoint = "/"; }; }; }; }; }; } ``` ### Step 2 — host config skeleton (plain NixOS first; clan optional) A bare service host needs: the disko module, a bootloader, the systemd-networkd config (Step 4), sops-nix, and the service feature modules (native systemd + `containers.<name>` nspawn, optionally `virtualisation.incus.enable`). **Option C add-on — wrap with clan** (only if we adopt clan for this tier): ```nix # flake.nix — NOT YET APPLIED. Exact wrapper symbol tracks the clan release; see the # "Convert Existing NixOS Configuration" guide. Conceptually: { inputs.clan-core.url = "git+https://git.clan.lol/clan/clan-core"; # wrap existing machines with the clan builder; per machine: # clan.core.networking.targetHost = "root@10.1.10.11"; } ``` `clan machines install villa-pve-01` == nixos-anywhere + disko (Option A/C). `clan machines update` == `nixos-rebuild switch --target-host` (identical to today's `homelab apply`). clan's real win here is **`vars`** (secret generators on top of sops-nix); its mesh-VPN feature is **redundant** with our UniFi WireGuard and should stay disabled. ### Step 3 — install ```bash # Option A/C (in-place, from within) — NOT YET APPLIED: nixos-anywhere --flake .#villa-pve-01 --target-host root@10.1.10.11 \ --generate-hardware-config nixos-generate-config ./hosts/villa-pve-01/hardware.nix # (nixos-facter is the more modern backend for unattended installs) # de-risk first: nixos-anywhere --flake .#villa-pve-01 --vm-test # pick the disk: ls -l /dev/disk/by-id # use the ata-/nvme- symlink, verify before wiping # Option B (boot image): build + write USB, boot the node from it, then: # nixos-generate -f install-iso -c ./installer.nix # bake in root SSH key # disko-install --flake .#villa-pve-01 --disk main /dev/disk/by-id/... ``` ### Step 4 — networking rebuild (the hard part) Reproduce Proxmox VLAN-aware `vmbr0.10/.40` in systemd-networkd. The repo **already** has the VLAN-on-NIC pattern in `modules/hosts/villa-router-01.nix` and `modules/features/gateway.nix` — extend it with VLAN 40 and a bridge+veth for guests: ```nix # NOT YET APPLIED. NIC name (ens18 on VMs; enpXsY on bare metal) must be confirmed. systemd.network = { netdevs."10-vlan40" = { netdevConfig = { Kind = "vlan"; Name = "vlan40"; }; vlanConfig.Id = 40; }; networks."10-uplink" = { matchConfig.Name = "enp1s0"; # confirm real NIC networkConfig.VLAN = [ "vlan10" "vlan40" ]; }; networks."20-vlan10" = { matchConfig.Name = "vlan10"; networkConfig = { Address = "10.1.10.11/24"; Gateway = "10.1.10.254"; }; }; networks."20-vlan40" = { matchConfig.Name = "vlan40"; networkConfig.Address = "10.1.40.11/24"; }; }; ``` For nspawn/Incus guests that need their **own** `10.1.40.NN` identity: a VLAN-aware bridge (`VLANFiltering=yes`) with veth pairs tagged into VLAN 40 (verify `bridgeVLANs`/PVID syntax against `systemd.network(5)`). This bridge+veth-per-guest is the genuinely new networking work the substrate map flags. Keep dnsmasq/router roles unchanged. ### Step 5 — bring services back - Host-intrinsic infra → native systemd (`services.<name>.enable`). - Cattle → `containers.<name>` (nspawn) with `/data` bind-mounted from NAS, veth on VLAN 40, Traefik route + DNS/tunnel updated. - Pets → Incus-LXC (`incus launch` from the `lxc-container.nix` image), restore `/data`; adopt clan `vars`/`update` here if desired. ### Step 6 — validation - Node reachable on Mgmt (`10.1.10.11`) and each service on VLAN 40 (`10.1.40.x`). - Traefik route + Cloudflare tunnel serve each moved service; wow↔villa WireGuard reachability intact. - Secrets decrypt (sops-nix), `/data` mounted, `nixos-rebuild switch` clean. ### Step 7 — rollback - Services already live on the two peers (Step 0) → no service outage from a failed convert. - To restore the node itself: boot Proxmox install media (or a peer's vzdump) and reinstall PVE. There is **no in-place rollback** of the disko wipe — the boot-USB *is* the recovery path. ## Where clan helps vs where it doesn't | Tier on the bare host | Substrate | Management | |---|---|---| | Host-intrinsic (router/dnsmasq, edge, NAS, monitoring agents) | native systemd | **clan fits** (clanServices = role modules on the host) | | Cattle needing own VLAN-40 IP | nspawn | plain NixOS + sops-nix (+ homelab CLI); **clan does NOT see nspawn sub-containers** | | Stateful pets | Incus-LXC | **clan shines** (machine = guest; vars + update) | clan fits **everywhere except nspawn**. For the nspawn tier, plain NixOS + sops-nix remains the honest substrate; clan would only manage the host layer. clan's standalone win regardless of substrate is **`vars`** (cherry-pickable onto our existing sops-nix without full adoption). ## Recommendation 1. **Drain-then-convert**, starting at one villa node (substrate map already endorses villa-first). 2. **First node via boot image (Option B)** given no IPMI; prove the disko layout + a booting closure + the VLAN bridge/veth networking. 3. **Later nodes via in-place nixos-anywhere kexec (Option A)** once proven, always with a boot-USB fallback. 4. **clan = optional, cherry-picked**: adopt `vars` for secret generation; consider clan as the management layer for the Incus-pet tier only; do **not** enable clan's mesh VPN; keep `homelab apply` for nspawn hosts. ## Open questions - Per-node disk topology (single vs dual disk) and real NIC names — must be read off each node. - VLAN-aware bridge in systemd-networkd: full `VLANFiltering` + per-veth `bridgeVLANs`, or simpler per-VLAN netdevs + macvlan for guests? - Do we want the Incus-pet tier at all on villa, or nspawn-cattle + one residual Proxmox-for-Windows to start? - Adopt clan now (for `vars`) or just replicate the `vars` *pattern* on sops-nix? --- *Generated as an exploration/decision artifact. No infrastructure changed. Grounded in nixos-anywhere/disko/clan docs and the repo's existing `villa-router-01.nix` + `gateway.nix` systemd-networkd patterns.*
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
67-phase4-forgejo-n8n-matrix
2-shadow-ct-replacement
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
mxm/homelab#5
No description provided.