Iac on Matteo ZINUTTI

Passing an NVIDIA GPU through to a Kubernetes cluster

Fri, 09 Oct 2026 00:00:00 +0000

Series “A Kubernetes cluster with a GPU for LLM inference”

Deploying an immutable Kubernetes cluster with Talos on Proxmox

Passing an NVIDIA GPU through to a Kubernetes cluster (this article)

Running a Kubernetes cluster with GitOps and Flux (in progress)

Self-hosted LLM inference on Kubernetes (in progress)

Introduction

The Talos cluster deployed in part 1 is up and running, but the talos-gpu-01 node does not see any graphics card yet. In this article, I explain how I made a GeForce RTX 3050 usable by Kubernetes pods, starting from the Proxmox host. The work breaks down into four steps: detach the card from the host, attach it to the VM with OpenTofu, load the NVIDIA modules in Talos, then expose the GPU to the Kubernetes scheduler.

The path of the GPU, from the Proxmox host to the Kubernetes pod

Before getting to the commands, I explain what passthrough does, since that is where most of the constraints met later come from, such as disabling ballooning or checking the IOMMU groups.

Context

The end goal of this series is to run LLM inference locally in my homelab, which requires a GPU accessible from the cluster. To give a graphics card to a VM, there are two main approaches: share it between several VMs with vGPU, or dedicate it entirely to a single VM with PCI passthrough.

In my case, the choice was quick. NVIDIA only offers vGPU on its datacenter and professional lines, it is not available on a GeForce. That leaves passthrough, with a direct consequence: the card is dedicated to the talos-gpu-01 VM, the Proxmox host no longer has access to it and no other VM can share it.

The hardware used is a GeForce RTX 3050 that I had on hand.

Prerequisites

A CPU and motherboard supporting the IOMMU (VT-d on Intel, AMD-Vi on AMD), enabled in the BIOS.
The Talos cluster from part 1.
A GPU that is not used for the host display. In my case, the Proxmox host runs headless and the CPU’s integrated GPU is disabled in the BIOS, so the RTX 3050 is the only graphics card in the machine.

Understanding passthrough

A PCIe device such as a GPU does not go through the CPU to read or write memory. It uses DMA (Direct Memory Access) to access RAM directly, which lets it transfer large volumes of data without keeping the processor busy.

In a VM, this causes a problem. The memory addresses seen by the guest system are guest physical addresses (GPA, Guest Physical Address), which do not match the host’s real physical addresses (HPA, Host Physical Address). For CPU accesses, the GPA to HPA translation is done in hardware by EPT (Extended Page Tables) on Intel or NPT (Nested Page Tables) on AMD, a second page table that the processor walks in addition to the guest’s. A device doing DMA does not go through this translation: if the VM’s driver gives a GPA to the GPU, the card writes at that address in the host’s RAM, which corresponds to something else entirely.

That is the job of the IOMMU (Input-Output Memory Management Unit). It does for devices what the MMU (Memory Management Unit), the processor component that translates virtual addresses into physical addresses, does for the CPU: it translates the addresses used by the device into real physical addresses, and it restricts the device to the memory areas assigned to it. The GPU passed to the VM can therefore only write into that VM’s memory.

This isolation is not done per device but per IOMMU group. A group gathers the devices that the hardware cannot isolate from one another, for example because they can communicate with each other without going through the IOMMU. The unit that can be passed to a VM is therefore the whole group, and you need to check the content of the card’s group before starting.

Finally, on the host side, the card must not be claimed by any graphics driver (nouveau or nvidia). The vfio-pci driver takes ownership of it. It is part of VFIO (Virtual Function I/O), the Linux kernel framework that gives a program outside the kernel, here QEMU, direct access to a device, relying on the IOMMU to limit what the device can reach in memory. That is how QEMU presents the card to the VM.

Address translation: CPU accesses and GPU DMA accesses

Why ballooning must be disabled

Ballooning lets Proxmox reclaim RAM that a VM is not using, on the fly, thanks to a driver in the guest that hands memory pages back to the host. With a device passed through VFIO, this mechanism can no longer work: the GPU can write into the VM’s pages at any time without going through the CPU, so the host has no way of knowing whether a page is still in use by the card. All of the VM’s memory is therefore pinned at boot and can no longer be given back to the host.

In the bpg/proxmox provider, ballooning is controlled by the floating argument of the memory block, and the value 0 disables it. I set it to 0 in the resource that creates the Talos nodes, so on all three nodes and not only on the GPU node:

1
2
3
4
5


#talos.tf
memory {
 dedicated = each.value.memory
 floating = 0
}

This affects sizing: the 12 GB of RAM of the talos-gpu-01 node are permanently reserved on the host, even when the node is idle.

Now that the mechanism is clear, let’s prepare the Proxmox host.

Step 1: preparing the Proxmox host

Checking the IOMMU and the groups

The first thing to do is identify the card and the IOMMU group it belongs to:

1
2
3
4
5


# The card and its functions
lspci -nn | grep -i nvidia

# The IOMMU group it belongs to
find /sys/kernel/iommu_groups -type l | grep '01:00'

Which gives in my case:

1
2
3
4
5


01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)

/sys/kernel/iommu_groups/10/devices/0000:01:00.0
/sys/kernel/iommu_groups/10/devices/0000:01:00.1

The card exposes two functions: 01:00.0 for the video part and 01:00.1 for the HDMI audio controller. Both are alone in IOMMU group 10, which means the group can be passed to the VM as is, without taking any other host device along.

If other devices shared this group, for example a USB controller, they would have to be passed to the VM too. The ACS override patch is often suggested on forums to split groups artificially, but it asks the kernel to ignore an isolation that the hardware does not guarantee, since the devices in the group can still communicate with each other. The Proxmox wiki actually presents it as a last resort, and not a risk-free one.

Binding the card to vfio-pci

I configured the Proxmox host so that vfio-pci claims the card before any other driver. This part is not managed by my OpenTofu code, I did it directly on the Proxmox host.

According to the Proxmox documentation, the IOMMU is enabled by default on Intel from kernel 6.8, so the intel_iommu=on option found in many guides is no longer needed. The only useful parameter is iommu=pt, which limits address translation to the devices passed to a VM and can improve performance for the others.

Kernel parameters are passed through the command line that GRUB, the host’s bootloader, hands to the kernel at every boot. When the host boots with GRUB, which is my case, it is set in /etc/default/grub:

1
2


#/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

GRUB_CMDLINE_LINUX_DEFAULT applies to normal boot entries, but not to recovery mode. quiet is present by default on Debian and Proxmox: it reduces the kernel messages displayed on screen during boot, without changing what is recorded in the logs.

The host’s dmesg confirms that the IOMMU is active, with the requested passthrough mode, and that the devices are split into IOMMU groups:

1
2
3


[ 0.284240] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.329656] pci 0000:00:00.0: Adding to iommu group 0
[ 0.329665] pci 0000:00:01.0: Adding to iommu group 1

The VFIO modules are loaded at boot through /etc/modules: vfio for the core of the framework, vfio_iommu_type1 to drive the IOMMU, and vfio_pci for PCI devices:

1
2
3
4


#/etc/modules
vfio
vfio_iommu_type1
vfio_pci

For vfio-pci to reserve the card, I give it the IDs of both functions, the video part 10de:2507 and the audio part 10de:228e:

1
2


#/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2507,10de:228e disable_vga=1

The disable_vga=1 option prevents vfio-pci from exposing the card’s legacy VGA resources to the host. On top of that, I block the NVIDIA graphics drivers on the host so that none of them claims the card at boot:

1
2
3


#/etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia*

Changes to /etc/modules and /etc/modprobe.d/ only take effect after regenerating the initramfs (update-initramfs -u -k all), and changes to /etc/default/grub after an update-grub. After rebooting the host, the check is done with:

1

lspci -nnk -s 01:00

Which gives in my case:

1
2
3
4
5
6
7
8


01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] (rev a1)
 Subsystem: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507]
 Kernel driver in use: vfio-pci
 Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
 Subsystem: NVIDIA Corporation Device [10de:2507]
 Kernel driver in use: vfio-pci
 Kernel modules: snd_hda_intel

Both functions are claimed by vfio-pci. The Kernel modules line lists the drivers able to handle the card, not those that are loaded: nouveau and nvidiafb are blocked and do not show up in lsmod.

⚠️ Warning

As long as lspci -nnk does not show Kernel driver in use: vfio-pci on both functions, there is no point going further. The VM may boot, but the card will not be usable.

The host is ready, the card can now be attached to the VM.

Step 2: attaching the GPU to the VM with OpenTofu

Why a hardware mapping and not a PCI ID

The bpg/proxmox provider offers two ways to declare a device in a VM’s hostpci block: its raw PCI ID with the id argument, or a hardware mapping declared at the Proxmox cluster level with the mapping argument.

The provider documentation states that id is not compatible with API token authentication and requires the root account with its password. All my IaC authenticates with the token of the terraform@pve user, so I went with the mapping, which was the only option without changing the provider’s authentication.

The mapping is declared with the proxmox_hardware_mapping_pci resource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#hardware_mapping.tf
resource "proxmox_hardware_mapping_pci" "this" {
 ...
 for_each = var.pci_hardware_mappings

 name = each.key
 comment = each.value.comment

 map = [
 {
 node = coalesce(each.value.node, var.proxmox_node)
 path = each.value.path
 id = each.value.id
 iommu_group = each.value.iommu_group
 subsystem_id = each.value.subsystem_id
 }
 ]
}

The values are set in the variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#variables.auto.tfvars
pci_hardware_mappings = {
 gpu = {
 comment = "GeForce RTX 3050 - managed by Terraform"
 path = "0000:01:00"
 id = "10de:2507"
 iommu_group = 10
 subsystem_id = "10de:2507"
 }
}

The path is declared without a function number (0000:01:00 rather than 0000:01:00.0), which covers both functions of the card at once.

The iommu_group and subsystem_id arguments are optional for the API, but without them Proxmox shows the mapping as incomplete in the UI. So I filled them in from the output of lspci -nnv -s 0000:01:00 and find /sys/kernel/iommu_groups -type l.

Proxmox privileges

The role used by OpenTofu needs three extra privileges:

1
2
3
4


#users.tf
"Mapping.Audit",
"Mapping.Modify",
"Mapping.Use",

Mapping.Audit: read existing mappings.
Mapping.Modify: create, modify and delete a mapping.
Mapping.Use: attach a mapping to a VM, on top of the usual VM modification rights.

Without Mapping.Use, the mapping is created but attaching it to the VM fails.

The hostpci block

On the VM side, a single resource serves all three Talos nodes. The hostpci block is therefore dynamic: it is generated from each node’s hostpci list, which defaults to []. The two nodes without a card thus have no device attached.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


#talos.tf
dynamic "hostpci" {
 for_each = { for device in each.value.hostpci : device.device => device }

 content {
 device = hostpci.value.device
 mapping = proxmox_hardware_mapping_pci.this[hostpci.value.mapping].name
 pcie = hostpci.value.pcie
 rombar = hostpci.value.rombar
 xvga = hostpci.value.xvga
 }
}

In part 1, I mentioned a sixth validation on the talos_nodes variable. Here it is, it checks that each hostpci.mapping matches a declared mapping:

1
2
3
4
5
6
7
8
9


#variables.tf
validation {
 condition = alltrue(flatten([
 for node in var.talos_nodes : [
 for device in node.hostpci : contains(keys(var.pci_hardware_mappings), device.mapping)
 ]
 ]))
 error_message = "Every Talos node hostpci.mapping must be a key of var.pci_hardware_mappings."
}

A typo in the mapping name is thus caught at plan time, and not as an API error in the middle of the apply.

Once the pipeline has run, the VM configuration on the Proxmox side reflects what was declared:

1

qm config 123 | grep -E '^(hostpci|balloon|memory)'

1
2
3


balloon: 0
hostpci0: mapping=gpu,pcie=1,rombar=1,x-vga=0
memory: 12288

The device is referenced by the mapping name and not by its PCI address, and balloon: 0 matches the floating = 0 of the memory block.

The card is now attached to the VM, it remains to make it usable by Talos.

Step 3: the NVIDIA modules in Talos

A schematic dedicated to the GPU node

As explained in part 1, the NVIDIA extensions are not in the base image: the nodes without a card reported module loading errors at every boot. The talos-gpu-01 node therefore uses its own schematic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


#variables.auto.tfvars
{
 hostname = "talos-gpu-01"
 ip = "192.168.1.123"
 cores = 4
 memory = 12 * 1024,
 disk_size = 100,
 role = "worker"
 schematic_id = "a31d81eceab05afdc2c59b0002701173e117a5fb81bdc1f64062d27655145848" # With NVIDIA GPU support
 hostpci = [
 {
 device = "hostpci0"
 mapping = "gpu"
 }
 ]
}

This schematic adds two extensions to qemu-guest-agent: nvidia-open-gpu-kernel-modules-production for the drivers, and nvidia-container-toolkit-production so that the container runtime knows how to expose the card to pods.

📝 Note

I use NVIDIA’s open source drivers. They support GPUs from the Turing architecture onwards, which includes the RTX 3050 (Ampere). On older hardware, you need the proprietary extensions.

Loading the modules

The extensions ship the kernel modules but do not load them. That is the job of a machine config patch, placed at the node level and not at the worker/ level:

1
2
3
4
5
6
7
8


# patches/nodes/talos-gpu-01/30-nvidia-modules.yaml.tftpl
machine:
 kernel:
 modules:
 - name: nvidia
 - name: nvidia_uvm
 - name: nvidia_drm
 - name: nvidia_modeset

This patch uses the nodes/ level presented in part 1: placed under worker/, it would also be applied to talos-worker-01, whose image does not contain the extensions.

The list of modules comes from the Talos documentation. The nvidia_uvm module is the one CUDA uses to manage unified memory.

Applying and checking

The schematic and the patch are deployed by the GitLab CI pipeline described in part 1, like the rest of the node configuration. I then check that the extensions are present in the node’s image:

1

talosctl -n 192.168.1.123 get extensions

Which gives the following result:

1
2
3
4
5
6


NODE NAMESPACE TYPE ID VERSION NAME VERSION
192.168.1.123 runtime ExtensionStatus 0 1 nvidia-container-toolkit-production 595.71.05-v1.19.1
192.168.1.123 runtime ExtensionStatus 1 1 nvidia-open-gpu-kernel-modules-production 595.71.05-v1.13.8
192.168.1.123 runtime ExtensionStatus 2 1 qemu-guest-agent 11.0.2
192.168.1.123 runtime ExtensionStatus 3 1 schematic a31d81eceab05afdc2c59b0002701173e117a5fb81bdc1f64062d27655145848
192.168.1.123 runtime ExtensionStatus modules.dep 1 modules.dep 6.18.42-talos

Both NVIDIA extensions are at version 595.71.05, the driver number, and the schematic ID is the one declared for this node. The four modules from the patch must then be loaded:

1

talosctl -n 192.168.1.123 get loadedkernelmodules | grep -i nvidia

1
2
3
4


192.168.1.123 runtime LoadedKernelModule nvidia 2 Live
192.168.1.123 runtime LoadedKernelModule nvidia_drm 2 Live
192.168.1.123 runtime LoadedKernelModule nvidia_modeset 2 Live
192.168.1.123 runtime LoadedKernelModule nvidia_uvm 1 Live

Finally, the /proc/driver/nvidia/version file only exists if the nvidia module is loaded, and it gives the exact driver version:

1

talosctl -n 192.168.1.123 read /proc/driver/nvidia/version

1
2


NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 595.71.05 Release Build (dvs-builder@U22-I3-G08-03-1) Fri Apr 24 06:42:30 UTC 2026
GCC version: Selected multilib: .;@m64

The Open Kernel Module mention confirms that the open source drivers are the ones running.

The driver is loaded in the VM, the last step is to make it known to Kubernetes.

Step 4: exposing the GPU to Kubernetes

Modules loaded on the node are not enough. The Kubernetes scheduler only knows about CPU and memory, and for it to allocate a GPU to a pod, an nvidia.com/gpu extended resource has to be published on the node by a device plugin. A device plugin is a component running on the node that declares the available GPUs and their state to the kubelet, then chooses which card to assign to each pod that requests one.

NVIDIA GPU Operator

To do this, I used the NVIDIA GPU Operator rather than the NVIDIA device plugin alone, for two reasons: it is the method described in the Talos documentation, and the operator also deploys the DCGM exporter, which lets me collect the GPU metrics in Prometheus.

By default, the operator installs the drivers and the container toolkit on the nodes itself, but on Talos these two components are already provided by the system extensions. They must therefore be disabled to avoid conflicts, as the Talos documentation states:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


kubectl create namespace gpu-operator
kubectl label --overwrite namespace gpu-operator pod-security.kubernetes.io/enforce=privileged

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
 --namespace gpu-operator \
 --version v26.3.3 \
 --set driver.enabled=false \
 --set toolkit.enabled=false \
 --set hostPaths.driverInstallDir=/usr/local \
 --set dcgmExporter.serviceMonitor.enabled=true

driver.enabled=false and toolkit.enabled=false: the drivers and the toolkit come from the Talos extensions.
hostPaths.driverInstallDir=/usr/local: the Talos extension installs the driver files in /usr/local, while the operator looks for them in /run/nvidia/driver by default.
dcgmExporter.serviceMonitor.enabled=true: creates the ServiceMonitor that lets Prometheus collect the GPU metrics.
The namespace must allow privileged pods, because the operator’s components access the node’s hardware.

📝 Note

The Talos documentation requires a GPU Operator version greater than or equal to v26.3.1.

I did not need a nodeSelector to target the GPU node. The operator first deploys Node Feature Discovery on all nodes, which detects the PCI devices of each one. The operator then adds the nvidia.com/gpu.present=true label to the nodes that have an NVIDIA card, and only deploys its components (device plugin, DCGM exporter, validation) on those nodes:

1

kubectl get pods -n gpu-operator -o wide

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpu-feature-discovery-9ll6h 1/1 Running 0 52d 10.244.0.134 talos-gpu-01 <none> <none>
gpu-operator-57d75775c8-ls7hf 1/1 Running 0 4d12h 10.244.2.150 talos-controlplane-01 <none> <none>
gpu-operator-node-feature-discovery-gc-847bb8f7b6-9gmkf 1/1 Running 0 4d12h 10.244.0.231 talos-gpu-01 <none> <none>
gpu-operator-node-feature-discovery-master-d98f944cd-vhzdw 1/1 Running 0 4d12h 10.244.2.188 talos-controlplane-01 <none> <none>
gpu-operator-node-feature-discovery-worker-kvvfl 1/1 Running 0 52d 10.244.0.249 talos-gpu-01 <none> <none>
gpu-operator-node-feature-discovery-worker-ntvzt 1/1 Running 0 52d 10.244.2.39 talos-controlplane-01 <none> <none>
gpu-operator-node-feature-discovery-worker-s95fk 1/1 Running 0 52d 10.244.1.208 talos-worker-01 <none> <none>
nvidia-cuda-validator-gfd6h 0/1 Completed 0 6m42s 10.244.0.109 talos-gpu-01 <none> <none>
nvidia-dcgm-exporter-rkt88 1/1 Running 0 52d 10.244.0.161 talos-gpu-01 <none> <none>
nvidia-device-plugin-daemonset-pgnqw 1/1 Running 0 52d 10.244.0.164 talos-gpu-01 <none> <none>
nvidia-operator-validator-xqdd8 1/1 Running 0 52d 10.244.0.50 talos-gpu-01 <none> <none>

Only the NFD workers run on all three nodes, all the NVIDIA components are on talos-gpu-01. The nvidia-cuda-validator pod in Completed is expected: it runs a test CUDA program on the card, then stops.

Checking the allocation

The nvidia.com/gpu resource must appear in the node’s capacity:

1

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, gpu: .status.capacity."nvidia.com/gpu"}'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "name": "talos-controlplane-01",
 "gpu": null
}
{
 "name": "talos-gpu-01",
 "gpu": "1"
}
{
 "name": "talos-worker-01",
 "gpu": null
}

The talos-gpu-01 node exposes one GPU, the other two nodes do not have the resource. The labels added to the node by GPU Feature Discovery also give the details of the card as seen by Kubernetes:

1

kubectl get node talos-gpu-01 --show-labels | tr ',' '\n' | grep nvidia.com

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


...
nvidia.com/cuda.driver-version.full=595.71.05
...
nvidia.com/cuda.runtime-version.full=13.2
...
nvidia.com/gpu.count=1
...
nvidia.com/gpu.family=ampere
...
nvidia.com/gpu.memory=8192
...
nvidia.com/gpu.product=NVIDIA-GeForce-RTX-3050
...

The nvidia.com/cuda.runtime-version.full=13.2 label gives the highest CUDA version supported by the driver, which is used to choose the image of the test pod.

To validate the whole chain, I run a test pod that executes nvidia-smi (NVIDIA System Management Interface), the command line tool shipped with the driver that shows the visible cards, the driver version, memory usage and running processes. If the command sees the card from inside a container, the driver, the runtime and the allocation all work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


apiVersion: v1
kind: Pod
metadata:
 name: nvidia-smi
spec:
 restartPolicy: Never
 containers:
 - name: nvidia-smi
 image: nvidia/cuda:13.2.1-base-ubuntu24.04
 command: ["nvidia-smi"]
 resources:
 limits:
 nvidia.com/gpu: 1

The nvidia.com/gpu: 1 field in the limits is enough: the pod requests the resource published by the device plugin, so the scheduler can only place it on talos-gpu-01, and the GPU is reserved for it as long as it runs.

The Talos documentation also adds runtimeClassName: nvidia to its test pod, to go through the NVIDIA runtime that the nvidia-container-toolkit extension declares in containerd. On my cluster, this is not needed, because CDI (Container Device Interface) is enabled in the GPU Operator:

1

kubectl get clusterpolicy -o jsonpath='{.items[0].spec.cdi}'

1

{"default":false,"enabled":true,"nriPluginEnabled":false}

With CDI, the device plugin tells containerd which device files and which driver libraries to add to the container, and containerd injects them itself with its default runtime. This is also how the nvidia-smi binary, absent from the nvidia/cuda image, ends up in the container. According to the NVIDIA documentation on CDI, this is the default behaviour since version v25.10.0 of the GPU Operator, and runtimeClassName: nvidia is only still needed for containers that access GPUs through the NVIDIA_VISIBLE_DEVICES environment variable without going through the device plugin, such as monitoring agents.

Once the pod has finished, its logs contain the output of nvidia-smi:

1

kubectl logs nvidia-smi

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3050 On | 00000000:01:00.0 Off | N/A |
| 0% 45C P8 7W / 130W | 1MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

The container sees the RTX 3050 with its 8 GB of memory, the 595.71.05 driver provided by the Talos extension and CUDA 13.2. The whole chain works, from vfio-pci on the Proxmox host all the way to the pod.

Conclusion

The GPU can now be allocated by the scheduler: the cluster exposes an nvidia.com/gpu resource on the talos-gpu-01 node, and a pod can request it just like it would request CPU or memory.

There is only one nvidia.com/gpu resource in the whole cluster, which means only one pod at a time can use the card. To share it between several pods, the GPU Operator offers time-slicing, where pods take turns on the card, and NVIDIA also offers MPS (Multi-Process Service), which runs the CUDA processes of several pods on the card at the same time. These are options I have not set up yet.

For my part, this stage taught me a lot about hardware. My knowledge of virtualisation had so far stayed at a fairly high level, that of the hypervisor, and I did not know the low-level mechanisms that passthrough relies on, such as device I/O, DMA, the MMU or the IOMMU.

At this point, the infrastructure is fully described in OpenTofu, but everything running in the cluster was installed with helm install commands run by hand. It is the same configuration drift problem that Talos solves at the OS level, showing up one floor higher. Part 3 therefore sets up GitOps with Flux before going further towards inference.

Useful links

Deploying an immutable Kubernetes cluster with Talos on Proxmox

Fri, 02 Oct 2026 13:33:17 +0000

Series “A Kubernetes cluster with a GPU for LLM inference”

Deploying an immutable Kubernetes cluster with Talos on Proxmox (this article)

Passing an NVIDIA GPU through to a Kubernetes cluster

Running a Kubernetes cluster with GitOps and Flux (in progress)

Self-hosted LLM inference on Kubernetes (in progress)

Introduction

After deploying a Kubernetes cluster on Proxmox with Kubespray, I wanted to fix something that bothered me: maintaining the OS underneath the cluster. In this article, I start over from scratch with Talos Linux, an immutable distribution dedicated to Kubernetes, which I provision entirely with OpenTofu.

This article is the first of a series whose end goal is to run LLM inference locally, on a GPU passed through to the cluster. The aim is to train myself in MLOps and in deploying AI workloads on Kubernetes. The cluster described here is the foundation, so it already includes a node intended for the GPU.

Context

My previous article covered deploying a Kubernetes cluster with Kubespray. The result was reproducible, but the automation stopped at the OS: the three Debian VMs were still regular systems, with their package updates, their SSH access and a configuration that could drift between two runs of the playbook.

With Talos, the OS can no longer be administered directly: there is no shell, no SSH and no package manager. The only interface is a gRPC API, and the whole machine is described by a YAML configuration file applied declaratively. To change a node, I change its configuration and Talos applies it.

Talos is not better than Kubespray in absolute terms. I lose the ability to step in by hand on a node, but in exchange I know that no node has been changed outside of the code.

Components

Component	Version	Role
Proxmox	9.1	Hypervisor
Talos Linux	1.13.8	Node OS
Kubernetes	v1.36.2	Orchestrator
OpenTofu	1.11.5	VM creation and Talos configuration
`bpg/proxmox` provider	0.114.0	Drives Proxmox
`siderolabs/talos` provider	0.12	Drives Talos
OpenTofu GitLab CI component	4.5.0	Deployment pipeline
Cilium	1.20.2	CNI, replaces kube-proxy

I chose OpenTofu over Terraform to stay consistent with my stance on open source software and licences. Since HashiCorp moved Terraform to the BUSL licence in 2023, OpenTofu is the fork that remains under the MPL 2.0 licence. Its development is also driven more by feature requests from the community, while Terraform’s is geared towards HashiCorp’s Enterprise and Cloud offerings, which I have no interest in.

Talos Linux

Talos is a Linux distribution that only contains what is needed to run Kubernetes: a kernel, its own init system, containerd and the cluster components. A few of its properties explain most of the choices made in the rest of the article:

The root filesystem is read-only, mounted from a squashfs image. Data that must persist (etcd, container images) is written under /var, on a dedicated partition. So there is nowhere to drop a binary or edit a configuration file by hand.
Updates follow an A/B scheme. The new version is installed next to the running one, and if the update fails, the node can reboot into the old one.
There is no systemd. The kubelet, etcd, containerd and the Talos API are managed by machined, and their configuration comes entirely from the machine config. You cannot add a service or override one.
Administration goes exclusively through a gRPC API, exposed on port 50000 of each node with mutual TLS authentication. The talosconfig file used by talosctl contains the cluster CA and a client certificate, and without this file a Talos node refuses any connection.

The entire configuration of a node therefore fits in a single YAML file, the machine config, which has two main sections: machine.* for the node itself (network, disks, kubelet, kernel modules) and cluster.* for the Kubernetes cluster (CNI, kube-proxy, extra manifests).

Machine configuration and cluster configuration

These two sections live in the same file, but they are not applied the same way:

machine.* is reconciled continuously. Talos controllers watch the configuration and adjust the state of the node. A patch that changes machine.network or machine.kubelet is applied immediately, sometimes after restarting the service involved.
cluster.* is only read at bootstrap. During talos_machine_bootstrap, Talos generates Kubernetes manifests from this section and applies them to the cluster. The section then stays in the configuration, but it is never read again.

In practice, changing cluster.network.cni.name on a cluster that is already running does not remove the CNI in place, and Talos shows no error and no warning. The manifests generated from cluster.* are only replayed, removing those that are no longer generated, during a talosctl upgrade-k8s.

The two lifecycles of the machine config

Target architecture

Role	Hostname	VM ID	IP	vCPU	RAM	Disk
Control plane	`talos-controlplane-01`	121	192.168.1.121	2	4 GB	50 GB
Worker	`talos-worker-01`	122	192.168.1.122	4	8 GB	100 GB
GPU worker	`talos-gpu-01`	123	192.168.1.123	4	12 GB	100 GB

The Kubernetes API does not answer on a node IP but on a VIP, 192.168.1.120. This address is reserved, which is why the nodes start at .121.

📝 Note

There is only one control plane, as in my Kubespray article, and for the same reason: it is a homelab. I still configured the VIP from the start, because it is written into the configuration of every node, and setting it up later would mean reconfiguring the whole cluster.

A GPU node declared from the start

The third node is called talos-gpu-01, even though no graphics card is used in this article. It already differs from the other two in three ways:

12 GB of RAM instead of 8. PCI passthrough requires ballooning to be disabled, so this memory will be permanently reserved on the host. I preferred to size the node accordingly right away.
A different Talos image. The NVIDIA modules are provided by system extensions, which are built into the image when it is created. This node therefore has its own image.
A configuration patch that only applies to it. Loading the NVIDIA modules on a node without a card produces an error at every boot.

This last point affects how the configuration is organised: as soon as one node differs from the others, a per-role configuration is no longer enough. I describe the solution in the section on patches.

Provisioning with OpenTofu

The images: Talos Image Factory

A Talos image cannot be modified afterwards. To add qemu-guest-agent or the NVIDIA modules, you need an image that already contains them, and that is the job of the Image Factory: you send it a list of extensions, and it returns a schematic ID, a hash that identifies this list. This hash is then used to build the image URL for any Talos version.

Schematic	Extensions	Nodes
`ce4c9805...`	`qemu-guest-agent`	controlplane-01, worker-01
`a31d81ec...`	`qemu-guest-agent`, `nvidia-open-gpu-kernel-modules-production`, `nvidia-container-toolkit-production`	gpu-01

The downloaded image is a compressed raw disk, nocloud-amd64.raw.xz. It changes two things compared to my previous deployments:

nocloud is the target platform. With it, Talos reads its initial network configuration from the cloud-init drive provided by Proxmox, which makes the node reachable from the first boot, before it has received its machine config.
There is no template anymore. My Debian VMs are cloned from a template built with Packer, whereas the Talos nodes import this image directly as their scsi0 disk. So I no longer have a template to rebuild for this cluster: to change the Talos version, I change a variable.

The download uses a for_each over the schematics actually in use, which avoids downloading the same image twice when several nodes share the same schematic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#image.tf
locals {
 talos_schematics = toset([
 for node in var.talos_nodes : coalesce(node.schematic_id, var.talos_image_factory_id)
 ])
}

resource "proxmox_download_file" "talos_image" {
 for_each = local.talos_schematics

 content_type = "iso"
 datastore_id = "local"
 file_name = "talos-${var.talos_version}-${substr(each.key, 0, 8)}-nocloud-amd64.img"
 node_name = var.proxmox_node
 url = "https://factory.talos.dev/image/${each.key}/v${var.talos_version}/nocloud-amd64.raw.xz"
 overwrite = false
 decompression_algorithm = "zst"
}

The image is downloaded as .raw.xz, while decompression_algorithm is set to zst. The provider only accepts gz, lzo, zst and bz2, but Proxmox’s zst decompression can also handle an xz archive. This behaviour is not documented, I found it in a bpg provider discussion about xz-compressed images.

The topology in a variable

The whole cluster topology is described in a single variable. To add a node, you just add an object to this list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


#variables.tf
variable "talos_nodes" {
 type = list(object({
 hostname = string
 ip = string
 cores = number
 memory = number
 vm_id = optional(number)
 disk_size = optional(number, 40)
 role = optional(string, "controlplane")

 # Schematic Image Factory. `null` retombe sur var.talos_image_factory_id.
 schematic_id = optional(string)

 # Périphériques PCI passés au nœud (partie 2).
 hostpci = optional(list(object({
 device = string
 mapping = string
 pcie = optional(bool, true)
 rombar = optional(bool, true)
 xvga = optional(bool, false)
 })), [])
 }))

 validation {
 condition = !contains([for node in var.talos_nodes : node.ip], var.talos_vip)
 error_message = "The Talos VIP must not collide with a node IP address."
 }
 ...
}

I added more validations than usual on this variable, five in total:

role must be controlplane or worker
hostnames must be unique
IPs must be unique
there must be at least one controlplane
the VIP must not match a node IP

Without these validations, each of these mistakes would show up in the middle of an apply, as an unclear Proxmox API error, with part of the infrastructure already created. With them, the error is caught at plan time, before anything is changed. A sixth validation comes in part 2, to check that each hostpci.mapping matches a declared mapping.

The VMs

A single resource creates the three nodes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


#talos.tf
resource "proxmox_virtual_environment_vm" "talos" {
 depends_on = [
 proxmox_download_file.talos_image,
 proxmox_virtual_environment_pool.TALOS,
 ]
 for_each = { for node in var.talos_nodes : node.hostname => node }

 name = each.key
 node_name = var.proxmox_node
 vm_id = tonumber(split(".", each.value.ip)[3])
 machine = "q35"
 ...

 bios = "ovmf"
 efi_disk {
 datastore_id = var.proxmox_datastore
 type = "4m"
 }

 memory {
 dedicated = each.value.memory
 floating = 0
 }

 disk {
 datastore_id = var.proxmox_datastore
 interface = "scsi0"
 size = each.value.disk_size
 file_id = proxmox_download_file.talos_image[coalesce(each.value.schematic_id, var.talos_image_factory_id)].id
 discard = "on"
 ssd = true
 iothread = true
 ...
 }

 initialization {
 datastore_id = var.proxmox_datastore
 ip_config {
 ipv4 {
 address = "${each.value.ip}/24"
 gateway = var.talos_gateway
 }
 }
 dns {
 servers = var.talos_nameservers
 }
 }
 ...
}

There is no clone block. The system disk is the image downloaded above, referenced by its file_id, and that is what replaces the template.
machine = "q35" and bios = "ovmf" with an efi_disk. The VMs boot in UEFI, and the OVMF firmware needs an EFI disk to store its variables.
The vm_id is derived from the last octet of the IP. 192.168.1.123 gives VM 123, which lets me map the Proxmox interface to the network directly.
floating = 0 disables ballooning, which is mandatory for the GPU node in part 2.

📝 Note

The cloud-init drive of a Talos node is not used to configure Talos. It only gives the node a static IP at first boot, so that it is reachable in maintenance mode. All the configuration is then applied through the Talos API, via the siderolabs/talos provider.

The VMs are ready to boot, now they need their configuration.

The machine configuration

Generating the secrets and the base configuration

The siderolabs/talos provider chains six resources, in this order:

1
2
3
4
5
6


talos_machine_secrets # cluster CA, keys and tokens
 └─> data.talos_client_configuration # the talosconfig for talosctl
 └─> data.talos_machine_configuration # a base config, per role
 └─> talos_machine_configuration_apply # applied per node, with the patches
 └─> talos_machine_bootstrap # etcd init, only once
 └─> talos_cluster_kubeconfig # the admin kubeconfig

talos_machine_secrets generates all the cryptographic material of the cluster: the Kubernetes, Talos and etcd CAs, and the node enrolment tokens. These secrets are never written to disk, they are stored in the OpenTofu state and only leave it through outputs.

The bootstrap and the kubeconfig retrieval have no for_each, because they are operations on the whole cluster. They target a single control plane, the first one in the list:

1
2
3
4
5


#talos.tf
locals {
 talos_controlplanes = [for node in var.talos_nodes : node if node.role == "controlplane"]
 talos_first_controlplane = local.talos_controlplanes[0].ip
}

The talos_machine_configuration_apply resource gets its node argument from the talos_nodes variable, not from a VM attribute. OpenTofu therefore sees no dependency between the two, and may try to apply the configuration to an IP that does not answer yet. So I added an explicit depends_on on the VMs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


#talos.tf
resource "talos_machine_configuration_apply" "this" {
 depends_on = [proxmox_virtual_environment_vm.talos]
 for_each = { for node in var.talos_nodes : node.hostname => node }

 client_configuration = talos_machine_secrets.this.client_configuration
 machine_configuration_input = data.talos_machine_configuration.this[each.value.role].machine_configuration
 node = each.value.ip

 config_patches = [
 for patch in local.talos_patch_files[each.key] :
 templatefile(patch, {
 hostname = each.key
 node_ip = each.value.ip
 role = each.value.role
 vip = var.talos_vip
 gateway = var.talos_gateway
 nameservers = var.talos_nameservers
 })
 ]
}

Patches, on three levels

A list of patches per role is no longer enough as soon as a single node has specific hardware. So I organised the patches on three levels, from the most general to the most specific:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


tofu/patches/
├── common/ # all nodes
│ └── 40-kubelet.yaml.tftpl
├── controlplane/ # control planes
│ ├── 10-network-vip.yaml.tftpl
│ ├── 20-cluster-cni-proxy.yaml.tftpl
│ └── 30-extra-manifests.yaml.tftpl
├── worker/ # workers
└── nodes/
 └── talos-gpu-01/ # this specific node
 └── 30-nvidia-modules.yaml.tftpl

The list of patches for each node is built with fileset:

1
2
3
4
5
6
7
8
9


#talos.tf
talos_patch_files = {
 for node in var.talos_nodes :
 node.hostname => concat(
 [for f in fileset("${path.module}/patches/common", "*.yaml.tftpl") : "${path.module}/patches/common/${f}"],
 [for f in fileset("${path.module}/patches/${node.role}", "*.yaml.tftpl") : "${path.module}/patches/${node.role}/${f}"],
 [for f in fileset("${path.module}/patches/nodes/${node.hostname}", "*.yaml.tftpl") : "${path.module}/patches/nodes/${node.hostname}/${f}"],
 )
}

This relies on three properties of fileset:

Files are sorted alphabetically, so the numeric prefix of each file sets the merge order. For a given key, the last patch wins: the node level overrides the role level, which overrides the common level. I leave gaps in the numbering (10, 20, 30) so I can insert a patch later.
A missing directory returns an empty set, without error. All three levels are therefore optional, which is handy since Git does not track empty directories.
Adding a patch requires no code change. You just drop a file in the right directory, talos.tf stays the same.

The GPU node is what led me to create the nodes/ level: an NVIDIA patch placed under worker/ would also be applied to talos-worker-01, whose image does not contain the extensions, and that node would fail to load the modules at every boot.

The VIP

Talos manages the VIP itself, without keepalived or an external load balancer. The control planes choose the node that holds the VIP through an election in etcd: only one holds it at any given time, and it fails over automatically to another control plane if that node goes away.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# patches/controlplane/10-network-vip.yaml.tftpl
machine:
 network:
 interfaces:
 - deviceSelector:
 physical: true
 dhcp: false
 addresses:
 - ${node_ip}/24
 routes:
 - network: 0.0.0.0/0
 gateway: ${gateway}
 vip:
 ip: ${vip}
 nameservers: ${jsonencode(nameservers)}

The interface is declared again in full, not just completed with a vip entry. The machine config takes precedence over the network configuration provided by the nocloud platform: a patch that only contained the VIP would wipe out the static addressing provided by the cloud-init drive, and the node would become unreachable.

cluster_endpoint also points to the VIP, not to a node IP. This value is written into the configuration of every node and into the kubeconfig: if it designated a specific node, losing that node would cut off access to the API, and adding a control plane would mean reconfiguring the whole cluster to change the endpoint.

CNI and kube-proxy disabled

1
2
3
4
5
6
7


# patches/controlplane/20-cluster-cni-proxy.yaml.tftpl
cluster:
 network:
 cni:
 name: none
 proxy:
 disabled: true

By default, Talos installs Flannel as the CNI and kube-proxy. I replace both with Cilium, so I disable them.

I chose Cilium for three reasons: it replaces kube-proxy, it handles L2 announcements, which I use later in the series to announce the IPs of LoadBalancer services on my local network, and it is all built on eBPF, a technology I want to rely on as my homelab evolves.

This choice has two consequences:

the nodes stay NotReady as long as no CNI is installed, which is expected
without kube-proxy, the CNI has to reach the API server directly, because the ClusterIP of the kubernetes.default service is not routable yet when it starts. That is the job of KubePrism, which I present in the section on Cilium.

This patch is placed on the control planes because these manifests are generated at bootstrap, from a control plane. Like everything under cluster.*, it only has an effect at that moment: applied to a cluster that is already running, it removes nothing.

Kubelet serving certificates

By default, the kubelet exposes its :10250 endpoint with a self-signed certificate that no cluster CA recognises. Hence the --kubelet-insecure-tls option found in most metrics-server installations: the client does not verify the server certificate, so any machine able to get in the middle on the network can pretend to be a kubelet.

The first part of the solution is on the node side:

1
2
3
4
5


# patches/common/40-kubelet.yaml.tftpl
machine:
 kubelet:
 extraArgs:
 rotate-server-certificates: true

With this option, the kubelet no longer generates a self-signed certificate and sends a CSR to the API server, with the kubernetes.io/kubelet-serving signer. The problem is that Kubernetes never automatically approves CSRs for this signer. The kube-controller-manager can sign the certificate, but no cluster component approves it: the CSRs stay Pending and the kubelet keeps its self-signed certificate. This is a deliberate choice of the Kubernetes project, which leaves that decision to each environment.

The second part is therefore on the cluster side:

1
2
3
4
5


# patches/controlplane/30-extra-manifests.yaml.tftpl
cluster:
 extraManifests:
 - https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/standalone-install.yaml
 - https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml

kubelet-serving-cert-approver is a controller that approves these CSRs. It only approves them: the signing is still done by the kube-controller-manager, with the cluster CA key. Before approving a request, it checks that it really matches a kubelet’s:

the subject organisation is system:nodes
the CN starts with system:node: and exactly matches the user who submitted the CSR
the requested usages are those of a server certificate
the request contains no email or URI SAN

However, it does not check that the IP addresses requested in the SANs actually belong to the node, since its ClusterRole gives it no access to Node objects. A node already enrolled in the cluster can therefore obtain a valid certificate for another node’s IP.

The security gain is real but limited: before, any machine on the network could pretend to be a kubelet, whereas now you need the identity of a cluster node. In my case, metrics-server now verifies the kubelet certificates against the cluster CA, without --kubelet-insecure-tls.

Both manifests must stay deployed permanently, because certificate rotation is continuous and each renewal produces a new CSR to approve. The URLs are pinned to a specific version, so that a re-apply does not fetch different content and so that Renovate can track updates.

The configuration is complete, I can move on to the deployment.

Deployment

I do not run any apply locally. The deployment goes through a GitLab CI pipeline that uses the GitLab OpenTofu component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


#.gitlab-ci.yml
default:
 tags:
 - homelab

variables:
 TF_VAR_proxmox_endpoint: '${proxmox_endpoint}'
 TF_VAR_proxmox_api_token: '${proxmox_api_token}'
 TF_VAR_proxmox_ssh_private_key: '${proxmox_ssh_private_key}'

include:
 # renovate: datasource=gitlab-releases packageName=components/opentofu registryUrl=https://gitlab.com
 - component: $CI_SERVER_FQDN/components/opentofu/validate-plan-apply@4.5.0
 inputs:
 # renovate: datasource=gitlab-releases packageName=components/opentofu registryUrl=https://gitlab.com
 version: 4.5.0
 opentofu_version: 1.11.5
 root_dir: tofu/

stages: [validate, build, deploy]

workflow:
 rules:
 - changes:
 - tofu/**
 - when: never

The validate-plan-apply template adds four jobs: fmt and validate in the validate stage, plan in build, then apply in deploy. By default, apply only exists on the default branch and must be started by hand: a merge request therefore stops at the plan, and nothing is applied until I trigger the job.

The rest of the configuration comes down to five points:

The jobs run on my own runner, thanks to the homelab tag: a GitLab runner with the Docker executor, hosted on my Proxmox. It is on the same network as the Proxmox API, which GitLab’s shared runners could not reach.
The state is managed by GitLab. The backend "http" {} block in providers.tf is empty, and the component fills in the project’s state address and the job credentials at run time.
The secrets stay in GitLab. The Proxmox API token and the SSH key are CI/CD variables, passed to OpenTofu through the TF_VAR_* variables.
The pipeline only runs if tofu/ changes, thanks to the workflow rule.
Versions are tracked by Renovate, thanks to the # renovate: comments above the component.

Retrieving the credentials

The cluster credentials are not written anywhere on disk, they are in the state, so they have to be extracted through the outputs. Since the state is stored in GitLab, I run a local tofu init once, only for this step. GitLab provides the full init command under Operate > Terraform states, with the Copy Terraform init command action. I then retrieve both files:

1
2


tofu output -raw talosconfig > ../talos/config
tofu output -raw kubeconfig > ../talos/kubeconfig

The talosconfig generated by data.talos_client_configuration contains two lists: the endpoints, which talosctl actually connects to and which only contain control planes, and the nodes, which are the default targets of commands and are reached through an endpoint. This is what lets you query a worker without connecting to it directly.

For example, to list the services of talos-worker-01:

1

talosctl --talosconfig talos/config -e 192.168.1.121 -n 192.168.1.122 services

talosctl opens the connection to the control plane 192.168.1.121, which relays the request to the worker 192.168.1.122. Without -e, the endpoints from the talosconfig are used, and -n is enough to change the target.

Endpoint and node with talosctl

Installing Cilium

The cluster starts without a CNI, so the nodes are NotReady. I install Cilium with Helm, as a replacement for kube-proxy:

1
2
3
4
5
6


helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium \
 --version 1.20.2 \
 --namespace kube-system \
 --values cilium-values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


#cilium-values.yaml
kubeProxyReplacement: true
k8sServiceHost: localhost
k8sServicePort: 7445

# Talos monte lui-même le cgroup2 et interdit à l'agent de le remonter.
cgroup:
 autoMount:
 enabled: false
 hostRoot: /sys/fs/cgroup

securityContext:
 capabilities:
 ciliumAgent:
 - CHOWN
 - KILL
 - NET_ADMIN
 - NET_RAW
 - IPC_LOCK
 - SYS_ADMIN
 - SYS_RESOURCE
 - DAC_OVERRIDE
 - FOWNER
 - SETGID
 - SETUID
 cleanCiliumState:
 - NET_ADMIN
 - SYS_ADMIN
 - SYS_RESOURCE

The cgroup and securityContext values are specific to Talos. By default, the chart mounts the cgroup2 itself and runs the agent in privileged mode, which Talos does not allow. I took the list of capabilities from the Talos documentation for Cilium rather than rebuilding it.

That leaves the k8sServiceHost / k8sServicePort pair. You could point Cilium at the VIP on 6443, since it is the cluster endpoint, but that is not what Talos recommends. Cilium has to reach the API server before the pod network exists, and the VIP depends on the etcd election between the control planes.

KubePrism solves this problem: it is a local load balancer present on every node, listening on localhost:7445. It automatically picks among the available endpoints (the cluster endpoint and each control plane), discarding those that do not answer and favouring the fastest ones. It is enabled by default, and the Talos documentation recommends giving it as the endpoint to CNIs that need to reach the API server, such as Cilium.

Access to the Kubernetes API

Verification

Once Cilium is installed, I check the state of the cluster on the Talos side with talosctl health. I removed the progress lines from the output, to keep only the result of each check:

1

talosctl --talosconfig talos/config -n 192.168.1.121 health

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


discovered nodes: ["192.168.1.120" "192.168.1.123" "192.168.1.122"]
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: OK
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: OK
waiting for no diagnostics: OK
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: OK
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: OK
waiting for all k8s nodes to report ready: SKIP
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: SKIP
waiting for all k8s nodes to report schedulable: OK

The first address listed is the VIP and not 192.168.1.121: the control plane holding it advertises it among its addresses, and it is sorted first.

The three SKIPs come from the choices made in the patches. talosctl health does not check the Ready state of the nodes or CoreDNS when the CNI is set to none, since these checks depend on a CNI that Talos did not install itself, and it skips kube-proxy when its DaemonSet does not exist. The state of the nodes is therefore checked on the Kubernetes side:

1

kubectl get nodes

1
2
3
4


NAME STATUS ROLES AGE VERSION
talos-controlplane-01 Ready control-plane 47d v1.36.2
talos-gpu-01 Ready <none> 47d v1.36.2
talos-worker-01 Ready <none> 47d v1.36.2

All three nodes are Ready, which confirms that Cilium is working on each of them.

Pitfalls

The Talos learning curve

Getting to grips with Talos took me some time. Without a shell or SSH, none of my usual habits for troubleshooting a node apply, and everything that used to be done directly on the machine has to go through talosctl. So you have to learn a new CLI, new concepts and new naming.

A first deployment without the Talos provider

My first deployment did not use the siderolabs/talos provider: OpenTofu created the VMs, and I generated and applied the configurations by hand with talosctl. Switching to the provider brought the whole node configuration back as code, along with the rest of the infrastructure, and makes an end-to-end deployment possible in a single OpenTofu run.

A single schematic for all nodes

At first, all the nodes used the same schematic, the one containing the NVIDIA extensions. The nodes without a GPU then reported errors related to the NVIDIA drivers, since they shipped modules meant for a card they do not have. That is what led me to give each node a schematic containing only the extensions it needs.

Conclusion

The cluster is up and running, and compared to Kubespray, I no longer have an OS to administer: no package updates, no open SSH, and the node configuration cannot drift from what the code describes. A Talos update comes down to changing a version in a variable, and each node switches to the new image.

Summary of the deployment workflow

Once everything was in place, what surprised me most was how easily and quickly a Kubernetes cluster comes up. Where Kubespray ran close to 600 Ansible tasks, a single pipeline run creates the VMs, applies the configurations and bootstraps the cluster, and all that is left is to install Cilium.

The talos-gpu-01 node is already running, with its 12 GB of reserved RAM and its image containing the NVIDIA modules, but it does not see any graphics card yet. Getting the GPU from the Proxmox host all the way to the pods is the subject of part 2.

Useful links

Deploying a Kubernetes cluster with Ansible

Mon, 24 Nov 2025 10:20:38 +0000

Introduction

In my homelab, I wanted to take automation further by deploying a complete Kubernetes cluster on Proxmox, relying on infrastructure as code and automation tools such as Ansible. The goal was to be able to quickly provision a reproducible k8s cluster, ready to host workloads, while keeping control over every step of the deployment through code. This progression naturally led me to Kubespray.

Context

After first installing a Kubernetes cluster manually with kubeadm, I found the process particularly tedious. That experience convinced me of the need to automate and standardise the deployment, to gain reproducibility and efficiency.

This is where Kubespray comes in: this open source project provides ready-to-use Ansible playbooks to install, configure and maintain a Kubernetes cluster with a 100% as code approach. Unlike kubeadm, it offers fine-grained control over the cluster components (CRI, CNI, DNS, etc.) while remaining automated and reproducible. In this article, I describe the approach I followed, the technical choices I made and the tools I used to reach this goal.

Kubespray logo

Prerequisites

Before starting the deployment, you need the following resources:

Infrastructure:

At least 24 GB of RAM available on the Proxmox hypervisor
150 GB of free storage
Network access with static IP assignment
An SSH key to access the VMs

Components

Proxmox 8.X+: provides the infrastructure to run and manage virtual machines.
Ansible 2.16+: the automation tool used by Kubespray to deploy and configure Kubernetes.
Terraform 1.10+ & Packer 1.12+: to automate the creation and configuration of the VMs.

Target cluster architecture

The cluster is made of three nodes with the following sizing:

Role	Hostname	vCPU	RAM	Storage
Control Plane	k8s-master-01	4	8 GB	50 GB
Worker Node	k8s-worker-01	4	8 GB	50 GB
Worker Node	k8s-worker-02	4	8 GB	50 GB

Target architecture

Creating the VMs

For my homelab, I went for a simple architecture made of three VMs: one control plane and two worker nodes.

There are several reasons for this choice:

Simplicity for learning: with three nodes, the cluster stays simple to manage while still allowing you to try most Kubernetes features (load balancing, pod affinity, fault tolerance, etc.).
Worker fault tolerance: two workers make it possible to test node failure scenarios and observe how Kubernetes behaves when rescheduling pods.
Resource control: this sizing stays reasonable in terms of resources consumed (4 vCPU, 8 GB RAM, 50 GB storage per VM), which matters in a homelab.

📝 Note

High availability: this architecture has a single control plane. For real high availability in production, you would need at least 3 control planes (an odd number for the etcd quorum) with a load balancer in front of the API servers, or dedicated etcd instances outside the cluster. For this homelab, a single control plane is enough for learning and experimenting.

With the sizing defined, let’s create the VMs. For this, I used Terraform and Packer, which lets me deploy my infrastructure quickly and in a fully automated way. I explain this approach in detail, in particular building Proxmox templates with Packer and deploying them with Terraform, in my article on Proxmox templates with Packer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


############################
# VMs for Kubernetes Cluster
############################
resource "proxmox_virtual_environment_vm" "k8s_cluster" {
 for_each = {
 k8s-master = {
 description = "K8S Control Plane"
 vm_id = 110
 ip_address = "192.168.1.110/24"
 tags = ["k8s_cluster", "kube_control_plane", "etcd"]
 }
 k8s-node1 = {
 description = "K8S Worker Node 1"
 vm_id = 111
 ip_address = "192.168.1.111/24"
 tags = ["k8s_cluster", "kube_node"]
 }
 k8s-node2 = {
 description = "K8S Worker Node 2"
 vm_id = 112
 ip_address = "192.168.1.112/24"
 tags = ["k8s_cluster", "kube_node"]
 }
 }

 name = each.key
 description = each.value.description
 tags = concat(["debian12", "terraform"], each.value.tags)
 node_name = "pve"
 vm_id = each.value.vm_id
 pool_id = "PROD"
 reboot_after_update = false

 ...

 clone {
 vm_id = 1101
 datastore_id = "local-lvm"
 full = true
 node_name = "pve"
 }
}

Once the Terraform code is applied, the three virtual machines are deployed on Proxmox. Each VM gets one or more of these tags: (k8s_cluster, kube_control_plane, etcd, kube_node). These tags make it easier to generate the Kubespray inventory, which I describe further down in the article.

VMs in Proxmox

Network configuration and SSH access

Thanks to Cloud-Init during the VM creation process (through the Packer template and the initialization argument of the proxmox_virtual_environment_vm resource of the Terraform provider), the network configuration and SSH access are automated:

Static IP address: each VM gets its static IP directly at deployment, defined in the Terraform ip_config section. This avoids any manual configuration after deployment.
SSH key: the public SSH key is injected automatically into each VM through Cloud-Init, which allows an immediate connection without copying the key by hand.

This process makes it possible to manage all the VMs with Ansible as soon as they are created, without any further manual step. In my case, I can therefore use the Kubespray playbooks directly.

⚠️ Warning

In a production environment, it is recommended to manage secrets (Proxmox API tokens, SSH keys, etc.) with dedicated tools such as Ansible Vault or external secret managers.

The resources are now ready and configured, we can move on to setting up the Kubernetes cluster with Kubespray.

Setting up Kubespray

Cloning and configuring the Kubespray repository

I added Kubespray to my project as a git submodule. This lets you include another repository while keeping its history separate. It makes dependency management easier and lets you track a specific version of Kubespray, update it easily, and follow the project’s changes without mixing source code.

1

git submodule add https://github.com/kubernetes-sigs/kubespray.git kubespray

To pin the submodule to a specific tag or commit, go into the submodule directory and check out the desired commit or tag. Then go back to the main project directory, commit the changes and push them.

1
2
3
4


cd kubespray/
git checkout v2.27.0 # Version used for this deployment
cd ../
git commit -m "Pinned submodule to v2.27.0"

Pinning the Kubespray version is important to guarantee reproducible deployments and avoid any instability caused by uncontrolled changes in the upstream project.

Once the Kubespray project is integrated and configured in the Ansible repository, we can move on to generating the Kubespray inventory.

Creating the Kubespray inventory

For the Ansible inventory used by Kubespray, I chose the Proxmox dynamic inventory. This gives an inventory that is always up to date, since it is loaded at every playbook run, which avoids maintaining the inventory file by hand.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


plugin: community.general.proxmox
url: https://<proxmox_host>:8006
user: "<user>"
token_id: "<token_id>"
token_secret: "<token_secret>"
validate_certs: false # Only in a homelab
want_facts: true
keyed_groups:
 - key: proxmox_tags_parsed
 separator: ""
compose:
 ansible_host: proxmox_agent_interfaces[1]["ip-addresses"][0].split('/')[0]

The inventory above retrieves all the VMs in Proxmox, groups them by their proxmox_tags_parsed tags and sets the ansible_host variable of each inventory entry to the IP of each VM.

For Kubespray, the tags must follow the naming described in the documentation, because that is what runs the playbooks on the right hosts.

To generate the inventory, I use the command below, where the --graph option makes the output a bit more visual, to clearly tell apart the groups and hosts of the Ansible inventory.

1

ansible-inventory -i inventories/proxmox.yml --graph

In my case, this gives the following result:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


@all:
 ...
 ...
 |--@debian12:
 | |--k8s-node1
 | |--k8s-master
 | |--k8s-node2
 |--@k8s_cluster:
 | |--k8s-node1
 | |--k8s-master
 | |--k8s-node2
 |--@kube_node:
 | |--k8s-node1
 | |--k8s-node2
 |--@terraform:
 | |--k8s-node1
 | |--k8s-master
 | |--k8s-node2
 |--@etcd:
 | |--k8s-master
 |--@kube_control_plane:
 | |--k8s-master
 ...
 ...

Once the inventory is in place, I can test the connectivity of the VMs with the ad hoc command below, which uses the Ansible ping module:

1

ansible -i inventories/proxmox.yml k8s_cluster -m ping -u <username>

Which gives the following result:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


k8s-node1 | SUCCESS => {
 "changed": false,
 "ping": "pong"
}
k8s-master | SUCCESS => {
 "changed": false,
 "ping": "pong"
}
k8s-node2 | SUCCESS => {
 "changed": false,
 "ping": "pong"
}

Now that I have the Proxmox dynamic inventory and reachable virtual machines, I can move on to customising my K8s cluster.

Customising the variables

To customise the Kubernetes cluster, you go through Ansible group_vars. These variables let us configure various aspects of the cluster, such as:

Kubernetes settings (kube version, internal network, DNS, …)
Container Runtime Interface (CRI)
Container Network Interface (CNI)
Container Storage Interface (CSI)
ETCD configuration
Addons and optional components (ArgoCD, Helm, cert-manager, MetalLB, …)
And much more…

These variables are spread across several inventory files to better structure the configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


inventories/group_vars/
├── etcd
│ ├── all.yml
│ └── etcd.yml
└── k8s_cluster
 ├── addons.yml
 ├── all.yml
 ├── cri-o.yml
 ├── k8s-cluster.yml
 └── k8s-net-flannel.yml

For more information about the possible cluster configuration and the available components, refer to the Kubespray documentation.

📝 Note

In this article, I mostly use the default configurations provided by Kubespray. The goal is to show the deployment process rather than justify specific technical choices for each component (CNI, CRI, etc.). These choices can be adapted to the specific needs of each project.

The target configuration of the Kubernetes cluster is defined, the next step is the deployment!

Deploying the Kubernetes cluster

Before running the deployment with Ansible, I need to set up my environment. To do this, I use a Python venv to pin the version of Ansible and of my dependencies, so that everything is isolated and potential conflicts with other projects are avoided. The venv also gives a reproducible environment, which is crucial to run deployments identically.

1
2
3


python3 -m venv venv
source venv/bin/activate
pip install -r kubespray/requirements.txt

Once the environment is ready, it is finally time to deploy the cluster, by running the cluster.yml playbook with the Proxmox dynamic inventory configured earlier.

1

ansible-playbook -i inventories/proxmox.yml -b -u <username> kubespray/cluster.yml

Once the playbook has finished, the Kubernetes cluster is ready to use with the configuration defined above.

1
2
3
4


PLAY RECAP *************************************************************************
k8s-master: ok=599 changed=39 unreachable=0 failed=0 skipped=965 rescued=0 ignored=1
k8s-node1 : ok=421 changed=25 unreachable=0 failed=0 skipped=618 rescued=0 ignored=1
k8s-node2 : ok=443 changed=28 unreachable=0 failed=0 skipped=681 rescued=0 ignored=1

Checking the cluster

Once the deployment is complete, it is essential to check that the Kubernetes cluster is operational and that all components work correctly. Several kubectl commands let you validate the state of the nodes and pods.

To check the state of the cluster nodes:

1

kubectl get nodes

Result:

1
2
3
4


NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane 1d v1.31.4
k8s-node1 Ready worker 1d v1.31.4
k8s-node2 Ready worker 1d v1.31.4

All nodes must be in the Ready state, which means they are ready to run workloads.

To make sure all pods are running, use the command:

1

kubectl get pods -A

Result:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


NAMESPACE NAME READY STATUS RESTARTS AGE
external-dns external-dns-754bcb687f-cqncz 1/1 Running 0 24h
ingress-nginx ingress-nginx-controller-6d7ff9f49f-2b9dq 1/1 Running 0 24h
kube-system coredns-d665d669-7h6nf 1/1 Running 0 24h
kube-system coredns-d665d669-fklwz 1/1 Running 0 24h
kube-system dns-autoscaler-5cb4578f5f-zkr67 1/1 Running 0 24h
kube-system kube-apiserver-k8s-master 1/1 Running 0 24h
kube-system kube-controller-manager-k8s-master 1/1 Running 0 24h
kube-system kube-flannel-9n6zs 1/1 Running 0 24h
kube-system kube-flannel-m5xtr 1/1 Running 0 24h
kube-system kube-flannel-qchzm 1/1 Running 0 24h
kube-system kube-proxy-54k4k 1/1 Running 0 24h
kube-system kube-proxy-qpb4m 1/1 Running 0 24h
kube-system kube-proxy-zq7ww 1/1 Running 0 24h
kube-system kube-scheduler-k8s-master 1/1 Running 0 24h
kube-system nginx-proxy-k8s-node1 1/1 Running 0 24h
kube-system nginx-proxy-k8s-node2 1/1 Running 0 24h
kube-system nodelocaldns-l2nc4 1/1 Running 0 24h
kube-system nodelocaldns-pp8ng 1/1 Running 0 24h
kube-system nodelocaldns-x522v 1/1 Running 0 24h
kubernetes-dashboard kubernetes-dashboard-8984b9757-2pcmz 1/1 Running 0 24h
kubernetes-dashboard kubernetes-metrics-scraper-6d4c5d99f9-z6g6b 1/1 Running 0 24h
metallb-system controller-7f649565d4-6dcd8 1/1 Running 0 24h
metallb-system speaker-6pj99 1/1 Running 0 24h
metallb-system speaker-v44zd 1/1 Running 0 24h
metallb-system speaker-w8wzp 1/1 Running 0 24h

All pods must be in the Running state. This guarantees that the cluster components are correctly deployed and working.

These checks confirm that the Kubernetes cluster is ready to host workloads and that the essential services are up.

Result

Kubernetes cluster after deployment

Post-deployment operations

Lifecycle management

Kubespray is not limited to the initial deployment, it also provides playbooks to manage the full lifecycle of the cluster:

Adding nodes:

1
2


# To add a new worker node to the cluster
ansible-playbook -i inventories/proxmox.yml -b -u <username> kubespray/scale.yml

Upgrading the cluster:

1
2


# To upgrade the Kubernetes version or the components
ansible-playbook -i inventories/proxmox.yml -b -u <username> kubespray/upgrade-cluster.yml

Full reset:

1
2


# To remove the cluster completely
ansible-playbook -i inventories/proxmox.yml -b -u <username> kubespray/reset.yml

Conclusion

Automating the deployment of a Kubernetes cluster with Ansible through the Kubespray project brings reliability, speed and reproducibility, which is essential for me in my homelab.

Thanks to the dynamic inventory and the Ansible variables, it is easy to adapt the cluster configuration to my needs, while keeping control over each component. This approach also makes maintaining and evolving the k8s cluster easier, since everything is done as code and Kubespray also provides playbooks to add a node to the cluster (scale.yml) or even reset the cluster (reset.yml).

Workflow

Workflow summary

Useful links

VM template in Proxmox

Fri, 28 Feb 2025 10:20:38 +0000

Introduction

In my homelab, I recently worked on a project that let me create virtual machine (VM) templates on Proxmox using a combination of tools such as Packer, Cloud-Init and Terraform. The goal was to create VMs in an automated and repeatable way, while keeping manual steps to a minimum.

Context

Managing VMs and their configuration by hand can quickly become tedious, especially when deploying several instances with almost identical configurations every time. So I needed a solution to standardise this whole process and have a ready-to-use VM template to serve as the base for all my VMs.

In the rest of this article, I explain how I reached this goal with different tools, in order to streamline deployments in my homelab.

Components

Proxmox: provides the infrastructure to run and manage virtual machines.
Packer: automates the installation and configuration of the OS to produce a standardised image or template.
Terraform: defines, provisions and manages infrastructure through declarative code.
Cloud-Init: configures machines dynamically at first boot.

Packer

Packer is a community tool for creating identical machine images for several platforms from a single source configuration. To do this, Packer uses what are called builders, Packer components able to create a virtual machine template or image on a specific platform.

Packer logo

In my case, I simply use Packer on my Proxmox environment through its dedicated builder.

1
2
3
4
5
6
7
8
9


#packer.pkr.hcl
packer {
 required_plugins {
 name = {
 version = "1.2.2"
 source = "github.com/hashicorp/Proxmox"
 }
 }
}

Template configuration

This builder can either clone a virtual machine to turn it into a template (Proxmox-clone) or create a virtual machine template from a source ISO (Proxmox-iso). To meet my initial need, I use the Proxmox-iso builder. For the configuration, I used the builder’s documentation.

Example packer.pkr.hcl file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66


#packer.pkr.hcl
source "Proxmox-iso" "debian12" {
 # Proxmox server
 Proxmox_url = "proxmox_api_url"
 node = "pve"
 username = "<Proxmox_token_id>"
 token = "<Proxmox_token>"
 insecure_skip_tls_verify = true

 # VM Info
 template_name = "pckr-debian-12"
 vm_name = "pckr-debian-12"
 template_description = "Debian 12.9.0, generated on ${timestamp()}"
 tags = "template;debian12"
 vm_id = 1101
 pool = "TEMPLATE"
 os = "l26"
 machine = "q35"

 # Boot
 boot_iso {
 type = "scsi"
 iso_file = "local:iso/debian-12.9.0-amd64-netinst.iso"
 unmount = true
 iso_checksum = "none"
 }
 boot_command = ["<esc><wait>auto console-keymaps-at/keymap=fr console-setup/ask_detect=false debconf/frontend=noninteractive fb=false url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg<enter>"]
 boot_wait 	 = "10s"
 http_directory = "http"

 # Disks
 disks {
 type = "scsi"
 storage_pool = "local-lvm"
 disk_size = "30G"
 format = "raw"
 }

 # Networks
 network_adapters {
 bridge = "vmbr0"
 model = "virtio"
 }

 # Spec
 bios = "seabios"
 qemu_agent = true
 scsi_controller = "virtio-scsi-pci"
 cpu_type = "x86-64-v2-AES"
 sockets = 1
 cores = 1
 memory = 1024
 ballooning_minimum = 1024

 # Communication
 communicator = "ssh"
 ssh_handshake_attempts = 5
 ssh_pty = true
 ssh_timeout = "10m"
 ssh_username = <username>
 ssh_password = <password>

 # Cloud-init
 cloud_init = "true"
 cloud_init_storage_pool = "local-lvm"
}

In this packer.pkr.hcl file, I give the builder the information about the target machine and its specifications for Proxmox, as well as the boot information (path of the ISO file, boot command, etc.).

Preseed file

The boot section gives the virtual machine, the future template, the path of the preseed.cfg file, served by a web server that Packer starts during the process. This file answers the questions asked during the OS installation, without having to type the answers by hand while the installation runs, which fully automates the installation. An example of this file is available in the Debian documentation.

Building the template

Once the configuration is ready, build the template with the Packer CLI using the commands below:

1
2
3
4
5
6
7
8


# Initialise the repository to fetch the dependencies
packer init packer.pkr.hcl

# Validate the syntax and configuration
packer validate packer.pkr.hcl

# Build the image, replacing the template if it already exists
packer build -on-error=ask -force packer.pkr.hcl

The template is now present in Proxmox with all the characteristics I set in the packer.pkr.hcl file, and it is ready to be used for future virtual machine deployments.

Template in Proxmox

Now on to the next step: dynamic configuration with Cloud-Init!

Cloud-init

Cloud-Init is an open source tool that automates the initial configuration of virtual machines at their first boot. Using configuration files (user-data and meta-data), Cloud-Init can perform various tasks such as creating users, injecting SSH keys, configuring the network or installing packages. This tool works with many cloud platforms and virtualisation environments, which makes it an agnostic way to customise instances dynamically without manual intervention.

Cloud-Init logo

Example user-data file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


#user-data.yml
#cloud-config
ssh_pwauth: false
disable_root: true
users:
 - name: user
 gecos: user
 groups: users, sudo
 shell: /bin/bash
 ssh_authorized_keys:
 - ssh-ed25519 AAAA.......
 sudo: ALL=(ALL) NOPASSWD:ALL
 lock_passwd: true
package_update: true
package_upgrade: true

In my case, I chose to use cloud-init to:

Disable SSH password authentication as well as the root account.
Create a user with an authorised SSH key, passwordless sudo access and a locked password.
Automatically update and upgrade the packages.

Once applied at the machine’s first boot, the machine is ready to use, and also ready to be provisioned with a tool such as Ansible. That was my goal, since I wanted to run my Ansible playbooks over an SSH key connection once the virtual machine was deployed, without copying my key by hand or having to log in with a password first.

Terraform

To deploy my virtual machine from my template and the cloud-init file, I chose to use Terraform. Terraform is an infrastructure as code tool, meaning a tool that lets you define your infrastructure through declarative configuration files in HCL (HashiCorp Configuration Language). To do this, Terraform relies on providers that handle communication with the underlying infrastructure. This method lets you manage infrastructure like code, which makes versioning and reproducible deployments easier.

These plugins expose resource types (create/modify resources) and data sources (read existing information) that let Terraform manage the infrastructure behind the plugin. Providers use the vendor’s SDK or API to carry out all the actions described in the Terraform configuration.

Terraform logo

For my homelab, I chose the bpg provider, a provider built by a community member that manages the whole Proxmox environment (resource pools, users, VMs, ISOs).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


#providers.tf
provider "proxmox" {
 endpoint = var.endpoint
 api_token = var.api_token
 insecure = true
 ssh {
 agent = false
 username = "root"
 private_key = var.private_key
 }
}

To deploy the virtual machine from the template, with cloud-init for provisioning, I used the proxmox_virtual_environment_vm resource, which creates and manages a virtual machine with all the characteristics you pass to it.

Example Terraform resource for a virtual machine in Proxmox:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


#virtual_machine.tf
resource "proxmox_virtual_environment_vm" "example_vm" {
 name = "example"
 description = "example VM"
 tags = ["debian12", "terraform"]
 node_name = "pve"
 vm_id = 101
 pool_id = "example"
 reboot_after_update = false
 
 agent {
 enabled = true
 }
 
 cpu {
 cores = 2
 sockets = 1
 type = "x86-64-v2-AES"
 }
 
 memory {
 dedicated = 4096
 floating = 4096
 }

 initialization {
 user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id
 dns {
 servers = ["9.9.9.9", "1.1.1.1"]
 }
 ip_config {
 ipv4 {
 address = "192.168.1.101/24"
 gateway = "192.168.1.1"
 }
 }
 }

 clone {
 vm_id = 1101
 datastore_id = "local-lvm"
 full = true
 node_name = "pve"
 }
}

In my case, the parameters I care about are:

initialization: the cloud-init configuration, where I set the DNS configuration and the IP address of my machine, and give the ID of my cloud-init file, also managed with Terraform through the proxmox_virtual_environment_file resource

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


initialization {
 user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id
 dns {
 servers = ["9.9.9.9", "1.1.1.1"]
 }
 ip_config {
 ipv4 {
 address = "192.168.1.101/24"
 gateway = "192.168.1.1"
 }
 }
}

clone: the clone configuration, where I give the information about my template in Proxmox.

1
2
3
4
5
6


clone {
 vm_id = 1101
 datastore_id = "local-lvm"
 full = true
 node_name = "pve"
}

I can then run my Infrastructure as Code project with the new resource added, and Terraform clones the template and gives it the cloud-init configuration to provision the virtual machine.

The result is a VM with a standard configuration that I defined to match my needs, ready to be provisioned with a tool such as Ansible.

Sequence diagram

Conclusion

Setting up an automated process to create and deploy virtual machines on Proxmox with Packer, Cloud-Init and Terraform has made my homelab more efficient, reproducible and flexible.

This workflow standardises the environments, reduces the risk of manual errors and makes provisioning the machines with Ansible or other automation tools easier. With this solution, I can now quickly deploy ready-to-use virtual machines, while adapting their configuration to my needs.

Kubernetes secrets from Azure Key Vault

Sat, 23 Nov 2024 10:20:38 +0000

Introduction

Managing secrets securely in Kubernetes is crucial to keeping our applications secure. In this article, I present secret management in Kubernetes using External Secrets Operator and Azure Key Vault with workload identity authentication, with the aim of having a robust and secure solution.

Context

I set up this solution during a one-week project as part of my master’s degree. The goal was to deploy an application stack in the cloud through a GitOps workflow. The project is available in the projet-cicd-m2-do GitLab group.

In this project, an Azure Kubernetes Service (AKS) cluster was deployed with Terraform, with the following configuration required to set up the solution:

1

oidc_issuer_enabled = true

⬆️ Enables the OIDC (OpenID Connect) issuer of the AKS cluster. This means the AKS cluster generates an OIDC token issuer URL, which is essential for the trust relationship I present later in this article.

📝 Note

OIDC is a modern authentication protocol built on the OAuth 2.0 standard. It lets applications verify a user’s identity by relying on an identity provider and retrieve information about that user securely.

1

workload_identity_enabled = true

⬆️ Enables the Azure Workload Identity feature, so that Kubernetes pods can access Azure resources using managed identities.

Components

External Secret Operator (ESO): a Kubernetes operator that integrates an external secret management system.
Azure Key Vault: an Azure cloud service to store secrets and access them securely.
Workload Identity: a method to authenticate to an Azure resource without explicitly configuring credentials.

Azure Key Vault

As soon as the AKS cluster is up, the Azure Key Vault resource has to be deployed. For this, I used Terraform, to do IaC and make my deployment reliable and automated. For the vault and the other Azure resources, I used the azurerm Terraform provider.

The following code creates the vault in the defined resource group and location, with RBAC (Role Based Access Control) enabled to manage access and the SKU (Stock Keeping Unit) set to standard. Other arguments are available, but I preferred to keep it simple given the length of the project, to get a POC quickly.

1
2
3
4
5
6
7
8
9


resource "azurerm_key_vault" "vault" {
 name = join("-", [var.prj, var.org, "vault"])
 location = var.location
 resource_group_name = azurerm_resource_group.aks.name
 tenant_id = data.azurerm_client_config.current.tenant_id

 sku_name = "standard"
 enable_rbac_authorization = true
}

Once the vault is deployed, you can put your secrets in it through the Azure UI or CLI.

Azure Vault UI

Managed Identity

Once the vault is deployed, the role has to be assigned to the right identity so that our secrets can be accessed through it.

Kubelet Identity

When an AKS cluster is created, a user assigned managed identity is created in the resource group that Azure manages automatically and that holds the actual infrastructure resources needed to run the AKS cluster. This managed identity is called kubelet_identity, and it lets the kubelet on the cluster nodes access other Azure resources. It is through this identity that I can access the vault.

Interaction between the Kubelet Identity and Azure resources

I gave this identity read access to the secrets in the vault with the Key Vault Secrets User role:

1
2
3
4
5


resource "azurerm_role_assignment" "keyvault_secrets_user" {
 scope = azurerm_key_vault.vault.id
 role_definition_name = "Key Vault Secrets User"
 principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
}

Identity federation

Once the right permissions are assigned, this identity has to be linked to a service account in Kubernetes so that the data in my Azure vault can be accessed.

To do this, you use a federated identity credential, which delegates authentication to an external identity provider, in my case the AKS cluster, and thus creates a trust relationship between AKS and the managed identity in Microsoft Entra ID.

1
2
3
4
5
6
7
8


resource "azurerm_federated_identity_credential" "identity" {
 name = "kub-federated-identity"
 resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group
 audience = ["api://AzureADTokenExchange"]
 issuer = azurerm_kubernetes_cluster.aks.oidc_issuer_url
 parent_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].user_assigned_identity_id
 subject = "system:serviceaccount:default:workload-identity-sa"
}

Here the audience is the intended recipient of the token issued by the OIDC issuer, the subject is the identity of the entity (the service account in my case) the federated identity is associated with, and issuer is the OIDC issuer URL of the AKS cluster.

Sequence diagram

Here is a more detailed diagram of how the connection to the Vault works with managed identity and federation:

Sequence diagram of the connection to the Vault

External Secrets Operator

Next, the Kubernetes operator, External Secrets Operator (ESO), has to be set up to connect to the vault through the Azure Key Vault provider that ESO ships. For this, Custom Resource Definitions (CRDs) exist that define how to access the vault (Secret Store) and which data to retrieve (External Secret).

Secret Store

In this resource, you specify the vault URL, the Azure tenant ID, the authentication type and the associated service account. The secret store is what lets me connect to Azure Key Vault.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
 name: azure-secret-store
spec:
 provider:
 azurekv:
 authType: WorkloadIdentity
 vaultUrl: ${azurerm_key_vault.vault.vault_uri}
 tenantId: ${azurerm_key_vault.vault.tenant_id}
 serviceAccountRef:
 name: workload-identity-sa

External Secret

This resource defines which secret needs to be synced between the vault and the Kubernetes secret. You define the secret store to use, the target secret in Kubernetes and the secrets to retrieve from the Azure vault.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: my-secret
spec:
 refreshInterval: 30s
 secretStoreRef:
 name: azure-secret-store
 kind: SecretStore
 target:
 name: ocf-secret
 data:
 - secretKey: pgadmin-password
 remoteRef:
 key: pgadmin-password

Service Account

You also need to create the service account and add the following annotation to it, to associate it with the ID of the identity: Kubelet Identity.

1
2
3
4
5
6
7


apiVersion: v1
kind: ServiceAccount
metadata:
 annotations:
 azure.workload.identity/client-id: "${azurerm_kubernetes_cluster.aks.kubelet_identity[0].client_id}"
 name: workload-identity-sa
 namespace: default

Deployment

For the deployment, I also used Terraform, with the helm and kubectl providers. This gives a unified infrastructure, automated end to end.

Deployment with Terraform

ESO

For ESO, a Helm chart is provided in the documentation, and it is deployed in the external-secrets namespace. This Terraform resource waits for the AKS cluster deployment to finish before being created, thanks to depend_on = [azurerm_kubernetes_cluster.aks].

1
2
3
4
5
6
7
8
9


resource "helm_release" "eso" {
 depends_on = [azurerm_kubernetes_cluster.aks]
 name = "external-secrets"
 namespace = "external-secrets"
 repository = "https://charts.external-secrets.io"
 chart = "external-secrets"
 create_namespace = true
 version = "0.10.5"
}

ESO pods deployed

Once ESO is deployed, you need to create the CRDs that configure the connection to the vault and the secrets to retrieve, as well as the service account. All of this with Kubernetes manifests, which declare the resources in YAML.

CRDs and Service Account

For the manifests, the kubectl Terraform provider lets me use variables from Terraform resources directly in the manifests to make them dynamic, as you can see in this manifest. This avoids hardcoding sensitive information in the manifests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


resource "kubectl_manifest" "sa" {
 depends_on = [helm_release.eso]
 yaml_body = <<YAML
apiVersion: v1
kind: ServiceAccount
.....
YAML
}

resource "kubectl_manifest" "secretstore" {
 depends_on = [kubectl_manifest.sa]
 yaml_body = <<YAML
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
.....
YAML
}

resource "kubectl_manifest" "externalsecret" {
 depends_on = [kubectl_manifest.secretstore]
 yaml_body = <<YAML
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
.....
YAML
}

The 3 Terraform resources are declared, and they match the 3 Kubernetes manifests / resources that must be created in the cluster to configure and use ESO.

Result

The connection to Azure Key Vault is validated, thanks to the secret-store resource.

Connection to Azure Key Vault with the secret store

Once connected to the vault, the secrets are synced. You can see it on the external-secret resource, which is in the SecretSynced state. It retrieves the secrets from the vault and puts them into a Kubernetes secret.

Secret sync between the vault and Kubernetes

The secret has been created and is present in Kubernetes, as you can see. For this demonstration the passwords are all the same, which of course should not be done in a production environment.

Kubernetes secret

Conclusion

Setting up this solution in a short time was a real challenge for me, as I had never done it before. I was able to work on and deepen my skills on Azure, and more specifically on the roles and identities of this cloud provider.

The goal was reached, and the complexity of the solution motivated me to write this article, first to keep a record of this work as a write-up, but also to help the people who will come across it sooner or later.