[{"content":" Series \u0026ldquo;A Kubernetes cluster with a GPU for LLM inference\u0026rdquo;\nDeploying an immutable Kubernetes cluster with Talos on Proxmox Passing an NVIDIA GPU through to a Kubernetes cluster (this article) Running a Kubernetes cluster with GitOps and Flux (in progress) Self-hosted LLM inference on Kubernetes (in progress) Introduction The Talos cluster deployed in part 1 is up and running, but the talos-gpu-01 node does not see any graphics card yet. In this article, I explain how I made a GeForce RTX 3050 usable by Kubernetes pods, starting from the Proxmox host. The work breaks down into four steps: detach the card from the host, attach it to the VM with OpenTofu, load the NVIDIA modules in Talos, then expose the GPU to the Kubernetes scheduler.\nThe path of the GPU, from the Proxmox host to the Kubernetes pod Before getting to the commands, I explain what passthrough does, since that is where most of the constraints met later come from, such as disabling ballooning or checking the IOMMU groups.\nContext The end goal of this series is to run LLM inference locally in my homelab, which requires a GPU accessible from the cluster. To give a graphics card to a VM, there are two main approaches: share it between several VMs with vGPU, or dedicate it entirely to a single VM with PCI passthrough.\nIn my case, the choice was quick. NVIDIA only offers vGPU on its datacenter and professional lines, it is not available on a GeForce. That leaves passthrough, with a direct consequence: the card is dedicated to the talos-gpu-01 VM, the Proxmox host no longer has access to it and no other VM can share it.\nThe hardware used is a GeForce RTX 3050 that I had on hand.\nPrerequisites A CPU and motherboard supporting the IOMMU (VT-d on Intel, AMD-Vi on AMD), enabled in the BIOS. The Talos cluster from part 1. A GPU that is not used for the host display. In my case, the Proxmox host runs headless and the CPU\u0026rsquo;s integrated GPU is disabled in the BIOS, so the RTX 3050 is the only graphics card in the machine. Understanding passthrough A PCIe device such as a GPU does not go through the CPU to read or write memory. It uses DMA (Direct Memory Access) to access RAM directly, which lets it transfer large volumes of data without keeping the processor busy.\nIn a VM, this causes a problem. The memory addresses seen by the guest system are guest physical addresses (GPA, Guest Physical Address), which do not match the host\u0026rsquo;s real physical addresses (HPA, Host Physical Address). For CPU accesses, the GPA to HPA translation is done in hardware by EPT (Extended Page Tables) on Intel or NPT (Nested Page Tables) on AMD, a second page table that the processor walks in addition to the guest\u0026rsquo;s. A device doing DMA does not go through this translation: if the VM\u0026rsquo;s driver gives a GPA to the GPU, the card writes at that address in the host\u0026rsquo;s RAM, which corresponds to something else entirely.\nThat is the job of the IOMMU (Input-Output Memory Management Unit). It does for devices what the MMU (Memory Management Unit), the processor component that translates virtual addresses into physical addresses, does for the CPU: it translates the addresses used by the device into real physical addresses, and it restricts the device to the memory areas assigned to it. The GPU passed to the VM can therefore only write into that VM\u0026rsquo;s memory.\nThis isolation is not done per device but per IOMMU group. A group gathers the devices that the hardware cannot isolate from one another, for example because they can communicate with each other without going through the IOMMU. The unit that can be passed to a VM is therefore the whole group, and you need to check the content of the card\u0026rsquo;s group before starting.\nFinally, on the host side, the card must not be claimed by any graphics driver (nouveau or nvidia). The vfio-pci driver takes ownership of it. It is part of VFIO (Virtual Function I/O), the Linux kernel framework that gives a program outside the kernel, here QEMU, direct access to a device, relying on the IOMMU to limit what the device can reach in memory. That is how QEMU presents the card to the VM.\nAddress translation: CPU accesses and GPU DMA accesses Why ballooning must be disabled Ballooning lets Proxmox reclaim RAM that a VM is not using, on the fly, thanks to a driver in the guest that hands memory pages back to the host. With a device passed through VFIO, this mechanism can no longer work: the GPU can write into the VM\u0026rsquo;s pages at any time without going through the CPU, so the host has no way of knowing whether a page is still in use by the card. All of the VM\u0026rsquo;s memory is therefore pinned at boot and can no longer be given back to the host.\nIn the bpg/proxmox provider, ballooning is controlled by the floating argument of the memory block, and the value 0 disables it. I set it to 0 in the resource that creates the Talos nodes, so on all three nodes and not only on the GPU node:\n1 2 3 4 5 #talos.tf memory { dedicated = each.value.memory floating = 0 } This affects sizing: the 12 GB of RAM of the talos-gpu-01 node are permanently reserved on the host, even when the node is idle.\nNow that the mechanism is clear, let\u0026rsquo;s prepare the Proxmox host.\nStep 1: preparing the Proxmox host Checking the IOMMU and the groups The first thing to do is identify the card and the IOMMU group it belongs to:\n1 2 3 4 5 # The card and its functions lspci -nn | grep -i nvidia # The IOMMU group it belongs to find /sys/kernel/iommu_groups -type l | grep \u0026#39;01:00\u0026#39; Which gives in my case:\n1 2 3 4 5 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1) /sys/kernel/iommu_groups/10/devices/0000:01:00.0 /sys/kernel/iommu_groups/10/devices/0000:01:00.1 The card exposes two functions: 01:00.0 for the video part and 01:00.1 for the HDMI audio controller. Both are alone in IOMMU group 10, which means the group can be passed to the VM as is, without taking any other host device along.\nIf other devices shared this group, for example a USB controller, they would have to be passed to the VM too. The ACS override patch is often suggested on forums to split groups artificially, but it asks the kernel to ignore an isolation that the hardware does not guarantee, since the devices in the group can still communicate with each other. The Proxmox wiki actually presents it as a last resort, and not a risk-free one.\nBinding the card to vfio-pci I configured the Proxmox host so that vfio-pci claims the card before any other driver. This part is not managed by my OpenTofu code, I did it directly on the Proxmox host.\nAccording to the Proxmox documentation, the IOMMU is enabled by default on Intel from kernel 6.8, so the intel_iommu=on option found in many guides is no longer needed. The only useful parameter is iommu=pt, which limits address translation to the devices passed to a VM and can improve performance for the others.\nKernel parameters are passed through the command line that GRUB, the host\u0026rsquo;s bootloader, hands to the kernel at every boot. When the host boots with GRUB, which is my case, it is set in /etc/default/grub:\n1 2 #/etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=\u0026#34;quiet iommu=pt\u0026#34; GRUB_CMDLINE_LINUX_DEFAULT applies to normal boot entries, but not to recovery mode. quiet is present by default on Debian and Proxmox: it reduces the kernel messages displayed on screen during boot, without changing what is recorded in the logs.\nThe host\u0026rsquo;s dmesg confirms that the IOMMU is active, with the requested passthrough mode, and that the devices are split into IOMMU groups:\n1 2 3 [ 0.284240] iommu: Default domain type: Passthrough (set via kernel command line) [ 0.329656] pci 0000:00:00.0: Adding to iommu group 0 [ 0.329665] pci 0000:00:01.0: Adding to iommu group 1 The VFIO modules are loaded at boot through /etc/modules: vfio for the core of the framework, vfio_iommu_type1 to drive the IOMMU, and vfio_pci for PCI devices:\n1 2 3 4 #/etc/modules vfio vfio_iommu_type1 vfio_pci For vfio-pci to reserve the card, I give it the IDs of both functions, the video part 10de:2507 and the audio part 10de:228e:\n1 2 #/etc/modprobe.d/vfio.conf options vfio-pci ids=10de:2507,10de:228e disable_vga=1 The disable_vga=1 option prevents vfio-pci from exposing the card\u0026rsquo;s legacy VGA resources to the host. On top of that, I block the NVIDIA graphics drivers on the host so that none of them claims the card at boot:\n1 2 3 #/etc/modprobe.d/blacklist.conf blacklist nouveau blacklist nvidia* Changes to /etc/modules and /etc/modprobe.d/ only take effect after regenerating the initramfs (update-initramfs -u -k all), and changes to /etc/default/grub after an update-grub. After rebooting the host, the check is done with:\n1 lspci -nnk -s 01:00 Which gives in my case:\n1 2 3 4 5 6 7 8 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] (rev a1) Subsystem: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau 01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1) Subsystem: NVIDIA Corporation Device [10de:2507] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel Both functions are claimed by vfio-pci. The Kernel modules line lists the drivers able to handle the card, not those that are loaded: nouveau and nvidiafb are blocked and do not show up in lsmod.\n⚠️ Warning As long as lspci -nnk does not show Kernel driver in use: vfio-pci on both functions, there is no point going further. The VM may boot, but the card will not be usable.\nThe host is ready, the card can now be attached to the VM.\nStep 2: attaching the GPU to the VM with OpenTofu Why a hardware mapping and not a PCI ID The bpg/proxmox provider offers two ways to declare a device in a VM\u0026rsquo;s hostpci block: its raw PCI ID with the id argument, or a hardware mapping declared at the Proxmox cluster level with the mapping argument.\nThe provider documentation states that id is not compatible with API token authentication and requires the root account with its password. All my IaC authenticates with the token of the terraform@pve user, so I went with the mapping, which was the only option without changing the provider\u0026rsquo;s authentication.\nThe mapping is declared with the proxmox_hardware_mapping_pci resource:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 #hardware_mapping.tf resource \u0026#34;proxmox_hardware_mapping_pci\u0026#34; \u0026#34;this\u0026#34; { ... for_each = var.pci_hardware_mappings name = each.key comment = each.value.comment map = [ { node = coalesce(each.value.node, var.proxmox_node) path = each.value.path id = each.value.id iommu_group = each.value.iommu_group subsystem_id = each.value.subsystem_id } ] } The values are set in the variables:\n1 2 3 4 5 6 7 8 9 10 #variables.auto.tfvars pci_hardware_mappings = { gpu = { comment = \u0026#34;GeForce RTX 3050 - managed by Terraform\u0026#34; path = \u0026#34;0000:01:00\u0026#34; id = \u0026#34;10de:2507\u0026#34; iommu_group = 10 subsystem_id = \u0026#34;10de:2507\u0026#34; } } The path is declared without a function number (0000:01:00 rather than 0000:01:00.0), which covers both functions of the card at once.\nThe iommu_group and subsystem_id arguments are optional for the API, but without them Proxmox shows the mapping as incomplete in the UI. So I filled them in from the output of lspci -nnv -s 0000:01:00 and find /sys/kernel/iommu_groups -type l.\nProxmox privileges The role used by OpenTofu needs three extra privileges:\n1 2 3 4 #users.tf \u0026#34;Mapping.Audit\u0026#34;, \u0026#34;Mapping.Modify\u0026#34;, \u0026#34;Mapping.Use\u0026#34;, Mapping.Audit: read existing mappings. Mapping.Modify: create, modify and delete a mapping. Mapping.Use: attach a mapping to a VM, on top of the usual VM modification rights. Without Mapping.Use, the mapping is created but attaching it to the VM fails.\nThe hostpci block On the VM side, a single resource serves all three Talos nodes. The hostpci block is therefore dynamic: it is generated from each node\u0026rsquo;s hostpci list, which defaults to []. The two nodes without a card thus have no device attached.\n1 2 3 4 5 6 7 8 9 10 11 12 #talos.tf dynamic \u0026#34;hostpci\u0026#34; { for_each = { for device in each.value.hostpci : device.device =\u0026gt; device } content { device = hostpci.value.device mapping = proxmox_hardware_mapping_pci.this[hostpci.value.mapping].name pcie = hostpci.value.pcie rombar = hostpci.value.rombar xvga = hostpci.value.xvga } } In part 1, I mentioned a sixth validation on the talos_nodes variable. Here it is, it checks that each hostpci.mapping matches a declared mapping:\n1 2 3 4 5 6 7 8 9 #variables.tf validation { condition = alltrue(flatten([ for node in var.talos_nodes : [ for device in node.hostpci : contains(keys(var.pci_hardware_mappings), device.mapping) ] ])) error_message = \u0026#34;Every Talos node hostpci.mapping must be a key of var.pci_hardware_mappings.\u0026#34; } A typo in the mapping name is thus caught at plan time, and not as an API error in the middle of the apply.\nOnce the pipeline has run, the VM configuration on the Proxmox side reflects what was declared:\n1 qm config 123 | grep -E \u0026#39;^(hostpci|balloon|memory)\u0026#39; 1 2 3 balloon: 0 hostpci0: mapping=gpu,pcie=1,rombar=1,x-vga=0 memory: 12288 The device is referenced by the mapping name and not by its PCI address, and balloon: 0 matches the floating = 0 of the memory block.\nThe card is now attached to the VM, it remains to make it usable by Talos.\nStep 3: the NVIDIA modules in Talos A schematic dedicated to the GPU node As explained in part 1, the NVIDIA extensions are not in the base image: the nodes without a card reported module loading errors at every boot. The talos-gpu-01 node therefore uses its own schematic:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #variables.auto.tfvars { hostname = \u0026#34;talos-gpu-01\u0026#34; ip = \u0026#34;192.168.1.123\u0026#34; cores = 4 memory = 12 * 1024, disk_size = 100, role = \u0026#34;worker\u0026#34; schematic_id = \u0026#34;a31d81eceab05afdc2c59b0002701173e117a5fb81bdc1f64062d27655145848\u0026#34; # With NVIDIA GPU support hostpci = [ { device = \u0026#34;hostpci0\u0026#34; mapping = \u0026#34;gpu\u0026#34; } ] } This schematic adds two extensions to qemu-guest-agent: nvidia-open-gpu-kernel-modules-production for the drivers, and nvidia-container-toolkit-production so that the container runtime knows how to expose the card to pods.\n📝 Note I use NVIDIA\u0026rsquo;s open source drivers. They support GPUs from the Turing architecture onwards, which includes the RTX 3050 (Ampere). On older hardware, you need the proprietary extensions.\nLoading the modules The extensions ship the kernel modules but do not load them. That is the job of a machine config patch, placed at the node level and not at the worker/ level:\n1 2 3 4 5 6 7 8 # patches/nodes/talos-gpu-01/30-nvidia-modules.yaml.tftpl machine: kernel: modules: - name: nvidia - name: nvidia_uvm - name: nvidia_drm - name: nvidia_modeset This patch uses the nodes/ level presented in part 1: placed under worker/, it would also be applied to talos-worker-01, whose image does not contain the extensions.\nThe list of modules comes from the Talos documentation. The nvidia_uvm module is the one CUDA uses to manage unified memory.\nApplying and checking The schematic and the patch are deployed by the GitLab CI pipeline described in part 1, like the rest of the node configuration. I then check that the extensions are present in the node\u0026rsquo;s image:\n1 talosctl -n 192.168.1.123 get extensions Which gives the following result:\n1 2 3 4 5 6 NODE NAMESPACE TYPE ID VERSION NAME VERSION 192.168.1.123 runtime ExtensionStatus 0 1 nvidia-container-toolkit-production 595.71.05-v1.19.1 192.168.1.123 runtime ExtensionStatus 1 1 nvidia-open-gpu-kernel-modules-production 595.71.05-v1.13.8 192.168.1.123 runtime ExtensionStatus 2 1 qemu-guest-agent 11.0.2 192.168.1.123 runtime ExtensionStatus 3 1 schematic a31d81eceab05afdc2c59b0002701173e117a5fb81bdc1f64062d27655145848 192.168.1.123 runtime ExtensionStatus modules.dep 1 modules.dep 6.18.42-talos Both NVIDIA extensions are at version 595.71.05, the driver number, and the schematic ID is the one declared for this node. The four modules from the patch must then be loaded:\n1 talosctl -n 192.168.1.123 get loadedkernelmodules | grep -i nvidia 1 2 3 4 192.168.1.123 runtime LoadedKernelModule nvidia 2 Live 192.168.1.123 runtime LoadedKernelModule nvidia_drm 2 Live 192.168.1.123 runtime LoadedKernelModule nvidia_modeset 2 Live 192.168.1.123 runtime LoadedKernelModule nvidia_uvm 1 Live Finally, the /proc/driver/nvidia/version file only exists if the nvidia module is loaded, and it gives the exact driver version:\n1 talosctl -n 192.168.1.123 read /proc/driver/nvidia/version 1 2 NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 595.71.05 Release Build (dvs-builder@U22-I3-G08-03-1) Fri Apr 24 06:42:30 UTC 2026 GCC version: Selected multilib: .;@m64 The Open Kernel Module mention confirms that the open source drivers are the ones running.\nThe driver is loaded in the VM, the last step is to make it known to Kubernetes.\nStep 4: exposing the GPU to Kubernetes Modules loaded on the node are not enough. The Kubernetes scheduler only knows about CPU and memory, and for it to allocate a GPU to a pod, an nvidia.com/gpu extended resource has to be published on the node by a device plugin. A device plugin is a component running on the node that declares the available GPUs and their state to the kubelet, then chooses which card to assign to each pod that requests one.\nNVIDIA GPU Operator To do this, I used the NVIDIA GPU Operator rather than the NVIDIA device plugin alone, for two reasons: it is the method described in the Talos documentation, and the operator also deploys the DCGM exporter, which lets me collect the GPU metrics in Prometheus.\nBy default, the operator installs the drivers and the container toolkit on the nodes itself, but on Talos these two components are already provided by the system extensions. They must therefore be disabled to avoid conflicts, as the Talos documentation states:\n1 2 3 4 5 6 7 8 9 10 11 12 kubectl create namespace gpu-operator kubectl label --overwrite namespace gpu-operator pod-security.kubernetes.io/enforce=privileged helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update helm install gpu-operator nvidia/gpu-operator \\ --namespace gpu-operator \\ --version v26.3.3 \\ --set driver.enabled=false \\ --set toolkit.enabled=false \\ --set hostPaths.driverInstallDir=/usr/local \\ --set dcgmExporter.serviceMonitor.enabled=true driver.enabled=false and toolkit.enabled=false: the drivers and the toolkit come from the Talos extensions. hostPaths.driverInstallDir=/usr/local: the Talos extension installs the driver files in /usr/local, while the operator looks for them in /run/nvidia/driver by default. dcgmExporter.serviceMonitor.enabled=true: creates the ServiceMonitor that lets Prometheus collect the GPU metrics. The namespace must allow privileged pods, because the operator\u0026rsquo;s components access the node\u0026rsquo;s hardware. 📝 Note The Talos documentation requires a GPU Operator version greater than or equal to v26.3.1.\nI did not need a nodeSelector to target the GPU node. The operator first deploys Node Feature Discovery on all nodes, which detects the PCI devices of each one. The operator then adds the nvidia.com/gpu.present=true label to the nodes that have an NVIDIA card, and only deploys its components (device plugin, DCGM exporter, validation) on those nodes:\n1 kubectl get pods -n gpu-operator -o wide 1 2 3 4 5 6 7 8 9 10 11 12 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES gpu-feature-discovery-9ll6h 1/1 Running 0 52d 10.244.0.134 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; gpu-operator-57d75775c8-ls7hf 1/1 Running 0 4d12h 10.244.2.150 talos-controlplane-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; gpu-operator-node-feature-discovery-gc-847bb8f7b6-9gmkf 1/1 Running 0 4d12h 10.244.0.231 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; gpu-operator-node-feature-discovery-master-d98f944cd-vhzdw 1/1 Running 0 4d12h 10.244.2.188 talos-controlplane-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; gpu-operator-node-feature-discovery-worker-kvvfl 1/1 Running 0 52d 10.244.0.249 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; gpu-operator-node-feature-discovery-worker-ntvzt 1/1 Running 0 52d 10.244.2.39 talos-controlplane-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; gpu-operator-node-feature-discovery-worker-s95fk 1/1 Running 0 52d 10.244.1.208 talos-worker-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; nvidia-cuda-validator-gfd6h 0/1 Completed 0 6m42s 10.244.0.109 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; nvidia-dcgm-exporter-rkt88 1/1 Running 0 52d 10.244.0.161 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; nvidia-device-plugin-daemonset-pgnqw 1/1 Running 0 52d 10.244.0.164 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; nvidia-operator-validator-xqdd8 1/1 Running 0 52d 10.244.0.50 talos-gpu-01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; Only the NFD workers run on all three nodes, all the NVIDIA components are on talos-gpu-01. The nvidia-cuda-validator pod in Completed is expected: it runs a test CUDA program on the card, then stops.\nChecking the allocation The nvidia.com/gpu resource must appear in the node\u0026rsquo;s capacity:\n1 kubectl get nodes -o json | jq \u0026#39;.items[] | {name: .metadata.name, gpu: .status.capacity.\u0026#34;nvidia.com/gpu\u0026#34;}\u0026#39; 1 2 3 4 5 6 7 8 9 10 11 12 { \u0026#34;name\u0026#34;: \u0026#34;talos-controlplane-01\u0026#34;, \u0026#34;gpu\u0026#34;: null } { \u0026#34;name\u0026#34;: \u0026#34;talos-gpu-01\u0026#34;, \u0026#34;gpu\u0026#34;: \u0026#34;1\u0026#34; } { \u0026#34;name\u0026#34;: \u0026#34;talos-worker-01\u0026#34;, \u0026#34;gpu\u0026#34;: null } The talos-gpu-01 node exposes one GPU, the other two nodes do not have the resource. The labels added to the node by GPU Feature Discovery also give the details of the card as seen by Kubernetes:\n1 kubectl get node talos-gpu-01 --show-labels | tr \u0026#39;,\u0026#39; \u0026#39;\\n\u0026#39; | grep nvidia.com 1 2 3 4 5 6 7 8 9 10 11 12 13 ... nvidia.com/cuda.driver-version.full=595.71.05 ... nvidia.com/cuda.runtime-version.full=13.2 ... nvidia.com/gpu.count=1 ... nvidia.com/gpu.family=ampere ... nvidia.com/gpu.memory=8192 ... nvidia.com/gpu.product=NVIDIA-GeForce-RTX-3050 ... The nvidia.com/cuda.runtime-version.full=13.2 label gives the highest CUDA version supported by the driver, which is used to choose the image of the test pod.\nTo validate the whole chain, I run a test pod that executes nvidia-smi (NVIDIA System Management Interface), the command line tool shipped with the driver that shows the visible cards, the driver version, memory usage and running processes. If the command sees the card from inside a container, the driver, the runtime and the allocation all work:\n1 2 3 4 5 6 7 8 9 10 11 12 13 apiVersion: v1 kind: Pod metadata: name: nvidia-smi spec: restartPolicy: Never containers: - name: nvidia-smi image: nvidia/cuda:13.2.1-base-ubuntu24.04 command: [\u0026#34;nvidia-smi\u0026#34;] resources: limits: nvidia.com/gpu: 1 The nvidia.com/gpu: 1 field in the limits is enough: the pod requests the resource published by the device plugin, so the scheduler can only place it on talos-gpu-01, and the GPU is reserved for it as long as it runs.\nThe Talos documentation also adds runtimeClassName: nvidia to its test pod, to go through the NVIDIA runtime that the nvidia-container-toolkit extension declares in containerd. On my cluster, this is not needed, because CDI (Container Device Interface) is enabled in the GPU Operator:\n1 kubectl get clusterpolicy -o jsonpath=\u0026#39;{.items[0].spec.cdi}\u0026#39; 1 {\u0026#34;default\u0026#34;:false,\u0026#34;enabled\u0026#34;:true,\u0026#34;nriPluginEnabled\u0026#34;:false} With CDI, the device plugin tells containerd which device files and which driver libraries to add to the container, and containerd injects them itself with its default runtime. This is also how the nvidia-smi binary, absent from the nvidia/cuda image, ends up in the container. According to the NVIDIA documentation on CDI, this is the default behaviour since version v25.10.0 of the GPU Operator, and runtimeClassName: nvidia is only still needed for containers that access GPUs through the NVIDIA_VISIBLE_DEVICES environment variable without going through the device plugin, such as monitoring agents.\nOnce the pod has finished, its logs contain the output of nvidia-smi:\n1 kubectl logs nvidia-smi 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3050 On | 00000000:01:00.0 Off | N/A | | 0% 45C P8 7W / 130W | 1MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ The container sees the RTX 3050 with its 8 GB of memory, the 595.71.05 driver provided by the Talos extension and CUDA 13.2. The whole chain works, from vfio-pci on the Proxmox host all the way to the pod.\nConclusion The GPU can now be allocated by the scheduler: the cluster exposes an nvidia.com/gpu resource on the talos-gpu-01 node, and a pod can request it just like it would request CPU or memory.\nThere is only one nvidia.com/gpu resource in the whole cluster, which means only one pod at a time can use the card. To share it between several pods, the GPU Operator offers time-slicing, where pods take turns on the card, and NVIDIA also offers MPS (Multi-Process Service), which runs the CUDA processes of several pods on the card at the same time. These are options I have not set up yet.\nFor my part, this stage taught me a lot about hardware. My knowledge of virtualisation had so far stayed at a fairly high level, that of the hypervisor, and I did not know the low-level mechanisms that passthrough relies on, such as device I/O, DMA, the MMU or the IOMMU.\nAt this point, the infrastructure is fully described in OpenTofu, but everything running in the cluster was installed with helm install commands run by hand. It is the same configuration drift problem that Talos solves at the OS level, showing up one floor higher. Part 3 therefore sets up GitOps with Flux before going further towards inference.\nUseful links PCI Passthrough - Proxmox wiki PCI(e) Passthrough - Proxmox documentation NVIDIA GPU on Talos (OSS drivers) NVIDIA GPU Operator proxmox_hardware_mapping_pci ","date":"2026-10-09T00:00:00Z","image":"/posts/gpu-passthrough-talos/images/cover.png","permalink":"/en/posts/gpu-passthrough-talos/","title":"Passing an NVIDIA GPU through to a Kubernetes cluster"},{"content":" Series \u0026ldquo;A Kubernetes cluster with a GPU for LLM inference\u0026rdquo;\nDeploying an immutable Kubernetes cluster with Talos on Proxmox (this article) Passing an NVIDIA GPU through to a Kubernetes cluster Running a Kubernetes cluster with GitOps and Flux (in progress) Self-hosted LLM inference on Kubernetes (in progress) Introduction After deploying a Kubernetes cluster on Proxmox with Kubespray, I wanted to fix something that bothered me: maintaining the OS underneath the cluster. In this article, I start over from scratch with Talos Linux, an immutable distribution dedicated to Kubernetes, which I provision entirely with OpenTofu.\nThis article is the first of a series whose end goal is to run LLM inference locally, on a GPU passed through to the cluster. The aim is to train myself in MLOps and in deploying AI workloads on Kubernetes. The cluster described here is the foundation, so it already includes a node intended for the GPU.\nContext My previous article covered deploying a Kubernetes cluster with Kubespray. The result was reproducible, but the automation stopped at the OS: the three Debian VMs were still regular systems, with their package updates, their SSH access and a configuration that could drift between two runs of the playbook.\nWith Talos, the OS can no longer be administered directly: there is no shell, no SSH and no package manager. The only interface is a gRPC API, and the whole machine is described by a YAML configuration file applied declaratively. To change a node, I change its configuration and Talos applies it.\nTalos is not better than Kubespray in absolute terms. I lose the ability to step in by hand on a node, but in exchange I know that no node has been changed outside of the code.\nComponents Component Version Role Proxmox 9.1 Hypervisor Talos Linux 1.13.8 Node OS Kubernetes v1.36.2 Orchestrator OpenTofu 1.11.5 VM creation and Talos configuration bpg/proxmox provider 0.114.0 Drives Proxmox siderolabs/talos provider 0.12 Drives Talos OpenTofu GitLab CI component 4.5.0 Deployment pipeline Cilium 1.20.2 CNI, replaces kube-proxy I chose OpenTofu over Terraform to stay consistent with my stance on open source software and licences. Since HashiCorp moved Terraform to the BUSL licence in 2023, OpenTofu is the fork that remains under the MPL 2.0 licence. Its development is also driven more by feature requests from the community, while Terraform\u0026rsquo;s is geared towards HashiCorp\u0026rsquo;s Enterprise and Cloud offerings, which I have no interest in.\nTalos Linux Talos is a Linux distribution that only contains what is needed to run Kubernetes: a kernel, its own init system, containerd and the cluster components. A few of its properties explain most of the choices made in the rest of the article:\nThe root filesystem is read-only, mounted from a squashfs image. Data that must persist (etcd, container images) is written under /var, on a dedicated partition. So there is nowhere to drop a binary or edit a configuration file by hand. Updates follow an A/B scheme. The new version is installed next to the running one, and if the update fails, the node can reboot into the old one. There is no systemd. The kubelet, etcd, containerd and the Talos API are managed by machined, and their configuration comes entirely from the machine config. You cannot add a service or override one. Administration goes exclusively through a gRPC API, exposed on port 50000 of each node with mutual TLS authentication. The talosconfig file used by talosctl contains the cluster CA and a client certificate, and without this file a Talos node refuses any connection. The entire configuration of a node therefore fits in a single YAML file, the machine config, which has two main sections: machine.* for the node itself (network, disks, kubelet, kernel modules) and cluster.* for the Kubernetes cluster (CNI, kube-proxy, extra manifests).\nMachine configuration and cluster configuration These two sections live in the same file, but they are not applied the same way:\nmachine.* is reconciled continuously. Talos controllers watch the configuration and adjust the state of the node. A patch that changes machine.network or machine.kubelet is applied immediately, sometimes after restarting the service involved. cluster.* is only read at bootstrap. During talos_machine_bootstrap, Talos generates Kubernetes manifests from this section and applies them to the cluster. The section then stays in the configuration, but it is never read again. In practice, changing cluster.network.cni.name on a cluster that is already running does not remove the CNI in place, and Talos shows no error and no warning. The manifests generated from cluster.* are only replayed, removing those that are no longer generated, during a talosctl upgrade-k8s.\nThe two lifecycles of the machine config Target architecture Role Hostname VM ID IP vCPU RAM Disk Control plane talos-controlplane-01 121 192.168.1.121 2 4 GB 50 GB Worker talos-worker-01 122 192.168.1.122 4 8 GB 100 GB GPU worker talos-gpu-01 123 192.168.1.123 4 12 GB 100 GB The Kubernetes API does not answer on a node IP but on a VIP, 192.168.1.120. This address is reserved, which is why the nodes start at .121.\n📝 Note There is only one control plane, as in my Kubespray article, and for the same reason: it is a homelab. I still configured the VIP from the start, because it is written into the configuration of every node, and setting it up later would mean reconfiguring the whole cluster.\nA GPU node declared from the start The third node is called talos-gpu-01, even though no graphics card is used in this article. It already differs from the other two in three ways:\n12 GB of RAM instead of 8. PCI passthrough requires ballooning to be disabled, so this memory will be permanently reserved on the host. I preferred to size the node accordingly right away. A different Talos image. The NVIDIA modules are provided by system extensions, which are built into the image when it is created. This node therefore has its own image. A configuration patch that only applies to it. Loading the NVIDIA modules on a node without a card produces an error at every boot. This last point affects how the configuration is organised: as soon as one node differs from the others, a per-role configuration is no longer enough. I describe the solution in the section on patches.\nProvisioning with OpenTofu The images: Talos Image Factory A Talos image cannot be modified afterwards. To add qemu-guest-agent or the NVIDIA modules, you need an image that already contains them, and that is the job of the Image Factory: you send it a list of extensions, and it returns a schematic ID, a hash that identifies this list. This hash is then used to build the image URL for any Talos version.\nSchematic Extensions Nodes ce4c9805... qemu-guest-agent controlplane-01, worker-01 a31d81ec... qemu-guest-agent, nvidia-open-gpu-kernel-modules-production, nvidia-container-toolkit-production gpu-01 The downloaded image is a compressed raw disk, nocloud-amd64.raw.xz. It changes two things compared to my previous deployments:\nnocloud is the target platform. With it, Talos reads its initial network configuration from the cloud-init drive provided by Proxmox, which makes the node reachable from the first boot, before it has received its machine config. There is no template anymore. My Debian VMs are cloned from a template built with Packer, whereas the Talos nodes import this image directly as their scsi0 disk. So I no longer have a template to rebuild for this cluster: to change the Talos version, I change a variable. The download uses a for_each over the schematics actually in use, which avoids downloading the same image twice when several nodes share the same schematic:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 #image.tf locals { talos_schematics = toset([ for node in var.talos_nodes : coalesce(node.schematic_id, var.talos_image_factory_id) ]) } resource \u0026#34;proxmox_download_file\u0026#34; \u0026#34;talos_image\u0026#34; { for_each = local.talos_schematics content_type = \u0026#34;iso\u0026#34; datastore_id = \u0026#34;local\u0026#34; file_name = \u0026#34;talos-${var.talos_version}-${substr(each.key, 0, 8)}-nocloud-amd64.img\u0026#34; node_name = var.proxmox_node url = \u0026#34;https://factory.talos.dev/image/${each.key}/v${var.talos_version}/nocloud-amd64.raw.xz\u0026#34; overwrite = false decompression_algorithm = \u0026#34;zst\u0026#34; } The image is downloaded as .raw.xz, while decompression_algorithm is set to zst. The provider only accepts gz, lzo, zst and bz2, but Proxmox\u0026rsquo;s zst decompression can also handle an xz archive. This behaviour is not documented, I found it in a bpg provider discussion about xz-compressed images.\nThe topology in a variable The whole cluster topology is described in a single variable. To add a node, you just add an object to this list:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 #variables.tf variable \u0026#34;talos_nodes\u0026#34; { type = list(object({ hostname = string ip = string cores = number memory = number vm_id = optional(number) disk_size = optional(number, 40) role = optional(string, \u0026#34;controlplane\u0026#34;) # Schematic Image Factory. `null` retombe sur var.talos_image_factory_id. schematic_id = optional(string) # Périphériques PCI passés au nœud (partie 2). hostpci = optional(list(object({ device = string mapping = string pcie = optional(bool, true) rombar = optional(bool, true) xvga = optional(bool, false) })), []) })) validation { condition = !contains([for node in var.talos_nodes : node.ip], var.talos_vip) error_message = \u0026#34;The Talos VIP must not collide with a node IP address.\u0026#34; } ... } I added more validations than usual on this variable, five in total:\nrole must be controlplane or worker hostnames must be unique IPs must be unique there must be at least one controlplane the VIP must not match a node IP Without these validations, each of these mistakes would show up in the middle of an apply, as an unclear Proxmox API error, with part of the infrastructure already created. With them, the error is caught at plan time, before anything is changed. A sixth validation comes in part 2, to check that each hostpci.mapping matches a declared mapping.\nThe VMs A single resource creates the three nodes:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 #talos.tf resource \u0026#34;proxmox_virtual_environment_vm\u0026#34; \u0026#34;talos\u0026#34; { depends_on = [ proxmox_download_file.talos_image, proxmox_virtual_environment_pool.TALOS, ] for_each = { for node in var.talos_nodes : node.hostname =\u0026gt; node } name = each.key node_name = var.proxmox_node vm_id = tonumber(split(\u0026#34;.\u0026#34;, each.value.ip)[3]) machine = \u0026#34;q35\u0026#34; ... bios = \u0026#34;ovmf\u0026#34; efi_disk { datastore_id = var.proxmox_datastore type = \u0026#34;4m\u0026#34; } memory { dedicated = each.value.memory floating = 0 } disk { datastore_id = var.proxmox_datastore interface = \u0026#34;scsi0\u0026#34; size = each.value.disk_size file_id = proxmox_download_file.talos_image[coalesce(each.value.schematic_id, var.talos_image_factory_id)].id discard = \u0026#34;on\u0026#34; ssd = true iothread = true ... } initialization { datastore_id = var.proxmox_datastore ip_config { ipv4 { address = \u0026#34;${each.value.ip}/24\u0026#34; gateway = var.talos_gateway } } dns { servers = var.talos_nameservers } } ... } There is no clone block. The system disk is the image downloaded above, referenced by its file_id, and that is what replaces the template. machine = \u0026quot;q35\u0026quot; and bios = \u0026quot;ovmf\u0026quot; with an efi_disk. The VMs boot in UEFI, and the OVMF firmware needs an EFI disk to store its variables. The vm_id is derived from the last octet of the IP. 192.168.1.123 gives VM 123, which lets me map the Proxmox interface to the network directly. floating = 0 disables ballooning, which is mandatory for the GPU node in part 2. 📝 Note The cloud-init drive of a Talos node is not used to configure Talos. It only gives the node a static IP at first boot, so that it is reachable in maintenance mode. All the configuration is then applied through the Talos API, via the siderolabs/talos provider.\nThe VMs are ready to boot, now they need their configuration.\nThe machine configuration Generating the secrets and the base configuration The siderolabs/talos provider chains six resources, in this order:\n1 2 3 4 5 6 talos_machine_secrets # cluster CA, keys and tokens └─\u0026gt; data.talos_client_configuration # the talosconfig for talosctl └─\u0026gt; data.talos_machine_configuration # a base config, per role └─\u0026gt; talos_machine_configuration_apply # applied per node, with the patches └─\u0026gt; talos_machine_bootstrap # etcd init, only once └─\u0026gt; talos_cluster_kubeconfig # the admin kubeconfig talos_machine_secrets generates all the cryptographic material of the cluster: the Kubernetes, Talos and etcd CAs, and the node enrolment tokens. These secrets are never written to disk, they are stored in the OpenTofu state and only leave it through outputs.\nThe bootstrap and the kubeconfig retrieval have no for_each, because they are operations on the whole cluster. They target a single control plane, the first one in the list:\n1 2 3 4 5 #talos.tf locals { talos_controlplanes = [for node in var.talos_nodes : node if node.role == \u0026#34;controlplane\u0026#34;] talos_first_controlplane = local.talos_controlplanes[0].ip } The talos_machine_configuration_apply resource gets its node argument from the talos_nodes variable, not from a VM attribute. OpenTofu therefore sees no dependency between the two, and may try to apply the configuration to an IP that does not answer yet. So I added an explicit depends_on on the VMs:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 #talos.tf resource \u0026#34;talos_machine_configuration_apply\u0026#34; \u0026#34;this\u0026#34; { depends_on = [proxmox_virtual_environment_vm.talos] for_each = { for node in var.talos_nodes : node.hostname =\u0026gt; node } client_configuration = talos_machine_secrets.this.client_configuration machine_configuration_input = data.talos_machine_configuration.this[each.value.role].machine_configuration node = each.value.ip config_patches = [ for patch in local.talos_patch_files[each.key] : templatefile(patch, { hostname = each.key node_ip = each.value.ip role = each.value.role vip = var.talos_vip gateway = var.talos_gateway nameservers = var.talos_nameservers }) ] } Patches, on three levels A list of patches per role is no longer enough as soon as a single node has specific hardware. So I organised the patches on three levels, from the most general to the most specific:\n1 2 3 4 5 6 7 8 9 10 11 tofu/patches/ ├── common/ # all nodes │ └── 40-kubelet.yaml.tftpl ├── controlplane/ # control planes │ ├── 10-network-vip.yaml.tftpl │ ├── 20-cluster-cni-proxy.yaml.tftpl │ └── 30-extra-manifests.yaml.tftpl ├── worker/ # workers └── nodes/ └── talos-gpu-01/ # this specific node └── 30-nvidia-modules.yaml.tftpl The list of patches for each node is built with fileset:\n1 2 3 4 5 6 7 8 9 #talos.tf talos_patch_files = { for node in var.talos_nodes : node.hostname =\u0026gt; concat( [for f in fileset(\u0026#34;${path.module}/patches/common\u0026#34;, \u0026#34;*.yaml.tftpl\u0026#34;) : \u0026#34;${path.module}/patches/common/${f}\u0026#34;], [for f in fileset(\u0026#34;${path.module}/patches/${node.role}\u0026#34;, \u0026#34;*.yaml.tftpl\u0026#34;) : \u0026#34;${path.module}/patches/${node.role}/${f}\u0026#34;], [for f in fileset(\u0026#34;${path.module}/patches/nodes/${node.hostname}\u0026#34;, \u0026#34;*.yaml.tftpl\u0026#34;) : \u0026#34;${path.module}/patches/nodes/${node.hostname}/${f}\u0026#34;], ) } This relies on three properties of fileset:\nFiles are sorted alphabetically, so the numeric prefix of each file sets the merge order. For a given key, the last patch wins: the node level overrides the role level, which overrides the common level. I leave gaps in the numbering (10, 20, 30) so I can insert a patch later. A missing directory returns an empty set, without error. All three levels are therefore optional, which is handy since Git does not track empty directories. Adding a patch requires no code change. You just drop a file in the right directory, talos.tf stays the same. The GPU node is what led me to create the nodes/ level: an NVIDIA patch placed under worker/ would also be applied to talos-worker-01, whose image does not contain the extensions, and that node would fail to load the modules at every boot.\nThe VIP Talos manages the VIP itself, without keepalived or an external load balancer. The control planes choose the node that holds the VIP through an election in etcd: only one holds it at any given time, and it fails over automatically to another control plane if that node goes away.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # patches/controlplane/10-network-vip.yaml.tftpl machine: network: interfaces: - deviceSelector: physical: true dhcp: false addresses: - ${node_ip}/24 routes: - network: 0.0.0.0/0 gateway: ${gateway} vip: ip: ${vip} nameservers: ${jsonencode(nameservers)} The interface is declared again in full, not just completed with a vip entry. The machine config takes precedence over the network configuration provided by the nocloud platform: a patch that only contained the VIP would wipe out the static addressing provided by the cloud-init drive, and the node would become unreachable.\ncluster_endpoint also points to the VIP, not to a node IP. This value is written into the configuration of every node and into the kubeconfig: if it designated a specific node, losing that node would cut off access to the API, and adding a control plane would mean reconfiguring the whole cluster to change the endpoint.\nCNI and kube-proxy disabled 1 2 3 4 5 6 7 # patches/controlplane/20-cluster-cni-proxy.yaml.tftpl cluster: network: cni: name: none proxy: disabled: true By default, Talos installs Flannel as the CNI and kube-proxy. I replace both with Cilium, so I disable them.\nI chose Cilium for three reasons: it replaces kube-proxy, it handles L2 announcements, which I use later in the series to announce the IPs of LoadBalancer services on my local network, and it is all built on eBPF, a technology I want to rely on as my homelab evolves.\nThis choice has two consequences:\nthe nodes stay NotReady as long as no CNI is installed, which is expected without kube-proxy, the CNI has to reach the API server directly, because the ClusterIP of the kubernetes.default service is not routable yet when it starts. That is the job of KubePrism, which I present in the section on Cilium. This patch is placed on the control planes because these manifests are generated at bootstrap, from a control plane. Like everything under cluster.*, it only has an effect at that moment: applied to a cluster that is already running, it removes nothing.\nKubelet serving certificates By default, the kubelet exposes its :10250 endpoint with a self-signed certificate that no cluster CA recognises. Hence the --kubelet-insecure-tls option found in most metrics-server installations: the client does not verify the server certificate, so any machine able to get in the middle on the network can pretend to be a kubelet.\nThe first part of the solution is on the node side:\n1 2 3 4 5 # patches/common/40-kubelet.yaml.tftpl machine: kubelet: extraArgs: rotate-server-certificates: true With this option, the kubelet no longer generates a self-signed certificate and sends a CSR to the API server, with the kubernetes.io/kubelet-serving signer. The problem is that Kubernetes never automatically approves CSRs for this signer. The kube-controller-manager can sign the certificate, but no cluster component approves it: the CSRs stay Pending and the kubelet keeps its self-signed certificate. This is a deliberate choice of the Kubernetes project, which leaves that decision to each environment.\nThe second part is therefore on the cluster side:\n1 2 3 4 5 # patches/controlplane/30-extra-manifests.yaml.tftpl cluster: extraManifests: - https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/standalone-install.yaml - https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml kubelet-serving-cert-approver is a controller that approves these CSRs. It only approves them: the signing is still done by the kube-controller-manager, with the cluster CA key. Before approving a request, it checks that it really matches a kubelet\u0026rsquo;s:\nthe subject organisation is system:nodes the CN starts with system:node: and exactly matches the user who submitted the CSR the requested usages are those of a server certificate the request contains no email or URI SAN However, it does not check that the IP addresses requested in the SANs actually belong to the node, since its ClusterRole gives it no access to Node objects. A node already enrolled in the cluster can therefore obtain a valid certificate for another node\u0026rsquo;s IP.\nThe security gain is real but limited: before, any machine on the network could pretend to be a kubelet, whereas now you need the identity of a cluster node. In my case, metrics-server now verifies the kubelet certificates against the cluster CA, without --kubelet-insecure-tls.\nBoth manifests must stay deployed permanently, because certificate rotation is continuous and each renewal produces a new CSR to approve. The URLs are pinned to a specific version, so that a re-apply does not fetch different content and so that Renovate can track updates.\nThe configuration is complete, I can move on to the deployment.\nDeployment I do not run any apply locally. The deployment goes through a GitLab CI pipeline that uses the GitLab OpenTofu component:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 #.gitlab-ci.yml default: tags: - homelab variables: TF_VAR_proxmox_endpoint: \u0026#39;${proxmox_endpoint}\u0026#39; TF_VAR_proxmox_api_token: \u0026#39;${proxmox_api_token}\u0026#39; TF_VAR_proxmox_ssh_private_key: \u0026#39;${proxmox_ssh_private_key}\u0026#39; include: # renovate: datasource=gitlab-releases packageName=components/opentofu registryUrl=https://gitlab.com - component: $CI_SERVER_FQDN/components/opentofu/validate-plan-apply@4.5.0 inputs: # renovate: datasource=gitlab-releases packageName=components/opentofu registryUrl=https://gitlab.com version: 4.5.0 opentofu_version: 1.11.5 root_dir: tofu/ stages: [validate, build, deploy] workflow: rules: - changes: - tofu/** - when: never The validate-plan-apply template adds four jobs: fmt and validate in the validate stage, plan in build, then apply in deploy. By default, apply only exists on the default branch and must be started by hand: a merge request therefore stops at the plan, and nothing is applied until I trigger the job.\nThe rest of the configuration comes down to five points:\nThe jobs run on my own runner, thanks to the homelab tag: a GitLab runner with the Docker executor, hosted on my Proxmox. It is on the same network as the Proxmox API, which GitLab\u0026rsquo;s shared runners could not reach. The state is managed by GitLab. The backend \u0026quot;http\u0026quot; {} block in providers.tf is empty, and the component fills in the project\u0026rsquo;s state address and the job credentials at run time. The secrets stay in GitLab. The Proxmox API token and the SSH key are CI/CD variables, passed to OpenTofu through the TF_VAR_* variables. The pipeline only runs if tofu/ changes, thanks to the workflow rule. Versions are tracked by Renovate, thanks to the # renovate: comments above the component. Retrieving the credentials The cluster credentials are not written anywhere on disk, they are in the state, so they have to be extracted through the outputs. Since the state is stored in GitLab, I run a local tofu init once, only for this step. GitLab provides the full init command under Operate \u0026gt; Terraform states, with the Copy Terraform init command action. I then retrieve both files:\n1 2 tofu output -raw talosconfig \u0026gt; ../talos/config tofu output -raw kubeconfig \u0026gt; ../talos/kubeconfig The talosconfig generated by data.talos_client_configuration contains two lists: the endpoints, which talosctl actually connects to and which only contain control planes, and the nodes, which are the default targets of commands and are reached through an endpoint. This is what lets you query a worker without connecting to it directly.\nFor example, to list the services of talos-worker-01:\n1 talosctl --talosconfig talos/config -e 192.168.1.121 -n 192.168.1.122 services talosctl opens the connection to the control plane 192.168.1.121, which relays the request to the worker 192.168.1.122. Without -e, the endpoints from the talosconfig are used, and -n is enough to change the target.\nEndpoint and node with talosctl Installing Cilium The cluster starts without a CNI, so the nodes are NotReady. I install Cilium with Helm, as a replacement for kube-proxy:\n1 2 3 4 5 6 helm repo add cilium https://helm.cilium.io/ helm repo update helm install cilium cilium/cilium \\ --version 1.20.2 \\ --namespace kube-system \\ --values cilium-values.yaml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 #cilium-values.yaml kubeProxyReplacement: true k8sServiceHost: localhost k8sServicePort: 7445 # Talos monte lui-même le cgroup2 et interdit à l\u0026#39;agent de le remonter. cgroup: autoMount: enabled: false hostRoot: /sys/fs/cgroup securityContext: capabilities: ciliumAgent: - CHOWN - KILL - NET_ADMIN - NET_RAW - IPC_LOCK - SYS_ADMIN - SYS_RESOURCE - DAC_OVERRIDE - FOWNER - SETGID - SETUID cleanCiliumState: - NET_ADMIN - SYS_ADMIN - SYS_RESOURCE The cgroup and securityContext values are specific to Talos. By default, the chart mounts the cgroup2 itself and runs the agent in privileged mode, which Talos does not allow. I took the list of capabilities from the Talos documentation for Cilium rather than rebuilding it.\nThat leaves the k8sServiceHost / k8sServicePort pair. You could point Cilium at the VIP on 6443, since it is the cluster endpoint, but that is not what Talos recommends. Cilium has to reach the API server before the pod network exists, and the VIP depends on the etcd election between the control planes.\nKubePrism solves this problem: it is a local load balancer present on every node, listening on localhost:7445. It automatically picks among the available endpoints (the cluster endpoint and each control plane), discarding those that do not answer and favouring the fastest ones. It is enabled by default, and the Talos documentation recommends giving it as the endpoint to CNIs that need to reach the API server, such as Cilium.\nAccess to the Kubernetes API Verification Once Cilium is installed, I check the state of the cluster on the Talos side with talosctl health. I removed the progress lines from the output, to keep only the result of each check:\n1 talosctl --talosconfig talos/config -n 192.168.1.121 health 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 discovered nodes: [\u0026#34;192.168.1.120\u0026#34; \u0026#34;192.168.1.123\u0026#34; \u0026#34;192.168.1.122\u0026#34;] waiting for etcd to be healthy: OK waiting for etcd members to be consistent across nodes: OK waiting for etcd members to be control plane nodes: OK waiting for apid to be ready: OK waiting for all nodes memory sizes: OK waiting for all nodes disk sizes: OK waiting for no diagnostics: OK waiting for kubelet to be healthy: OK waiting for all nodes to finish boot sequence: OK waiting for all k8s nodes to report: OK waiting for all control plane static pods to be running: OK waiting for all control plane components to be ready: OK waiting for all k8s nodes to report ready: SKIP waiting for kube-proxy to report ready: SKIP waiting for coredns to report ready: SKIP waiting for all k8s nodes to report schedulable: OK The first address listed is the VIP and not 192.168.1.121: the control plane holding it advertises it among its addresses, and it is sorted first.\nThe three SKIPs come from the choices made in the patches. talosctl health does not check the Ready state of the nodes or CoreDNS when the CNI is set to none, since these checks depend on a CNI that Talos did not install itself, and it skips kube-proxy when its DaemonSet does not exist. The state of the nodes is therefore checked on the Kubernetes side:\n1 kubectl get nodes 1 2 3 4 NAME STATUS ROLES AGE VERSION talos-controlplane-01 Ready control-plane 47d v1.36.2 talos-gpu-01 Ready \u0026lt;none\u0026gt; 47d v1.36.2 talos-worker-01 Ready \u0026lt;none\u0026gt; 47d v1.36.2 All three nodes are Ready, which confirms that Cilium is working on each of them.\nPitfalls The Talos learning curve Getting to grips with Talos took me some time. Without a shell or SSH, none of my usual habits for troubleshooting a node apply, and everything that used to be done directly on the machine has to go through talosctl. So you have to learn a new CLI, new concepts and new naming.\nA first deployment without the Talos provider My first deployment did not use the siderolabs/talos provider: OpenTofu created the VMs, and I generated and applied the configurations by hand with talosctl. Switching to the provider brought the whole node configuration back as code, along with the rest of the infrastructure, and makes an end-to-end deployment possible in a single OpenTofu run.\nA single schematic for all nodes At first, all the nodes used the same schematic, the one containing the NVIDIA extensions. The nodes without a GPU then reported errors related to the NVIDIA drivers, since they shipped modules meant for a card they do not have. That is what led me to give each node a schematic containing only the extensions it needs.\nConclusion The cluster is up and running, and compared to Kubespray, I no longer have an OS to administer: no package updates, no open SSH, and the node configuration cannot drift from what the code describes. A Talos update comes down to changing a version in a variable, and each node switches to the new image.\nSummary of the deployment workflow Once everything was in place, what surprised me most was how easily and quickly a Kubernetes cluster comes up. Where Kubespray ran close to 600 Ansible tasks, a single pipeline run creates the VMs, applies the configurations and bootstraps the cluster, and all that is left is to install Cilium.\nThe talos-gpu-01 node is already running, with its 12 GB of reserved RAM and its image containing the NVIDIA modules, but it does not see any graphics card yet. Getting the GPU from the Proxmox host all the way to the pods is the subject of part 2.\nUseful links Talos Linux documentation Talos Image Factory siderolabs/talos Terraform provider bpg/proxmox Terraform provider Talos control plane shared VIP KubePrism Deploying Cilium on Talos kubelet-serving-cert-approver ","date":"2026-10-02T13:33:17Z","image":"/posts/talos-cluster-proxmox/images/cover.png","permalink":"/en/posts/talos-cluster-proxmox/","title":"Deploying an immutable Kubernetes cluster with Talos on Proxmox"},{"content":"Introduction As an SRE / Platform Engineer, one of my responsibilities is to provide reliable Helm charts to the development teams. These charts are the foundation for deploying their applications on our Kubernetes platforms. A faulty chart means impacted applications, hours of investigation, and eroded trust in the components we provide.\nThe trigger for this work was an avoidable incident: undocumented breaking changes were introduced and broke every deployment using the chart, which forced us to ask ourselves a fundamental question: how do we guarantee the quality of the Helm charts we provide?\nContext The problem Managing Helm charts by hand and without automated validation carries several risks:\nUndocumented breaking changes: incompatible changes can be introduced without the consuming teams being informed. Syntax errors: typos or YAML errors can go unnoticed until the production deployment. Invalid templates: templates that do not render correctly or generate invalid Kubernetes manifests. Functional regressions: changes that break existing features without being detected. No traceability: hard to understand the history of changes and their impact. The goal I therefore needed a framework able to automatically validate each change before it is merged into the main branch, explicitly flag breaking changes, check the Kubernetes compliance of the generated manifests, test the expected behaviours and track changes in a usable changelog.\nBut it was not only about adding tools. The answer also had to be organisational: better frame contributions, make collaboration easier to follow, and make sure the quality of the chart no longer relies only on individual attention.\nComponents The workflow is built around several tools and practices:\nGitLab CI/CD: orchestrates the whole validation and publication pipeline. Helm \u0026amp; plugins: the package manager for Kubernetes, with its native validation tools and its plugins. Git Flow: branching strategy and Merge Request process. SemVer: versioning to clearly communicate the type of changes. Keep a Changelog: a standard format to document every change. Pipeline architecture The GitLab CI pipeline is organised in three main stages, each with a specific goal.\nPipeline overview Linting The first level of validation in the pipeline checks the syntax and overall structure of the chart with Helm\u0026rsquo;s native helm lint command. It gives fast, immediately understandable feedback. Although it does not cover every case, it removes gross errors very early, before they reach the next steps, which avoids running deeper validations and wasting time.\n1 helm lint . 📝 Note What Helm lint checks:\nCorrect YAML syntax Chart structure (Chart.yaml, templates/, etc.) Formatting issues Warnings on best practices Once the structure of the chart is confirmed, we can move on to more specific validation steps.\nTemplate validation To answer the question \u0026ldquo;does the chart actually render correctly with the provided values?\u0026rdquo;, I chose helm template, which renders the templates. This is often where you find missing variables, badly written conditions, or errors in Go Template functions.\nIt works simply: Helm interprets the templates and produces the final Kubernetes manifests, without applying them to a cluster. You get a concrete view of what would be deployed, which makes it a particularly useful step to detect rendering errors as early as possible.\n1 helm template . 📝 Note What Helm template checks:\nCorrect rendering of Go templates (variables, functions, pipelines) Resolution of values and of the helpers defined in _helpers.tpl Correct syntax of {{- if }} conditions and {{- range }} loops No undefined or wrongly referenced variables Schema compliance helm template guarantees that rendering works, but not that the result is acceptable to Kubernetes. That is precisely why I chose to use Kubeconform. This tool was selected because it is fast, easy to integrate into a CI, and very effective at validating manifests against known schemas.\nIt compares the generated resources to the expected Kubernetes schemas. So it does not only check that the YAML is well formed: it also validates the structure of the manifest, the allowed fields, the expected types and some version incompatibilities. With the CRD catalog added, it remains useful even in Kubernetes environments extended by operators.\nOriginally, this tool is a binary, but a Helm plugin was made so it can be used directly from the Helm CLI.\n1 2 helm plugin install https://github.com/jtyr/kubeconform-helm helm kubeconform . Kubeconform configuration (.kubeconform):\n1 2 3 4 5 6 schema-location: - default # Default K8s resources schema - https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json summary: true verbose: true ignore-missing-schemas: true # Don\u0026#39;t fail on resources where schema is not found. 📝 Note What kubeconform checks:\nInvalid or deprecated fields Incorrect data types Missing values for required fields Compatibility issues with the targeted Kubernetes version Example output:\n1 2 3 4 5 6 7 stdin - Secret mariadb-cluster is valid stdin - StatefulSet mariadb-cluster is valid stdin - Cluster pg-cluster is valid stdin - MongoDBCommunity mongodb-cluster is valid ... ... Summary: 17 resources found parsing stdin - Valid: 17, Invalid: 0, Errors: 0, Skipped: 0 Unit tests I added Helm Unittest because, beyond syntax and compliance, the functional behaviour of the chart had to be protected. It is the tool that lets me state explicit expectations: a name must follow a convention, a resource must exist in a given case, an image must be built with a given value, etc.\nThe principle is close to a classic test framework, but applied to Helm rendering. You describe a set of input values, then express assertions on the produced manifest. This approach is particularly useful to avoid silent regressions when a chart evolves and starts handling several configuration variants.\n1 2 helm plugin install https://github.com/helm-unittest/helm-unittest.git helm unittest . Unit tests check the expected behaviour of the chart in different configurations.\nExample test (tests/deployment_test.yaml):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 suite: test deployment templates: - deployment.yaml tests: - it: should create a valid deployment set: image.tag: latest asserts: - isKind: of: Deployment - matchRegex: path: metadata.name pattern: -my-chart$ - equal: path: spec.template.spec.containers[0].image value: nginx:latest 📝 Note These tests check that:\nThe default values produce valid manifests The different configurations work correctly Critical behaviours are preserved (non-regression) Release management Publication is only triggered when a Git tag is created on the main branch. This process guarantees that only tagged versions are published, and makes it possible to accumulate several changes before creating a release.\nKey point: the CHANGELOG\nEvery Merge Request must include an update of CHANGELOG.md. This file follows the Keep a Changelog format and uses Semantic Versioning.\n1 2 3 4 5 6 7 8 9 ## [1.1.0] - 2026-01-26 ### Added - Resource limits and requests for all containers ### Changed - Updated default memory limit to 512Mi [1.1.0]: https://gitlab.com/project/compare/1.0.9...1.1.0 Available categories:\nAdded: new features Changed: changes to existing features Fixed: bug fixes Removed: removed features 📌 Important If a change introduces a breaking change, it must be:\nClearly stated in the CHANGELOG Explained in the Merge Request description Accompanied by a migration guide if needed Semantic Versioning We strictly follow Semantic Versioning (MAJOR.MINOR.PATCH):\nMAJOR: breaking changes (incompatible with previous versions) MINOR: new backward-compatible features PATCH: backward-compatible bug fixes Example:\n1.0.9 → 1.1.0: adding resource limits (backward-compatible) → MINOR 1.1.0 → 2.0.0: changing the structure of values.yaml (breaking) → MAJOR Observed benefits Since this workflow was set up, we have observed:\nQuality Zero undocumented breaking changes in 6 months 90% fewer deployment errors related to the charts Early detection of problems (before code review) Collaboration A clear process for all contributors (juniors and seniors) More effective code reviews (focus on logic, not syntax) Full traceability of changes through the CHANGELOG Trust The development teams trust the stability of the charts Updates are less feared because the process is transparent Rollbacks are easier thanks to clear versioning Productivity Less time spent on manual validation Automation of repetitive tasks Focus on added value rather than on checking syntax Sequence diagram Complete sequence diagram Points of attention A few lessons learned while setting this up:\nComplete unit tests Do not only test the nominal cases. Include:\nThe default configurations Edge cases The different combinations of values Upgrade scenarios CHANGELOG documentation The CHANGELOG is the contract with users. It must be:\nComplete: every change must be documented Clear: understandable language, no unnecessary technical jargon Actionable: for breaking changes, state what to do Protecting the main branch The protection of main must be strict:\nNo direct commits Mandatory pipeline Mandatory approval The CHANGELOG must be up to date before the merge Protected tags Restrict tag creation to Maintainers to avoid accidental publications.\nCommunication When a breaking change happens, even a well-documented one:\nNotify the teams in advance Provide a migration guide Offer support during the transition Future improvements We are considering adding:\nNon-regression tests: compare the manifests generated between versions with helm diff Security scanning: integrate tools such as Trivy to detect vulnerabilities Policy enforcement: use tools such as OPA or Kyverno to validate security policies Deployment monitoring: track the adoption of new versions Conclusion Setting up a rigorous validation pipeline for our Helm charts has changed the way we work. The incident that pushed us to implement this workflow is now impossible thanks to the multiple levels of validation.\nThis process not only guarantees the technical quality of the charts, it also sets a clear contract with the development teams: they can trust the charts we provide, and when a breaking change is needed, it is clearly documented and communicated.\nFor an SRE team, providing a reliable platform is paramount. This workflow lets us keep that promise while keeping the ability to evolve and continuously improve our charts.\nIf you manage Helm charts for a Kubernetes platform, I strongly encourage you to set up a similar process. The initial investment quickly pays off through fewer incidents and better collaboration.\nUseful resources:\nHelm Documentation Kubeconform Helm Unittest Keep a Changelog Semantic Versioning ","date":"2026-05-11T14:00:00Z","image":"/posts/helm-chart-quality-pipeline/img/helm-flow.svg","permalink":"/en/posts/helm-chart-quality-pipeline/","title":"Ensuring the quality of Helm Charts"},{"content":"Introduction In my homelab, I wanted to take automation further by deploying a complete Kubernetes cluster on Proxmox, relying on infrastructure as code and automation tools such as Ansible. The goal was to be able to quickly provision a reproducible k8s cluster, ready to host workloads, while keeping control over every step of the deployment through code. This progression naturally led me to Kubespray.\nContext After first installing a Kubernetes cluster manually with kubeadm, I found the process particularly tedious. That experience convinced me of the need to automate and standardise the deployment, to gain reproducibility and efficiency.\nThis is where Kubespray comes in: this open source project provides ready-to-use Ansible playbooks to install, configure and maintain a Kubernetes cluster with a 100% as code approach. Unlike kubeadm, it offers fine-grained control over the cluster components (CRI, CNI, DNS, etc.) while remaining automated and reproducible. In this article, I describe the approach I followed, the technical choices I made and the tools I used to reach this goal.\nKubespray logo Prerequisites Before starting the deployment, you need the following resources:\nInfrastructure:\nAt least 24 GB of RAM available on the Proxmox hypervisor 150 GB of free storage Network access with static IP assignment An SSH key to access the VMs Components Proxmox 8.X+: provides the infrastructure to run and manage virtual machines. Ansible 2.16+: the automation tool used by Kubespray to deploy and configure Kubernetes. Terraform 1.10+ \u0026amp; Packer 1.12+: to automate the creation and configuration of the VMs. Target cluster architecture The cluster is made of three nodes with the following sizing:\nRole Hostname vCPU RAM Storage Control Plane k8s-master-01 4 8 GB 50 GB Worker Node k8s-worker-01 4 8 GB 50 GB Worker Node k8s-worker-02 4 8 GB 50 GB Target architecture Creating the VMs For my homelab, I went for a simple architecture made of three VMs: one control plane and two worker nodes.\nThere are several reasons for this choice:\nSimplicity for learning: with three nodes, the cluster stays simple to manage while still allowing you to try most Kubernetes features (load balancing, pod affinity, fault tolerance, etc.). Worker fault tolerance: two workers make it possible to test node failure scenarios and observe how Kubernetes behaves when rescheduling pods. Resource control: this sizing stays reasonable in terms of resources consumed (4 vCPU, 8 GB RAM, 50 GB storage per VM), which matters in a homelab. 📝 Note High availability: this architecture has a single control plane. For real high availability in production, you would need at least 3 control planes (an odd number for the etcd quorum) with a load balancer in front of the API servers, or dedicated etcd instances outside the cluster. For this homelab, a single control plane is enough for learning and experimenting.\nWith the sizing defined, let\u0026rsquo;s create the VMs. For this, I used Terraform and Packer, which lets me deploy my infrastructure quickly and in a fully automated way. I explain this approach in detail, in particular building Proxmox templates with Packer and deploying them with Terraform, in my article on Proxmox templates with Packer.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 ############################ # VMs for Kubernetes Cluster ############################ resource \u0026#34;proxmox_virtual_environment_vm\u0026#34; \u0026#34;k8s_cluster\u0026#34; { for_each = { k8s-master = { description = \u0026#34;K8S Control Plane\u0026#34; vm_id = 110 ip_address = \u0026#34;192.168.1.110/24\u0026#34; tags = [\u0026#34;k8s_cluster\u0026#34;, \u0026#34;kube_control_plane\u0026#34;, \u0026#34;etcd\u0026#34;] } k8s-node1 = { description = \u0026#34;K8S Worker Node 1\u0026#34; vm_id = 111 ip_address = \u0026#34;192.168.1.111/24\u0026#34; tags = [\u0026#34;k8s_cluster\u0026#34;, \u0026#34;kube_node\u0026#34;] } k8s-node2 = { description = \u0026#34;K8S Worker Node 2\u0026#34; vm_id = 112 ip_address = \u0026#34;192.168.1.112/24\u0026#34; tags = [\u0026#34;k8s_cluster\u0026#34;, \u0026#34;kube_node\u0026#34;] } } name = each.key description = each.value.description tags = concat([\u0026#34;debian12\u0026#34;, \u0026#34;terraform\u0026#34;], each.value.tags) node_name = \u0026#34;pve\u0026#34; vm_id = each.value.vm_id pool_id = \u0026#34;PROD\u0026#34; reboot_after_update = false ... clone { vm_id = 1101 datastore_id = \u0026#34;local-lvm\u0026#34; full = true node_name = \u0026#34;pve\u0026#34; } } Once the Terraform code is applied, the three virtual machines are deployed on Proxmox. Each VM gets one or more of these tags: (k8s_cluster, kube_control_plane, etcd, kube_node). These tags make it easier to generate the Kubespray inventory, which I describe further down in the article.\nVMs in Proxmox Network configuration and SSH access Thanks to Cloud-Init during the VM creation process (through the Packer template and the initialization argument of the proxmox_virtual_environment_vm resource of the Terraform provider), the network configuration and SSH access are automated:\nStatic IP address: each VM gets its static IP directly at deployment, defined in the Terraform ip_config section. This avoids any manual configuration after deployment. SSH key: the public SSH key is injected automatically into each VM through Cloud-Init, which allows an immediate connection without copying the key by hand. This process makes it possible to manage all the VMs with Ansible as soon as they are created, without any further manual step. In my case, I can therefore use the Kubespray playbooks directly.\n⚠️ Warning In a production environment, it is recommended to manage secrets (Proxmox API tokens, SSH keys, etc.) with dedicated tools such as Ansible Vault or external secret managers.\nThe resources are now ready and configured, we can move on to setting up the Kubernetes cluster with Kubespray.\nSetting up Kubespray Cloning and configuring the Kubespray repository I added Kubespray to my project as a git submodule. This lets you include another repository while keeping its history separate. It makes dependency management easier and lets you track a specific version of Kubespray, update it easily, and follow the project\u0026rsquo;s changes without mixing source code.\n1 git submodule add https://github.com/kubernetes-sigs/kubespray.git kubespray To pin the submodule to a specific tag or commit, go into the submodule directory and check out the desired commit or tag. Then go back to the main project directory, commit the changes and push them.\n1 2 3 4 cd kubespray/ git checkout v2.27.0 # Version used for this deployment cd ../ git commit -m \u0026#34;Pinned submodule to v2.27.0\u0026#34; Pinning the Kubespray version is important to guarantee reproducible deployments and avoid any instability caused by uncontrolled changes in the upstream project.\nOnce the Kubespray project is integrated and configured in the Ansible repository, we can move on to generating the Kubespray inventory.\nCreating the Kubespray inventory For the Ansible inventory used by Kubespray, I chose the Proxmox dynamic inventory. This gives an inventory that is always up to date, since it is loaded at every playbook run, which avoids maintaining the inventory file by hand.\n1 2 3 4 5 6 7 8 9 10 11 12 plugin: community.general.proxmox url: https://\u0026lt;proxmox_host\u0026gt;:8006 user: \u0026#34;\u0026lt;user\u0026gt;\u0026#34; token_id: \u0026#34;\u0026lt;token_id\u0026gt;\u0026#34; token_secret: \u0026#34;\u0026lt;token_secret\u0026gt;\u0026#34; validate_certs: false # Only in a homelab want_facts: true keyed_groups: - key: proxmox_tags_parsed separator: \u0026#34;\u0026#34; compose: ansible_host: proxmox_agent_interfaces[1][\u0026#34;ip-addresses\u0026#34;][0].split(\u0026#39;/\u0026#39;)[0] The inventory above retrieves all the VMs in Proxmox, groups them by their proxmox_tags_parsed tags and sets the ansible_host variable of each inventory entry to the IP of each VM.\nFor Kubespray, the tags must follow the naming described in the documentation, because that is what runs the playbooks on the right hosts.\nTo generate the inventory, I use the command below, where the --graph option makes the output a bit more visual, to clearly tell apart the groups and hosts of the Ansible inventory.\n1 ansible-inventory -i inventories/proxmox.yml --graph In my case, this gives the following result:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 @all: ... ... |--@debian12: | |--k8s-node1 | |--k8s-master | |--k8s-node2 |--@k8s_cluster: | |--k8s-node1 | |--k8s-master | |--k8s-node2 |--@kube_node: | |--k8s-node1 | |--k8s-node2 |--@terraform: | |--k8s-node1 | |--k8s-master | |--k8s-node2 |--@etcd: | |--k8s-master |--@kube_control_plane: | |--k8s-master ... ... Once the inventory is in place, I can test the connectivity of the VMs with the ad hoc command below, which uses the Ansible ping module:\n1 ansible -i inventories/proxmox.yml k8s_cluster -m ping -u \u0026lt;username\u0026gt; Which gives the following result:\n1 2 3 4 5 6 7 8 9 10 11 12 k8s-node1 | SUCCESS =\u0026gt; { \u0026#34;changed\u0026#34;: false, \u0026#34;ping\u0026#34;: \u0026#34;pong\u0026#34; } k8s-master | SUCCESS =\u0026gt; { \u0026#34;changed\u0026#34;: false, \u0026#34;ping\u0026#34;: \u0026#34;pong\u0026#34; } k8s-node2 | SUCCESS =\u0026gt; { \u0026#34;changed\u0026#34;: false, \u0026#34;ping\u0026#34;: \u0026#34;pong\u0026#34; } Now that I have the Proxmox dynamic inventory and reachable virtual machines, I can move on to customising my K8s cluster.\nCustomising the variables To customise the Kubernetes cluster, you go through Ansible group_vars. These variables let us configure various aspects of the cluster, such as:\nKubernetes settings (kube version, internal network, DNS, \u0026hellip;) Container Runtime Interface (CRI) Container Network Interface (CNI) Container Storage Interface (CSI) ETCD configuration Addons and optional components (ArgoCD, Helm, cert-manager, MetalLB, \u0026hellip;) And much more\u0026hellip; These variables are spread across several inventory files to better structure the configuration:\n1 2 3 4 5 6 7 8 9 10 inventories/group_vars/ ├── etcd │ ├── all.yml │ └── etcd.yml └── k8s_cluster ├── addons.yml ├── all.yml ├── cri-o.yml ├── k8s-cluster.yml └── k8s-net-flannel.yml For more information about the possible cluster configuration and the available components, refer to the Kubespray documentation.\n📝 Note In this article, I mostly use the default configurations provided by Kubespray. The goal is to show the deployment process rather than justify specific technical choices for each component (CNI, CRI, etc.). These choices can be adapted to the specific needs of each project.\nThe target configuration of the Kubernetes cluster is defined, the next step is the deployment!\nDeploying the Kubernetes cluster Before running the deployment with Ansible, I need to set up my environment. To do this, I use a Python venv to pin the version of Ansible and of my dependencies, so that everything is isolated and potential conflicts with other projects are avoided. The venv also gives a reproducible environment, which is crucial to run deployments identically.\n1 2 3 python3 -m venv venv source venv/bin/activate pip install -r kubespray/requirements.txt Once the environment is ready, it is finally time to deploy the cluster, by running the cluster.yml playbook with the Proxmox dynamic inventory configured earlier.\n1 ansible-playbook -i inventories/proxmox.yml -b -u \u0026lt;username\u0026gt; kubespray/cluster.yml Once the playbook has finished, the Kubernetes cluster is ready to use with the configuration defined above.\n1 2 3 4 PLAY RECAP ************************************************************************* k8s-master: ok=599 changed=39 unreachable=0 failed=0 skipped=965 rescued=0 ignored=1 k8s-node1 : ok=421 changed=25 unreachable=0 failed=0 skipped=618 rescued=0 ignored=1 k8s-node2 : ok=443 changed=28 unreachable=0 failed=0 skipped=681 rescued=0 ignored=1 Checking the cluster Once the deployment is complete, it is essential to check that the Kubernetes cluster is operational and that all components work correctly. Several kubectl commands let you validate the state of the nodes and pods.\nTo check the state of the cluster nodes:\n1 kubectl get nodes Result:\n1 2 3 4 NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane 1d v1.31.4 k8s-node1 Ready worker 1d v1.31.4 k8s-node2 Ready worker 1d v1.31.4 All nodes must be in the Ready state, which means they are ready to run workloads.\nTo make sure all pods are running, use the command:\n1 kubectl get pods -A Result:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 NAMESPACE NAME READY STATUS RESTARTS AGE external-dns external-dns-754bcb687f-cqncz 1/1 Running 0 24h ingress-nginx ingress-nginx-controller-6d7ff9f49f-2b9dq 1/1 Running 0 24h kube-system coredns-d665d669-7h6nf 1/1 Running 0 24h kube-system coredns-d665d669-fklwz 1/1 Running 0 24h kube-system dns-autoscaler-5cb4578f5f-zkr67 1/1 Running 0 24h kube-system kube-apiserver-k8s-master 1/1 Running 0 24h kube-system kube-controller-manager-k8s-master 1/1 Running 0 24h kube-system kube-flannel-9n6zs 1/1 Running 0 24h kube-system kube-flannel-m5xtr 1/1 Running 0 24h kube-system kube-flannel-qchzm 1/1 Running 0 24h kube-system kube-proxy-54k4k 1/1 Running 0 24h kube-system kube-proxy-qpb4m 1/1 Running 0 24h kube-system kube-proxy-zq7ww 1/1 Running 0 24h kube-system kube-scheduler-k8s-master 1/1 Running 0 24h kube-system nginx-proxy-k8s-node1 1/1 Running 0 24h kube-system nginx-proxy-k8s-node2 1/1 Running 0 24h kube-system nodelocaldns-l2nc4 1/1 Running 0 24h kube-system nodelocaldns-pp8ng 1/1 Running 0 24h kube-system nodelocaldns-x522v 1/1 Running 0 24h kubernetes-dashboard kubernetes-dashboard-8984b9757-2pcmz 1/1 Running 0 24h kubernetes-dashboard kubernetes-metrics-scraper-6d4c5d99f9-z6g6b 1/1 Running 0 24h metallb-system controller-7f649565d4-6dcd8 1/1 Running 0 24h metallb-system speaker-6pj99 1/1 Running 0 24h metallb-system speaker-v44zd 1/1 Running 0 24h metallb-system speaker-w8wzp 1/1 Running 0 24h All pods must be in the Running state. This guarantees that the cluster components are correctly deployed and working.\nThese checks confirm that the Kubernetes cluster is ready to host workloads and that the essential services are up.\nResult Kubernetes cluster after deployment Post-deployment operations Lifecycle management Kubespray is not limited to the initial deployment, it also provides playbooks to manage the full lifecycle of the cluster:\nAdding nodes:\n1 2 # To add a new worker node to the cluster ansible-playbook -i inventories/proxmox.yml -b -u \u0026lt;username\u0026gt; kubespray/scale.yml Upgrading the cluster:\n1 2 # To upgrade the Kubernetes version or the components ansible-playbook -i inventories/proxmox.yml -b -u \u0026lt;username\u0026gt; kubespray/upgrade-cluster.yml Full reset:\n1 2 # To remove the cluster completely ansible-playbook -i inventories/proxmox.yml -b -u \u0026lt;username\u0026gt; kubespray/reset.yml Conclusion Automating the deployment of a Kubernetes cluster with Ansible through the Kubespray project brings reliability, speed and reproducibility, which is essential for me in my homelab.\nThanks to the dynamic inventory and the Ansible variables, it is easy to adapt the cluster configuration to my needs, while keeping control over each component. This approach also makes maintaining and evolving the k8s cluster easier, since everything is done as code and Kubespray also provides playbooks to add a node to the cluster (scale.yml) or even reset the cluster (reset.yml).\nWorkflow Workflow summary Useful links Kubespray documentation Ansible inventory for Proxmox documentation Example Kubespray group_vars ","date":"2025-11-24T10:20:38Z","image":"/posts/k8s-cluster-deployment/img/workflow.png","permalink":"/en/posts/k8s-cluster-deployment/","title":"Deploying a Kubernetes cluster with Ansible"},{"content":"Introduction In my homelab, I recently worked on a project that let me create virtual machine (VM) templates on Proxmox using a combination of tools such as Packer, Cloud-Init and Terraform. The goal was to create VMs in an automated and repeatable way, while keeping manual steps to a minimum.\nContext Managing VMs and their configuration by hand can quickly become tedious, especially when deploying several instances with almost identical configurations every time. So I needed a solution to standardise this whole process and have a ready-to-use VM template to serve as the base for all my VMs.\nIn the rest of this article, I explain how I reached this goal with different tools, in order to streamline deployments in my homelab.\nComponents Proxmox: provides the infrastructure to run and manage virtual machines. Packer: automates the installation and configuration of the OS to produce a standardised image or template. Terraform: defines, provisions and manages infrastructure through declarative code. Cloud-Init: configures machines dynamically at first boot. Packer Packer is a community tool for creating identical machine images for several platforms from a single source configuration. To do this, Packer uses what are called builders, Packer components able to create a virtual machine template or image on a specific platform.\nPacker logo In my case, I simply use Packer on my Proxmox environment through its dedicated builder.\n1 2 3 4 5 6 7 8 9 #packer.pkr.hcl packer { required_plugins { name = { version = \u0026#34;1.2.2\u0026#34; source = \u0026#34;github.com/hashicorp/Proxmox\u0026#34; } } } Template configuration This builder can either clone a virtual machine to turn it into a template (Proxmox-clone) or create a virtual machine template from a source ISO (Proxmox-iso). To meet my initial need, I use the Proxmox-iso builder. For the configuration, I used the builder\u0026rsquo;s documentation.\nExample packer.pkr.hcl file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 #packer.pkr.hcl source \u0026#34;Proxmox-iso\u0026#34; \u0026#34;debian12\u0026#34; { # Proxmox server Proxmox_url = \u0026#34;proxmox_api_url\u0026#34; node = \u0026#34;pve\u0026#34; username = \u0026#34;\u0026lt;Proxmox_token_id\u0026gt;\u0026#34; token = \u0026#34;\u0026lt;Proxmox_token\u0026gt;\u0026#34; insecure_skip_tls_verify = true # VM Info template_name = \u0026#34;pckr-debian-12\u0026#34; vm_name = \u0026#34;pckr-debian-12\u0026#34; template_description = \u0026#34;Debian 12.9.0, generated on ${timestamp()}\u0026#34; tags = \u0026#34;template;debian12\u0026#34; vm_id = 1101 pool = \u0026#34;TEMPLATE\u0026#34; os = \u0026#34;l26\u0026#34; machine = \u0026#34;q35\u0026#34; # Boot boot_iso { type = \u0026#34;scsi\u0026#34; iso_file = \u0026#34;local:iso/debian-12.9.0-amd64-netinst.iso\u0026#34; unmount = true iso_checksum = \u0026#34;none\u0026#34; } boot_command = [\u0026#34;\u0026lt;esc\u0026gt;\u0026lt;wait\u0026gt;auto console-keymaps-at/keymap=fr console-setup/ask_detect=false debconf/frontend=noninteractive fb=false url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg\u0026lt;enter\u0026gt;\u0026#34;] boot_wait = \u0026#34;10s\u0026#34; http_directory = \u0026#34;http\u0026#34; # Disks disks { type = \u0026#34;scsi\u0026#34; storage_pool = \u0026#34;local-lvm\u0026#34; disk_size = \u0026#34;30G\u0026#34; format = \u0026#34;raw\u0026#34; } # Networks network_adapters { bridge = \u0026#34;vmbr0\u0026#34; model = \u0026#34;virtio\u0026#34; } # Spec bios = \u0026#34;seabios\u0026#34; qemu_agent = true scsi_controller = \u0026#34;virtio-scsi-pci\u0026#34; cpu_type = \u0026#34;x86-64-v2-AES\u0026#34; sockets = 1 cores = 1 memory = 1024 ballooning_minimum = 1024 # Communication communicator = \u0026#34;ssh\u0026#34; ssh_handshake_attempts = 5 ssh_pty = true ssh_timeout = \u0026#34;10m\u0026#34; ssh_username = \u0026lt;username\u0026gt; ssh_password = \u0026lt;password\u0026gt; # Cloud-init cloud_init = \u0026#34;true\u0026#34; cloud_init_storage_pool = \u0026#34;local-lvm\u0026#34; } In this packer.pkr.hcl file, I give the builder the information about the target machine and its specifications for Proxmox, as well as the boot information (path of the ISO file, boot command, etc.).\nPreseed file The boot section gives the virtual machine, the future template, the path of the preseed.cfg file, served by a web server that Packer starts during the process. This file answers the questions asked during the OS installation, without having to type the answers by hand while the installation runs, which fully automates the installation. An example of this file is available in the Debian documentation.\nBuilding the template Once the configuration is ready, build the template with the Packer CLI using the commands below:\n1 2 3 4 5 6 7 8 # Initialise the repository to fetch the dependencies packer init packer.pkr.hcl # Validate the syntax and configuration packer validate packer.pkr.hcl # Build the image, replacing the template if it already exists packer build -on-error=ask -force packer.pkr.hcl The template is now present in Proxmox with all the characteristics I set in the packer.pkr.hcl file, and it is ready to be used for future virtual machine deployments.\nTemplate in Proxmox Now on to the next step: dynamic configuration with Cloud-Init!\nCloud-init Cloud-Init is an open source tool that automates the initial configuration of virtual machines at their first boot. Using configuration files (user-data and meta-data), Cloud-Init can perform various tasks such as creating users, injecting SSH keys, configuring the network or installing packages. This tool works with many cloud platforms and virtualisation environments, which makes it an agnostic way to customise instances dynamically without manual intervention.\nCloud-Init logo Example user-data file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 #user-data.yml #cloud-config ssh_pwauth: false disable_root: true users: - name: user gecos: user groups: users, sudo shell: /bin/bash ssh_authorized_keys: - ssh-ed25519 AAAA....... sudo: ALL=(ALL) NOPASSWD:ALL lock_passwd: true package_update: true package_upgrade: true In my case, I chose to use cloud-init to:\nDisable SSH password authentication as well as the root account. Create a user with an authorised SSH key, passwordless sudo access and a locked password. Automatically update and upgrade the packages. Once applied at the machine\u0026rsquo;s first boot, the machine is ready to use, and also ready to be provisioned with a tool such as Ansible. That was my goal, since I wanted to run my Ansible playbooks over an SSH key connection once the virtual machine was deployed, without copying my key by hand or having to log in with a password first.\nTerraform To deploy my virtual machine from my template and the cloud-init file, I chose to use Terraform. Terraform is an infrastructure as code tool, meaning a tool that lets you define your infrastructure through declarative configuration files in HCL (HashiCorp Configuration Language). To do this, Terraform relies on providers that handle communication with the underlying infrastructure. This method lets you manage infrastructure like code, which makes versioning and reproducible deployments easier.\nThese plugins expose resource types (create/modify resources) and data sources (read existing information) that let Terraform manage the infrastructure behind the plugin. Providers use the vendor\u0026rsquo;s SDK or API to carry out all the actions described in the Terraform configuration.\nTerraform logo For my homelab, I chose the bpg provider, a provider built by a community member that manages the whole Proxmox environment (resource pools, users, VMs, ISOs).\n1 2 3 4 5 6 7 8 9 10 11 #providers.tf provider \u0026#34;proxmox\u0026#34; { endpoint = var.endpoint api_token = var.api_token insecure = true ssh { agent = false username = \u0026#34;root\u0026#34; private_key = var.private_key } } To deploy the virtual machine from the template, with cloud-init for provisioning, I used the proxmox_virtual_environment_vm resource, which creates and manages a virtual machine with all the characteristics you pass to it.\nExample Terraform resource for a virtual machine in Proxmox:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 #virtual_machine.tf resource \u0026#34;proxmox_virtual_environment_vm\u0026#34; \u0026#34;example_vm\u0026#34; { name = \u0026#34;example\u0026#34; description = \u0026#34;example VM\u0026#34; tags = [\u0026#34;debian12\u0026#34;, \u0026#34;terraform\u0026#34;] node_name = \u0026#34;pve\u0026#34; vm_id = 101 pool_id = \u0026#34;example\u0026#34; reboot_after_update = false agent { enabled = true } cpu { cores = 2 sockets = 1 type = \u0026#34;x86-64-v2-AES\u0026#34; } memory { dedicated = 4096 floating = 4096 } initialization { user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id dns { servers = [\u0026#34;9.9.9.9\u0026#34;, \u0026#34;1.1.1.1\u0026#34;] } ip_config { ipv4 { address = \u0026#34;192.168.1.101/24\u0026#34; gateway = \u0026#34;192.168.1.1\u0026#34; } } } clone { vm_id = 1101 datastore_id = \u0026#34;local-lvm\u0026#34; full = true node_name = \u0026#34;pve\u0026#34; } } In my case, the parameters I care about are:\ninitialization: the cloud-init configuration, where I set the DNS configuration and the IP address of my machine, and give the ID of my cloud-init file, also managed with Terraform through the proxmox_virtual_environment_file resource 1 2 3 4 5 6 7 8 9 10 11 12 initialization { user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id dns { servers = [\u0026#34;9.9.9.9\u0026#34;, \u0026#34;1.1.1.1\u0026#34;] } ip_config { ipv4 { address = \u0026#34;192.168.1.101/24\u0026#34; gateway = \u0026#34;192.168.1.1\u0026#34; } } } clone: the clone configuration, where I give the information about my template in Proxmox. 1 2 3 4 5 6 clone { vm_id = 1101 datastore_id = \u0026#34;local-lvm\u0026#34; full = true node_name = \u0026#34;pve\u0026#34; } I can then run my Infrastructure as Code project with the new resource added, and Terraform clones the template and gives it the cloud-init configuration to provision the virtual machine.\nThe result is a VM with a standard configuration that I defined to match my needs, ready to be provisioned with a tool such as Ansible.\nSequence diagram Sequence diagram Conclusion Setting up an automated process to create and deploy virtual machines on Proxmox with Packer, Cloud-Init and Terraform has made my homelab more efficient, reproducible and flexible.\nThis workflow standardises the environments, reduces the risk of manual errors and makes provisioning the machines with Ansible or other automation tools easier. With this solution, I can now quickly deploy ready-to-use virtual machines, while adapting their configuration to my needs.\n","date":"2025-02-28T10:20:38Z","image":"/posts/proxmox-packer-template/images/sequence_diagram.svg","permalink":"/en/posts/proxmox-packer-template/","title":"VM template in Proxmox"},{"content":"Introduction Managing secrets securely in Kubernetes is crucial to keeping our applications secure. In this article, I present secret management in Kubernetes using External Secrets Operator and Azure Key Vault with workload identity authentication, with the aim of having a robust and secure solution.\nContext I set up this solution during a one-week project as part of my master\u0026rsquo;s degree. The goal was to deploy an application stack in the cloud through a GitOps workflow. The project is available in the projet-cicd-m2-do GitLab group.\nIn this project, an Azure Kubernetes Service (AKS) cluster was deployed with Terraform, with the following configuration required to set up the solution:\n1 oidc_issuer_enabled = true ⬆️ Enables the OIDC (OpenID Connect) issuer of the AKS cluster. This means the AKS cluster generates an OIDC token issuer URL, which is essential for the trust relationship I present later in this article.\n📝 Note OIDC is a modern authentication protocol built on the OAuth 2.0 standard. It lets applications verify a user\u0026rsquo;s identity by relying on an identity provider and retrieve information about that user securely.\n1 workload_identity_enabled = true ⬆️ Enables the Azure Workload Identity feature, so that Kubernetes pods can access Azure resources using managed identities.\nComponents External Secret Operator (ESO): a Kubernetes operator that integrates an external secret management system. Azure Key Vault: an Azure cloud service to store secrets and access them securely. Workload Identity: a method to authenticate to an Azure resource without explicitly configuring credentials. Azure Key Vault As soon as the AKS cluster is up, the Azure Key Vault resource has to be deployed. For this, I used Terraform, to do IaC and make my deployment reliable and automated. For the vault and the other Azure resources, I used the azurerm Terraform provider.\nThe following code creates the vault in the defined resource group and location, with RBAC (Role Based Access Control) enabled to manage access and the SKU (Stock Keeping Unit) set to standard. Other arguments are available, but I preferred to keep it simple given the length of the project, to get a POC quickly.\n1 2 3 4 5 6 7 8 9 resource \u0026#34;azurerm_key_vault\u0026#34; \u0026#34;vault\u0026#34; { name = join(\u0026#34;-\u0026#34;, [var.prj, var.org, \u0026#34;vault\u0026#34;]) location = var.location resource_group_name = azurerm_resource_group.aks.name tenant_id = data.azurerm_client_config.current.tenant_id sku_name = \u0026#34;standard\u0026#34; enable_rbac_authorization = true } Once the vault is deployed, you can put your secrets in it through the Azure UI or CLI.\nAzure Vault UI Managed Identity Once the vault is deployed, the role has to be assigned to the right identity so that our secrets can be accessed through it.\nKubelet Identity When an AKS cluster is created, a user assigned managed identity is created in the resource group that Azure manages automatically and that holds the actual infrastructure resources needed to run the AKS cluster. This managed identity is called kubelet_identity, and it lets the kubelet on the cluster nodes access other Azure resources. It is through this identity that I can access the vault.\nInteraction between the Kubelet Identity and Azure resources I gave this identity read access to the secrets in the vault with the Key Vault Secrets User role:\n1 2 3 4 5 resource \u0026#34;azurerm_role_assignment\u0026#34; \u0026#34;keyvault_secrets_user\u0026#34; { scope = azurerm_key_vault.vault.id role_definition_name = \u0026#34;Key Vault Secrets User\u0026#34; principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id } Identity federation Once the right permissions are assigned, this identity has to be linked to a service account in Kubernetes so that the data in my Azure vault can be accessed.\nTo do this, you use a federated identity credential, which delegates authentication to an external identity provider, in my case the AKS cluster, and thus creates a trust relationship between AKS and the managed identity in Microsoft Entra ID.\n1 2 3 4 5 6 7 8 resource \u0026#34;azurerm_federated_identity_credential\u0026#34; \u0026#34;identity\u0026#34; { name = \u0026#34;kub-federated-identity\u0026#34; resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group audience = [\u0026#34;api://AzureADTokenExchange\u0026#34;] issuer = azurerm_kubernetes_cluster.aks.oidc_issuer_url parent_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].user_assigned_identity_id subject = \u0026#34;system:serviceaccount:default:workload-identity-sa\u0026#34; } Here the audience is the intended recipient of the token issued by the OIDC issuer, the subject is the identity of the entity (the service account in my case) the federated identity is associated with, and issuer is the OIDC issuer URL of the AKS cluster.\nSequence diagram Here is a more detailed diagram of how the connection to the Vault works with managed identity and federation:\nSequence diagram of the connection to the Vault External Secrets Operator Next, the Kubernetes operator, External Secrets Operator (ESO), has to be set up to connect to the vault through the Azure Key Vault provider that ESO ships. For this, Custom Resource Definitions (CRDs) exist that define how to access the vault (Secret Store) and which data to retrieve (External Secret).\nSecret Store In this resource, you specify the vault URL, the Azure tenant ID, the authentication type and the associated service account. The secret store is what lets me connect to Azure Key Vault.\n1 2 3 4 5 6 7 8 9 10 11 12 apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: azure-secret-store spec: provider: azurekv: authType: WorkloadIdentity vaultUrl: ${azurerm_key_vault.vault.vault_uri} tenantId: ${azurerm_key_vault.vault.tenant_id} serviceAccountRef: name: workload-identity-sa External Secret This resource defines which secret needs to be synced between the vault and the Kubernetes secret. You define the secret store to use, the target secret in Kubernetes and the secrets to retrieve from the Azure vault.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: my-secret spec: refreshInterval: 30s secretStoreRef: name: azure-secret-store kind: SecretStore target: name: ocf-secret data: - secretKey: pgadmin-password remoteRef: key: pgadmin-password Service Account You also need to create the service account and add the following annotation to it, to associate it with the ID of the identity: Kubelet Identity.\n1 2 3 4 5 6 7 apiVersion: v1 kind: ServiceAccount metadata: annotations: azure.workload.identity/client-id: \u0026#34;${azurerm_kubernetes_cluster.aks.kubelet_identity[0].client_id}\u0026#34; name: workload-identity-sa namespace: default Deployment For the deployment, I also used Terraform, with the helm and kubectl providers. This gives a unified infrastructure, automated end to end.\nDeployment with Terraform ESO For ESO, a Helm chart is provided in the documentation, and it is deployed in the external-secrets namespace. This Terraform resource waits for the AKS cluster deployment to finish before being created, thanks to depend_on = [azurerm_kubernetes_cluster.aks].\n1 2 3 4 5 6 7 8 9 resource \u0026#34;helm_release\u0026#34; \u0026#34;eso\u0026#34; { depends_on = [azurerm_kubernetes_cluster.aks] name = \u0026#34;external-secrets\u0026#34; namespace = \u0026#34;external-secrets\u0026#34; repository = \u0026#34;https://charts.external-secrets.io\u0026#34; chart = \u0026#34;external-secrets\u0026#34; create_namespace = true version = \u0026#34;0.10.5\u0026#34; } ESO pods deployed Once ESO is deployed, you need to create the CRDs that configure the connection to the vault and the secrets to retrieve, as well as the service account. All of this with Kubernetes manifests, which declare the resources in YAML.\nCRDs and Service Account For the manifests, the kubectl Terraform provider lets me use variables from Terraform resources directly in the manifests to make them dynamic, as you can see in this manifest. This avoids hardcoding sensitive information in the manifests.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 resource \u0026#34;kubectl_manifest\u0026#34; \u0026#34;sa\u0026#34; { depends_on = [helm_release.eso] yaml_body = \u0026lt;\u0026lt;YAML apiVersion: v1 kind: ServiceAccount ..... YAML } resource \u0026#34;kubectl_manifest\u0026#34; \u0026#34;secretstore\u0026#34; { depends_on = [kubectl_manifest.sa] yaml_body = \u0026lt;\u0026lt;YAML apiVersion: external-secrets.io/v1beta1 kind: SecretStore ..... YAML } resource \u0026#34;kubectl_manifest\u0026#34; \u0026#34;externalsecret\u0026#34; { depends_on = [kubectl_manifest.secretstore] yaml_body = \u0026lt;\u0026lt;YAML apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret ..... YAML } The 3 Terraform resources are declared, and they match the 3 Kubernetes manifests / resources that must be created in the cluster to configure and use ESO.\nResult The connection to Azure Key Vault is validated, thanks to the secret-store resource.\nConnection to Azure Key Vault with the secret store Once connected to the vault, the secrets are synced. You can see it on the external-secret resource, which is in the SecretSynced state. It retrieves the secrets from the vault and puts them into a Kubernetes secret.\nSecret sync between the vault and Kubernetes The secret has been created and is present in Kubernetes, as you can see. For this demonstration the passwords are all the same, which of course should not be done in a production environment.\nKubernetes secret Conclusion Setting up this solution in a short time was a real challenge for me, as I had never done it before. I was able to work on and deepen my skills on Azure, and more specifically on the roles and identities of this cloud provider.\nThe goal was reached, and the complexity of the solution motivated me to write this article, first to keep a record of this work as a write-up, but also to help the people who will come across it sooner or later.\n","date":"2024-11-23T10:20:38Z","image":"/posts/k8s-eso-azvault/images/cover.png","permalink":"/en/posts/k8s-eso-azvault/","title":"Kubernetes secrets from Azure Key Vault"},{"content":"Why tech watch matters As a DevOps engineer, keeping a tech watch is a crucial practice: it keeps you up to date and lets you constantly follow the arrival of new tools in a field that never stops evolving.\nThere are several reasons for this. First, as a DevOps engineer, you constantly have to find ways to improve the CI/CD pipelines and the various processes in place in a company, so looking for new tools, new versions or new technologies is essential.\nThen, in my case, this watch lets me be a source of proposals and a driving force in my environment, by bringing my knowledge of recent topics.\nTech watch tools The tools used for the watch are the key: they are what makes it quick and efficient to manage. So you need to re-evaluate these tools periodically, so that they do not become obsolete and still fit the need.\nWhich means you also need to keep a watch on your watch tools!\nFeeder After using feedly daily for a while, I was not convinced by the app, I needed a simpler and cleaner tool.\nAfter some research, I came across Feeder, an open source RSS reader and aggregator available on Android. The app does not require logging in with an account and it runs locally, which keeps my data private.\nThe design of the app is very simple and clean, and its configuration has the same qualities. Feeder also lets you save articles, add them to favourites or even back up your configuration. These are very handy features, and necessary for a tool of this kind.\nFeeder interface Feeder lets me fetch the articles from my main IT sites, listed in the next section, and read them from the app. For example, I can use my commuting time to do my watch every day.\nI sort the RSS feeds into different categories according to their field:\nDevOps Linux General IT Cybersecurity Wiki Once I have gathered information from my various sources, I sort it to keep only what I find interesting and/or useful. Once sorted, I format the items I kept and put them into my wiki. (I actually wrote an article on building this wiki.)\nThis gives a clean and condensed result, so I can find what interests me in no time, which is the whole point of the watch for me.\nSources of information I use several sources of information, but I find that the article / post format is the best, the one that suits me most. Written content is often straight to the point, which is not always the case with a video or a podcast, so this format is efficient and concise.\nMost of the sources below are in French, I marked the English ones with 🇬🇧.\nWebsites Blogs Stéphane Robert: DevOps Une tasse de café - Quentin Joly: DevOps IT Connect: Sysadmin / General IT Korben: General IT 🇬🇧 It\u0026rsquo;s foss: Linux RSS aggregators Journal du hacker: General IT 🇬🇧 Hacker News: General IT YouTube channels Xavki: DevOps Yves Rougy: Linux Underscore: General IT / Cybersecurity Devoxx: DevOps / Linux 🇬🇧 NetworkChuck: Cybersecurity Community The community feeds below are mainly English-speaking communities.\nMedium Devops Kubernetes Linux Cloud Reddit r/devops r/homelab r/kubernetes r/selfhosted Conclusion To conclude, I find that the watch lets me learn something every day in a field of IT. It is a kind of continuous improvement process at a human scale, and it keeps me from stagnating and helps my skills grow.\nPersonally, I also like to put my discoveries into practice through a POC in my homelab, to quickly form an opinion on the subject.\n","date":"2024-11-01T22:56:48Z","image":"/posts/veille/cover.png","permalink":"/en/posts/veille/","title":"Tech watch"},{"content":"Introduction With the amount of information coming my way every day across many fields, handwritten notes scattered everywhere and never reused, and the recurring hassle of solving a problem for the umpteenth time without remembering the solution, just like every time before. 😆\nI was missing a tool to act as a second brain 🧠, so that all these notes and already solved errors would be gathered in one place.\nThe need was therefore there, with mandatory requirements that took shape over the various tests:\nQuick to access Pages written in Markdown Lightweight tool Hosted locally What is MkDocs? MkDocs is a fast, simple static site generator geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.\nThis definition from the MkDocs website appealed to me as soon as I came across it, because it met the 4 needs listed above, so I decided to give this tool a try.\nMkDocs site The first version of my wiki used this tool. I used a Dockerfile to containerise the tool and use it locally without installing the mkdocs Python package.\nThe only downside of this tool is that the user interface is a bit hard on the eyes. That is what motivated me to find a theme!\nMaterial Material is a framework used on top of MkDocs that generates a static site customisable through a multitude of options, with the ability to add plugins as well.\nIt gives a very clean result while keeping configuration to a minimum, because many features are already built into the framework and a single line in the configuration file is enough to enable them.\nMkDocs example with Material Setting up the wiki To set up this tool, the Material documentation explains how to deploy it with the mkdocs utility or with a docker container. I went for the docker option, so that I can use this container in my different environments without needing the utility on all my machines.\nFirst, you initialise the project to create the skeleton of the wiki. To do this, you pass the new . command to the docker container, which already has the mkdocs utility as its entrypoint.\n1 docker run --rm -it -v ${PWD}:/docs squidfunk/mkdocs-material new . Once the project is initialised, you end up with the following tree:\n1 2 3 4 . ├─ docs/ │ └─ index.md └─ mkdocs.yml docs: the tree of the wiki mkdocs.yaml: the wiki configuration file Configuration The wiki is configured through the mkdocs.yaml file at the root of the project. This is where you set the theme and the features you want, listed in the Material configuration documentation, as well as the Python extensions you want to use.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 # Configuration du site site_name: My site site_url: https://mydomain.org/mysite # Configuration du thème material theme: name: material icon: repo: fontawesome/brands/gitlab logo: assets/logo.png # Ajout de features features: - content.code.copy - navigation.instant - navigation.tabs - navigation.tabs.sticky - navigation.sections - navigation.path - navigation.top # Automatic light \u0026amp; dark mode palette: # Palette toggle for automatic mode - media: \u0026#34;(prefers-color-scheme)\u0026#34; toggle: icon: material/brightness-auto name: Switch to light mode # Palette toggle for light mode - media: \u0026#34;(prefers-color-scheme: light)\u0026#34; scheme: default toggle: icon: material/brightness-7 name: Switch to dark mode # Palette toggle for dark mode - media: \u0026#34;(prefers-color-scheme: dark)\u0026#34; scheme: slate toggle: icon: material/brightness-4 name: Switch to system preference # Déclaration des plugins du thème material plugins: - search # Extension python pour le markdown markdown_extensions: - pymdownx.superfences: custom_fences: - name: mermaid class: mermaid format: !!python/name:pymdownx.superfences.fence_code_format - admonition - pymdownx.details - pymdownx.superfences - attr_list - pymdownx.emoji: emoji_index: !!python/name:material.extensions.emoji.twemoji emoji_generator: !!python/name:material.extensions.emoji.to_svg In my case, I added several features, for example to copy code blocks, to have tabs per section name, or to choose between light and dark mode. I added the search plugin to get a search bar in my wiki.\nI added Python extensions to use emojis and mermaid diagrams, and to add HTML/CSS classes to my Markdown blocks.\nAdding content The content lives under the docs/ directory: each folder is a section and each file is a page.\nYou can build a very modular structure with N sections, N sub-sections and N files per sub-section, or only create files. It leaves free rein to your imagination.\nThe folder name is the name of the section or sub-section, and for pages, the title is the file name, or the Markdown header (# Title1) if there is one.\n1 2 3 4 5 6 7 8 9 10 11 docs/ ├── assets │ └── logo.png ├── Plantes │ ├── ficus.jpg │ └── Ficus.md └── Tech ├── Git │ └── config.md └── Golang └── issues.md Deployment For the deployment, I run this project locally in a docker container with the docker-compose.yml file, since I have no interest in hosting it on the web for now.\n1 2 3 4 5 6 7 8 9 services: mkdocs-material: restart: \u0026#34;always\u0026#34; ports: - \u0026#34;8000:8000\u0026#34; container_name: \u0026#34;wiki\u0026#34; volumes: - \u0026#34;/path/to/wiki:/docs\u0026#34; image: \u0026#34;squidfunk/mkdocs-material\u0026#34; Conclusion I think adding this wiki to my toolbox will be more and more useful over time. All the knowledge I gather will be in one place, sorted by field. This gives me easy and quick access to information, which increases my productivity and reduces the time I waste.\n","date":"2024-10-06T22:56:48Z","image":"/posts/wiki/cover.png","permalink":"/en/posts/wiki/","title":"Building a personal wiki"},{"content":"This project was done during my first year of Master\u0026rsquo;s at Sup De Vinci, as part of a module on the Go language.\nIntroduction The specifications provided by the lecturer asked for a Golang CLI for one of the following three imposed authentication solutions:\nCasdoor PocketBase Authentik Documentation had to be provided, so that the project would be turnkey and usable by anyone.\nChoosing the solutions Backend After comparing the 3 authentication solutions, I noticed that Authentik and Casdoor are very complete, so they take some time to get to grips with. Given the project deadline and its needs, I went with PocketBase. It is simple to deploy and use, and it meets exactly the requirements of the project.\nPocketBase UI Golang framework For the Go framework, I saw that 2 frameworks are widely favoured in this area: Cobra 🐍 and Kong 🦍.\nBy following YouTube tutorials that used the Cobra framework, its ease of use let me quickly build a POC of my CLI with very intuitive flag handling and automated documentation. So I stuck with this framework. A huge number of projects use Cobra, as you can see in this list.\nI also added the Viper package to the project, which handles configuration in the Golang project, to work with the conf.json file.\nFinally, adding the promptUI package improves the UX, in particular with interactive prompts.\nArchitecture CLI For the architecture of the CLI, I chose to base it on Docker\u0026rsquo;s, which I personally find very intuitive and convenient to use, and which is also built with Cobra. So I went with the same command [resource] [action] [flags] model.\nCLI architecture The CLI can therefore log in to and out of PocketBase, and perform CRUD actions (Create, Read, Update, Delete) on the users in the backend, which meets the request in the specifications.\nOnce the architecture of my CLI was drawn up, the design phase was over and I could start programming the CLI to make it concrete.\nCode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ├── cmd │ ├── auth │ │ ├── login.go │ │ └── logout.go │ ├── root.go │ ├── user │ │ ├── create.go │ │ ├── delete.go │ │ ├── inspect.go │ │ ├── update.go │ │ └── user.go │ └── version │ └── version.go ├── common │ └── common.go ├── go.mod ├── go.sum ├── go-cli └── main.go main.go: initialises the program through the main() function, which is the entry point of the program.\ncmd/: this folder holds all the commands available in the CLI. The files are organised according to the architecture of the CLI.\ncommon/: a module that holds the functions used in several modules of the project.\nDeployment The deployment of the project comes down to a simple docker-compose.yaml file that creates the PocketBase container holding the SQLite database. The pb_data file is also mounted, to have a database already populated with users.\n1 2 3 4 5 6 7 8 services: pocketbase: build: . container_name: pocketbase ports: - 8090:8080 volumes: - ./pb_data:/pb/pb_data The CLI itself runs as a binary that can be downloaded from the GitLab releases or built locally from the code.\nDemo Conclusion This project was quite challenging, because it came right after the one-week course module on Golang. I had to dig further into the language to handle the project properly. Having done some C in the not-so-distant past, this project brought back some good memories. 😄\nI also gained experience in choosing the right tools to meet the project\u0026rsquo;s expectations, and to save time on development.\nFor more details on this project, the code is available in the go-cli GitLab repository!\n","date":"2024-04-10T22:56:48Z","image":"/posts/go-cli/cover.png","permalink":"/en/posts/go-cli/","title":"Golang CLI"}]