Series “A Kubernetes cluster with a GPU for LLM inference”

1. Deploying an immutable Kubernetes cluster with Talos on Proxmox
2. Passing an NVIDIA GPU through to a Kubernetes cluster (this article)
3. Running a Kubernetes cluster with GitOps and Flux (in progress)
4. Self-hosted LLM inference on Kubernetes (in progress)

Introduction

The Talos cluster deployed in part 1 is up and running, but the talos-gpu-01 node does not see any graphics card yet. In this article, I explain how I made a GeForce RTX 3050 usable by Kubernetes pods, starting from the Proxmox host. The work breaks down into four steps: detach the card from the host, attach it to the VM with OpenTofu, load the NVIDIA modules in Talos, then expose the GPU to the Kubernetes scheduler.

Before getting to the commands, I explain what passthrough does, since that is where most of the constraints met later come from, such as disabling ballooning or checking the IOMMU groups.

Context

The end goal of this series is to run LLM inference locally in my homelab, which requires a GPU accessible from the cluster. To give a graphics card to a VM, there are two main approaches: share it between several VMs with vGPU, or dedicate it entirely to a single VM with PCI passthrough.

In my case, the choice was quick. NVIDIA only offers vGPU on its datacenter and professional lines, it is not available on a GeForce. That leaves passthrough, with a direct consequence: the card is dedicated to the talos-gpu-01 VM, the Proxmox host no longer has access to it and no other VM can share it.

The hardware used is a GeForce RTX 3050 that I had on hand.

Prerequisites

• A CPU and motherboard supporting the IOMMU (VT-d on Intel, AMD-Vi on AMD), enabled in the BIOS.
• The Talos cluster from part 1.
• A GPU that is not used for the host display. In my case, the Proxmox host runs headless and the CPU’s integrated GPU is disabled in the BIOS, so the RTX 3050 is the only graphics card in the machine.

Understanding passthrough

A PCIe device such as a GPU does not go through the CPU to read or write memory. It uses DMA (Direct Memory Access) to access RAM directly, which lets it transfer large volumes of data without keeping the processor busy.

In a VM, this causes a problem. The memory addresses seen by the guest system are guest physical addresses (GPA, Guest Physical Address), which do not match the host’s real physical addresses (HPA, Host Physical Address). For CPU accesses, the GPA to HPA translation is done in hardware by EPT (Extended Page Tables) on Intel or NPT (Nested Page Tables) on AMD, a second page table that the processor walks in addition to the guest’s. A device doing DMA does not go through this translation: if the VM’s driver gives a GPA to the GPU, the card writes at that address in the host’s RAM, which corresponds to something else entirely.

That is the job of the IOMMU (Input-Output Memory Management Unit). It does for devices what the MMU (Memory Management Unit), the processor component that translates virtual addresses into physical addresses, does for the CPU: it translates the addresses used by the device into real physical addresses, and it restricts the device to the memory areas assigned to it. The GPU passed to the VM can therefore only write into that VM’s memory.

This isolation is not done per device but per IOMMU group. A group gathers the devices that the hardware cannot isolate from one another, for example because they can communicate with each other without going through the IOMMU. The unit that can be passed to a VM is therefore the whole group, and you need to check the content of the card’s group before starting.

Finally, on the host side, the card must not be claimed by any graphics driver (nouveau or nvidia). The vfio-pci driver takes ownership of it. It is part of VFIO (Virtual Function I/O), the Linux kernel framework that gives a program outside the kernel, here QEMU, direct access to a device, relying on the IOMMU to limit what the device can reach in memory. That is how QEMU presents the card to the VM.

Why ballooning must be disabled

Ballooning lets Proxmox reclaim RAM that a VM is not using, on the fly, thanks to a driver in the guest that hands memory pages back to the host. With a device passed through VFIO, this mechanism can no longer work: the GPU can write into the VM’s pages at any time without going through the CPU, so the host has no way of knowing whether a page is still in use by the card. All of the VM’s memory is therefore pinned at boot and can no longer be given back to the host.

In the bpg/proxmox provider, ballooning is controlled by the floating argument of the memory block, and the value 0 disables it. I set it to 0 in the resource that creates the Talos nodes, so on all three nodes and not only on the GPU node:

1
2
3
4
5

#talos.tf
memory {
  dedicated = each.value.memory
  floating  = 0
}

This affects sizing: the 12 GB of RAM of the talos-gpu-01 node are permanently reserved on the host, even when the node is idle.

Now that the mechanism is clear, let’s prepare the Proxmox host.

Step 1: preparing the Proxmox host

Checking the IOMMU and the groups

The first thing to do is identify the card and the IOMMU group it belongs to:

1
2
3
4
5

# The card and its functions
lspci -nn | grep -i nvidia

# The IOMMU group it belongs to
find /sys/kernel/iommu_groups -type l | grep '01:00'

Which gives in my case:

1
2
3
4
5

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)

/sys/kernel/iommu_groups/10/devices/0000:01:00.0
/sys/kernel/iommu_groups/10/devices/0000:01:00.1

The card exposes two functions: 01:00.0 for the video part and 01:00.1 for the HDMI audio controller. Both are alone in IOMMU group 10, which means the group can be passed to the VM as is, without taking any other host device along.

If other devices shared this group, for example a USB controller, they would have to be passed to the VM too. The ACS override patch is often suggested on forums to split groups artificially, but it asks the kernel to ignore an isolation that the hardware does not guarantee, since the devices in the group can still communicate with each other. The Proxmox wiki actually presents it as a last resort, and not a risk-free one.

Binding the card to vfio-pci

I configured the Proxmox host so that vfio-pci claims the card before any other driver. This part is not managed by my OpenTofu code, I did it directly on the Proxmox host.

According to the Proxmox documentation, the IOMMU is enabled by default on Intel from kernel 6.8, so the intel_iommu=on option found in many guides is no longer needed. The only useful parameter is iommu=pt, which limits address translation to the devices passed to a VM and can improve performance for the others.

Kernel parameters are passed through the command line that GRUB, the host’s bootloader, hands to the kernel at every boot. When the host boots with GRUB, which is my case, it is set in /etc/default/grub:

1
2

#/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

GRUB_CMDLINE_LINUX_DEFAULT applies to normal boot entries, but not to recovery mode. quiet is present by default on Debian and Proxmox: it reduces the kernel messages displayed on screen during boot, without changing what is recorded in the logs.

The host’s dmesg confirms that the IOMMU is active, with the requested passthrough mode, and that the devices are split into IOMMU groups:

1
2
3

[    0.284240] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.329656] pci 0000:00:00.0: Adding to iommu group 0
[    0.329665] pci 0000:00:01.0: Adding to iommu group 1

The VFIO modules are loaded at boot through /etc/modules: vfio for the core of the framework, vfio_iommu_type1 to drive the IOMMU, and vfio_pci for PCI devices:

1
2
3
4

#/etc/modules
vfio
vfio_iommu_type1
vfio_pci

For vfio-pci to reserve the card, I give it the IDs of both functions, the video part 10de:2507 and the audio part 10de:228e:

1
2

#/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2507,10de:228e disable_vga=1

The disable_vga=1 option prevents vfio-pci from exposing the card’s legacy VGA resources to the host. On top of that, I block the NVIDIA graphics drivers on the host so that none of them claims the card at boot:

1
2
3

#/etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia*

Changes to /etc/modules and /etc/modprobe.d/ only take effect after regenerating the initramfs (update-initramfs -u -k all), and changes to /etc/default/grub after an update-grub. After rebooting the host, the check is done with:

1

lspci -nnk -s 01:00

Which gives in my case:

1
2
3
4
5
6
7
8

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507] (rev a1)
        Subsystem: NVIDIA Corporation GA106 [Geforce RTX 3050] [10de:2507]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:2507]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

Both functions are claimed by vfio-pci. The Kernel modules line lists the drivers able to handle the card, not those that are loaded: nouveau and nvidiafb are blocked and do not show up in lsmod.

⚠️ Warning

As long as lspci -nnk does not show Kernel driver in use: vfio-pci on both functions, there is no point going further. The VM may boot, but the card will not be usable.

The host is ready, the card can now be attached to the VM.

Step 2: attaching the GPU to the VM with OpenTofu

Why a hardware mapping and not a PCI ID

The bpg/proxmox provider offers two ways to declare a device in a VM’s hostpci block: its raw PCI ID with the id argument, or a hardware mapping declared at the Proxmox cluster level with the mapping argument.

The provider documentation states that id is not compatible with API token authentication and requires the root account with its password. All my IaC authenticates with the token of the terraform@pve user, so I went with the mapping, which was the only option without changing the provider’s authentication.

The mapping is declared with the proxmox_hardware_mapping_pci resource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#hardware_mapping.tf
resource "proxmox_hardware_mapping_pci" "this" {
  ...
  for_each = var.pci_hardware_mappings

  name    = each.key
  comment = each.value.comment

  map = [
    {
      node         = coalesce(each.value.node, var.proxmox_node)
      path         = each.value.path
      id           = each.value.id
      iommu_group  = each.value.iommu_group
      subsystem_id = each.value.subsystem_id
    }
  ]
}

The values are set in the variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#variables.auto.tfvars
pci_hardware_mappings = {
  gpu = {
    comment      = "GeForce RTX 3050 - managed by Terraform"
    path         = "0000:01:00"
    id           = "10de:2507"
    iommu_group  = 10
    subsystem_id = "10de:2507"
  }
}

The path is declared without a function number (0000:01:00 rather than 0000:01:00.0), which covers both functions of the card at once.

The iommu_group and subsystem_id arguments are optional for the API, but without them Proxmox shows the mapping as incomplete in the UI. So I filled them in from the output of lspci -nnv -s 0000:01:00 and find /sys/kernel/iommu_groups -type l.

Proxmox privileges

The role used by OpenTofu needs three extra privileges:

1
2
3
4

#users.tf
"Mapping.Audit",
"Mapping.Modify",
"Mapping.Use",

• Mapping.Audit: read existing mappings.
• Mapping.Modify: create, modify and delete a mapping.
• Mapping.Use: attach a mapping to a VM, on top of the usual VM modification rights.

Without Mapping.Use, the mapping is created but attaching it to the VM fails.

The hostpci block

On the VM side, a single resource serves all three Talos nodes. The hostpci block is therefore dynamic: it is generated from each node’s hostpci list, which defaults to []. The two nodes without a card thus have no device attached.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#talos.tf
dynamic "hostpci" {
  for_each = { for device in each.value.hostpci : device.device => device }

  content {
    device  = hostpci.value.device
    mapping = proxmox_hardware_mapping_pci.this[hostpci.value.mapping].name
    pcie    = hostpci.value.pcie
    rombar  = hostpci.value.rombar
    xvga    = hostpci.value.xvga
  }
}

In part 1, I mentioned a sixth validation on the talos_nodes variable. Here it is, it checks that each hostpci.mapping matches a declared mapping:

1
2
3
4
5
6
7
8
9

#variables.tf
validation {
  condition = alltrue(flatten([
    for node in var.talos_nodes : [
      for device in node.hostpci : contains(keys(var.pci_hardware_mappings), device.mapping)
    ]
  ]))
  error_message = "Every Talos node hostpci.mapping must be a key of var.pci_hardware_mappings."
}

A typo in the mapping name is thus caught at plan time, and not as an API error in the middle of the apply.

Once the pipeline has run, the VM configuration on the Proxmox side reflects what was declared:

1

qm config 123 | grep -E '^(hostpci|balloon|memory)'

1
2
3

balloon: 0
hostpci0: mapping=gpu,pcie=1,rombar=1,x-vga=0
memory: 12288

The device is referenced by the mapping name and not by its PCI address, and balloon: 0 matches the floating = 0 of the memory block.

The card is now attached to the VM, it remains to make it usable by Talos.

Step 3: the NVIDIA modules in Talos

A schematic dedicated to the GPU node

As explained in part 1, the NVIDIA extensions are not in the base image: the nodes without a card reported module loading errors at every boot. The talos-gpu-01 node therefore uses its own schematic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

#variables.auto.tfvars
{
  hostname     = "talos-gpu-01"
  ip           = "192.168.1.123"
  cores        = 4
  memory       = 12 * 1024,
  disk_size    = 100,
  role         = "worker"
  schematic_id = "a31d81eceab05afdc2c59b0002701173e117a5fb81bdc1f64062d27655145848" # With NVIDIA GPU support
  hostpci = [
    {
      device  = "hostpci0"
      mapping = "gpu"
    }
  ]
}

This schematic adds two extensions to qemu-guest-agent: nvidia-open-gpu-kernel-modules-production for the drivers, and nvidia-container-toolkit-production so that the container runtime knows how to expose the card to pods.

📝 Note

I use NVIDIA’s open source drivers. They support GPUs from the Turing architecture onwards, which includes the RTX 3050 (Ampere). On older hardware, you need the proprietary extensions.

Loading the modules

The extensions ship the kernel modules but do not load them. That is the job of a machine config patch, placed at the node level and not at the worker/ level:

1
2
3
4
5
6
7
8

# patches/nodes/talos-gpu-01/30-nvidia-modules.yaml.tftpl
machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset

This patch uses the nodes/ level presented in part 1: placed under worker/, it would also be applied to talos-worker-01, whose image does not contain the extensions.

The list of modules comes from the Talos documentation. The nvidia_uvm module is the one CUDA uses to manage unified memory.

Applying and checking

The schematic and the patch are deployed by the GitLab CI pipeline described in part 1, like the rest of the node configuration. I then check that the extensions are present in the node’s image:

1

talosctl -n 192.168.1.123 get extensions

Which gives the following result:

1
2
3
4
5
6

NODE            NAMESPACE   TYPE              ID            VERSION   NAME                                        VERSION
192.168.1.123   runtime     ExtensionStatus   0             1         nvidia-container-toolkit-production         595.71.05-v1.19.1
192.168.1.123   runtime     ExtensionStatus   1             1         nvidia-open-gpu-kernel-modules-production   595.71.05-v1.13.8
192.168.1.123   runtime     ExtensionStatus   2             1         qemu-guest-agent                            11.0.2
192.168.1.123   runtime     ExtensionStatus   3             1         schematic                                   a31d81eceab05afdc2c59b0002701173e117a5fb81bdc1f64062d27655145848
192.168.1.123   runtime     ExtensionStatus   modules.dep   1         modules.dep                                 6.18.42-talos

Both NVIDIA extensions are at version 595.71.05, the driver number, and the schematic ID is the one declared for this node. The four modules from the patch must then be loaded:

1

talosctl -n 192.168.1.123 get loadedkernelmodules | grep -i nvidia

1
2
3
4

192.168.1.123   runtime     LoadedKernelModule   nvidia                      2         Live
192.168.1.123   runtime     LoadedKernelModule   nvidia_drm                  2         Live
192.168.1.123   runtime     LoadedKernelModule   nvidia_modeset              2         Live
192.168.1.123   runtime     LoadedKernelModule   nvidia_uvm                  1         Live

Finally, the /proc/driver/nvidia/version file only exists if the nvidia module is loaded, and it gives the exact driver version:

1

talosctl -n 192.168.1.123 read /proc/driver/nvidia/version

1
2

NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  595.71.05  Release Build  (dvs-builder@U22-I3-G08-03-1)  Fri Apr 24 06:42:30 UTC 2026
GCC version:  Selected multilib: .;@m64

The Open Kernel Module mention confirms that the open source drivers are the ones running.

The driver is loaded in the VM, the last step is to make it known to Kubernetes.

Step 4: exposing the GPU to Kubernetes

Modules loaded on the node are not enough. The Kubernetes scheduler only knows about CPU and memory, and for it to allocate a GPU to a pod, an nvidia.com/gpu extended resource has to be published on the node by a device plugin. A device plugin is a component running on the node that declares the available GPUs and their state to the kubelet, then chooses which card to assign to each pod that requests one.

NVIDIA GPU Operator

To do this, I used the NVIDIA GPU Operator rather than the NVIDIA device plugin alone, for two reasons: it is the method described in the Talos documentation, and the operator also deploys the DCGM exporter, which lets me collect the GPU metrics in Prometheus.

By default, the operator installs the drivers and the container toolkit on the nodes itself, but on Talos these two components are already provided by the system extensions. They must therefore be disabled to avoid conflicts, as the Talos documentation states:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

kubectl create namespace gpu-operator
kubectl label --overwrite namespace gpu-operator pod-security.kubernetes.io/enforce=privileged

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version v26.3.3 \
  --set driver.enabled=false \
  --set toolkit.enabled=false \
  --set hostPaths.driverInstallDir=/usr/local \
  --set dcgmExporter.serviceMonitor.enabled=true

• driver.enabled=false and toolkit.enabled=false: the drivers and the toolkit come from the Talos extensions.
• hostPaths.driverInstallDir=/usr/local: the Talos extension installs the driver files in /usr/local, while the operator looks for them in /run/nvidia/driver by default.
• dcgmExporter.serviceMonitor.enabled=true: creates the ServiceMonitor that lets Prometheus collect the GPU metrics.
• The namespace must allow privileged pods, because the operator’s components access the node’s hardware.

📝 Note

The Talos documentation requires a GPU Operator version greater than or equal to v26.3.1.

I did not need a nodeSelector to target the GPU node. The operator first deploys Node Feature Discovery on all nodes, which detects the PCI devices of each one. The operator then adds the nvidia.com/gpu.present=true label to the nodes that have an NVIDIA card, and only deploys its components (device plugin, DCGM exporter, validation) on those nodes:

1

kubectl get pods -n gpu-operator -o wide

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

NAME                                                         READY   STATUS      RESTARTS         AGE     IP             NODE                    NOMINATED NODE   READINESS GATES
gpu-feature-discovery-9ll6h                                  1/1     Running     0                52d     10.244.0.134   talos-gpu-01            <none>           <none>
gpu-operator-57d75775c8-ls7hf                                1/1     Running     0                4d12h   10.244.2.150   talos-controlplane-01   <none>           <none>
gpu-operator-node-feature-discovery-gc-847bb8f7b6-9gmkf      1/1     Running     0                4d12h   10.244.0.231   talos-gpu-01            <none>           <none>
gpu-operator-node-feature-discovery-master-d98f944cd-vhzdw   1/1     Running     0                4d12h   10.244.2.188   talos-controlplane-01   <none>           <none>
gpu-operator-node-feature-discovery-worker-kvvfl             1/1     Running     0                52d     10.244.0.249   talos-gpu-01            <none>           <none>
gpu-operator-node-feature-discovery-worker-ntvzt             1/1     Running     0                52d     10.244.2.39    talos-controlplane-01   <none>           <none>
gpu-operator-node-feature-discovery-worker-s95fk             1/1     Running     0                52d     10.244.1.208   talos-worker-01         <none>           <none>
nvidia-cuda-validator-gfd6h                                  0/1     Completed   0                6m42s   10.244.0.109   talos-gpu-01            <none>           <none>
nvidia-dcgm-exporter-rkt88                                   1/1     Running     0                52d     10.244.0.161   talos-gpu-01            <none>           <none>
nvidia-device-plugin-daemonset-pgnqw                         1/1     Running     0                52d     10.244.0.164   talos-gpu-01            <none>           <none>
nvidia-operator-validator-xqdd8                              1/1     Running     0                52d     10.244.0.50    talos-gpu-01            <none>           <none>

Only the NFD workers run on all three nodes, all the NVIDIA components are on talos-gpu-01. The nvidia-cuda-validator pod in Completed is expected: it runs a test CUDA program on the card, then stops.

Checking the allocation

The nvidia.com/gpu resource must appear in the node’s capacity:

1

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, gpu: .status.capacity."nvidia.com/gpu"}'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "name": "talos-controlplane-01",
  "gpu": null
}
{
  "name": "talos-gpu-01",
  "gpu": "1"
}
{
  "name": "talos-worker-01",
  "gpu": null
}

The talos-gpu-01 node exposes one GPU, the other two nodes do not have the resource. The labels added to the node by GPU Feature Discovery also give the details of the card as seen by Kubernetes:

1

kubectl get node talos-gpu-01 --show-labels | tr ',' '\n' | grep nvidia.com

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

...
nvidia.com/cuda.driver-version.full=595.71.05
...
nvidia.com/cuda.runtime-version.full=13.2
...
nvidia.com/gpu.count=1
...
nvidia.com/gpu.family=ampere
...
nvidia.com/gpu.memory=8192
...
nvidia.com/gpu.product=NVIDIA-GeForce-RTX-3050
...

The nvidia.com/cuda.runtime-version.full=13.2 label gives the highest CUDA version supported by the driver, which is used to choose the image of the test pod.

To validate the whole chain, I run a test pod that executes nvidia-smi (NVIDIA System Management Interface), the command line tool shipped with the driver that shows the visible cards, the driver version, memory usage and running processes. If the command sees the card from inside a container, the driver, the runtime and the allocation all work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
spec:
  restartPolicy: Never
  containers:
    - name: nvidia-smi
      image: nvidia/cuda:13.2.1-base-ubuntu24.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

The nvidia.com/gpu: 1 field in the limits is enough: the pod requests the resource published by the device plugin, so the scheduler can only place it on talos-gpu-01, and the GPU is reserved for it as long as it runs.

The Talos documentation also adds runtimeClassName: nvidia to its test pod, to go through the NVIDIA runtime that the nvidia-container-toolkit extension declares in containerd. On my cluster, this is not needed, because CDI (Container Device Interface) is enabled in the GPU Operator:

1

kubectl get clusterpolicy -o jsonpath='{.items[0].spec.cdi}'

1

{"default":false,"enabled":true,"nriPluginEnabled":false}

With CDI, the device plugin tells containerd which device files and which driver libraries to add to the container, and containerd injects them itself with its default runtime. This is also how the nvidia-smi binary, absent from the nvidia/cuda image, ends up in the container. According to the NVIDIA documentation on CDI, this is the default behaviour since version v25.10.0 of the GPU Operator, and runtimeClassName: nvidia is only still needed for containers that access GPUs through the NVIDIA_VISIBLE_DEVICES environment variable without going through the device plugin, such as monitoring agents.

Once the pod has finished, its logs contain the output of nvidia-smi:

1

kubectl logs nvidia-smi

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05              Driver Version: 595.71.05      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050        On  |   00000000:01:00.0 Off |                  N/A |
|  0%   45C    P8              7W /  130W |       1MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

The container sees the RTX 3050 with its 8 GB of memory, the 595.71.05 driver provided by the Talos extension and CUDA 13.2. The whole chain works, from vfio-pci on the Proxmox host all the way to the pod.

Conclusion

The GPU can now be allocated by the scheduler: the cluster exposes an nvidia.com/gpu resource on the talos-gpu-01 node, and a pod can request it just like it would request CPU or memory.

There is only one nvidia.com/gpu resource in the whole cluster, which means only one pod at a time can use the card. To share it between several pods, the GPU Operator offers time-slicing, where pods take turns on the card, and NVIDIA also offers MPS (Multi-Process Service), which runs the CUDA processes of several pods on the card at the same time. These are options I have not set up yet.

For my part, this stage taught me a lot about hardware. My knowledge of virtualisation had so far stayed at a fairly high level, that of the hypervisor, and I did not know the low-level mechanisms that passthrough relies on, such as device I/O, DMA, the MMU or the IOMMU.

At this point, the infrastructure is fully described in OpenTofu, but everything running in the cluster was installed with helm install commands run by hand. It is the same configuration drift problem that Talos solves at the OS level, showing up one floor higher. Part 3 therefore sets up GitOps with Flux before going further towards inference.

Useful links

• PCI Passthrough - Proxmox wiki
• PCI(e) Passthrough - Proxmox documentation
• NVIDIA GPU on Talos (OSS drivers)
• NVIDIA GPU Operator
• proxmox_hardware_mapping_pci