Proxmox GPU Passthrough to VM

Introduction

GPU Passthrough is a feature that allows physical graphics cards to be directly assigned to virtual machines. This allows the virtual machine to have direct access to the GPU hardware, providing a more seamless experience for graphics-intensive applications and games.

Hardware

Motherboard: ASUS TUF Gaming x570
CPU: Ryzen 7 2700x
Memory: 32GB DDR4 3000
GPU: AMD Radeon RX 5700 XT

Software

Proxmox Version 8.0.4
VM: Ubuntu Desktop 22.04

Enable IOMMU in BIOS

Shut down your server
Connect a Display and Keyboard
Power on your server
Enter BIOS by pressing the DEL key (It may be a different key if using a different motherboard)
Go to Advanced -> AMD CBS -> Set IOMMU to Enabled. Auto may not work properly (For other motherboards, please refer to the manual to find this setting)
Save and reset

Verify that IOMMU is Enabled by running the following command on your Proxmox host:

dmesg | grep -e DMAR -e IOMMU

There should be an output saying that it is enabled

[    0.428988] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.430222] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.430381] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

If there is no output then something is wrong

Proxmox Configuration for GPU Passthrough

Find the IDs of the devices connected to the GPU you wish to Passthrough

Find the PCI address of the GPU by running the following command on the Proxmox host.

lspci -nnv | grep VGA

For me this results in:

0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c1) (prog-if 00 [VGA controller])

Once you have found the GPU that you wish to passthrough, run the following command but replace "0b:00" with your GPU

lspci -s 0b:00

This will list all connected devices to your GPU. Usually, it would contain a VGA Device and Audio Device but your GPU may have others as well.

0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1)
0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio

Now we can find the IDs of the devices by running the following command. Again, please be sure to replace it with your GPU address

lspci -s 0b:00 -n

0b:00.0 0300: 1002:731f (rev c1)
0b:00.1 0403: 1002:ab38

Keep note of the "1002:731f" and "1002:ab38". We will use those IDs to split the PCI Group into separate devices.

echo "options vfio-pci ids=1002:731f,1002:ab38 disable_vga=1"> /etc/modprobe.d/vfio.conf

Edit the bootloader configuration for our Proxmox Host

GRUB

Edit /etc/default/grub and modify the appropriate line:

For Intel CPUs

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

For AMD CPUs

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Save and close the file then run the following command to apply the changes

update-grub2

ZFS

Edit /etc/kernel/cmdline and modify the appropriate line:

Intel: root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt
AMD: root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt

Save and close the file then run the following command to apply the changes

pve-efiboot-tool refresh

Edit Modules

Edit the file /etc/modules and add the following lines:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Save and close the file then run the following command to add the kernel modules

update-initramfs -u -k all

Now you can reboot the Proxmox Host

reboot

Now your Proxmox host should be ready for GPU passthrough!

VM Configuration for GPU Passthrough

Create or edit the VM with the following attributes

 BIOS: OVMF(UEFI)
 Machine: q35

Note: If you did not create the VM with OVMF then you may need to add an EFI disk since it will complain about missing an EFI disk.

Add the PCI Device and make sure that All Functions and PCI-Express is enabled

Proxmox GPU Passthrough to VM - Add PCI Device

Now start your VM, and install the OS if you haven't already. I am installing Ubuntu but you can install Windows, etc.

If you received the following error when starting your VM, make sure IOMMU is enabled


kvm: ../hw/pci/pci.c:1613: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
TASK ERROR: start failed: QEMU exited with code 1

Check that the GPU is detected in the VM

lspci -nnv | grep VGA

You should now see the GPU listed!

00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) (prog-if 00 [VGA controller])
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c1) (prog-if 00 [VGA controller])

Update your VM and install the GPU drivers

sudo apt update
sudo apt upgrade -y

Install AMD GPU Drivers for Ubuntu

Download the appropriate DEB file from AMD Graphics Drivers & Support

Run the following commands to install the driver (Replace the filename with the one you have downloaded)

sudo apt install -f ./amdgpu-install_5.7.50700-1_all.deb
amdgpu-install

If you get the following error

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 amdgpu-lib32 : Depends: libdrm2-amdgpu:i386 but it is not installable
                Depends: libdrm-amdgpu-amdgpu1:i386 but it is not installable
                Depends: libllvm-amdgpu:i386 but it is not installable
                Depends: libwayland-amdgpu-client0:i386 but it is not installable or
                         libwayland-client0:i386 but it is not installable
                Depends: libwayland-amdgpu-server0:i386 but it is not installable or
                         libwayland-server0:i386 but it is not installable
                Depends: libwayland-amdgpu-egl1:i386 but it is not installable or
                         libwayland-egl1:i386 but it is not installable
                Depends: libxatracker2-amdgpu:i386 but it is not installable
                Depends: libgbm1-amdgpu:i386 but it is not installable
                Depends: libegl1-amdgpu-mesa:i386 but it is not installable
                Depends: libegl1-amdgpu-mesa-drivers:i386 but it is not installable
                Depends: libglapi-amdgpu-mesa:i386 but it is not installable
                Depends: libgl1-amdgpu-mesa-glx:i386 but it is not installable
                Depends: libgl1-amdgpu-mesa-dri:i386 but it is not installable
                Depends: mesa-amdgpu-va-drivers:i386 but it is not installable
                Depends: mesa-amdgpu-vdpau-drivers:i386 but it is not installable
E: Unable to correct problems, you have held broken packages.

Try installing the i386 architecture and try again

sudo dpkg --add-architecture i386

Conclusion

You have now successfully passed through your GPU to your Proxmox VM. If you liked this guide, please check out our other homelab guides.

If you have any questions or suggestions, please feel free to leave a comment and I will try to answer them!