Introduction
GPU Passthrough is a feature that allows physical graphics cards to be directly assigned to virtual machines. This allows the virtual machine to have direct access to the GPU hardware, providing a more seamless experience for graphics-intensive applications and games.
Hardware
Software
Proxmox Version 8.0.4
VM: Ubuntu Desktop 22.04
Enable IOMMU in BIOS
Shut down your server
Connect a Display and Keyboard
Power on your server
Enter BIOS by pressing the DEL key (It may be a different key if using a different motherboard)
Go to Advanced -> AMD CBS -> Set IOMMU to Enabled. Auto may not work properly (For other motherboards, please refer to the manual to find this setting)
Save and reset
Verify that IOMMU is Enabled by running the following command on your Proxmox host:
dmesg | grep -e DMAR -e IOMMU
There should be an output saying that it is enabled
[ 0.428988] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.430222] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.430381] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
If there is no output then something is wrong
Proxmox Configuration for GPU Passthrough
Find the IDs of the devices connected to the GPU you wish to Passthrough
Find the PCI address of the GPU by running the following command on the Proxmox host.
lspci -nnv | grep VGA
For me this results in:
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c1) (prog-if 00 [VGA controller])
Once you have found the GPU that you wish to passthrough, run the following command but replace "0b:00" with your GPU
lspci -s 0b:00
This will list all connected devices to your GPU. Usually, it would contain a VGA Device and Audio Device but your GPU may have others as well.
0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1)
0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
Now we can find the IDs of the devices by running the following command. Again, please be sure to replace it with your GPU address
lspci -s 0b:00 -n
0b:00.0 0300: 1002:731f (rev c1)
0b:00.1 0403: 1002:ab38
Keep note of the "1002:731f" and "1002:ab38". We will use those IDs to split the PCI Group into separate devices.
echo "options vfio-pci ids=1002:731f,1002:ab38 disable_vga=1"> /etc/modprobe.d/vfio.conf
Edit the bootloader configuration for our Proxmox Host
GRUB
Edit /etc/default/grub and modify the appropriate line:
For Intel CPUs
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
For AMD CPUs
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
Save and close the file then run the following command to apply the changes
update-grub2
ZFS
Edit /etc/kernel/cmdline and modify the appropriate line:
Save and close the file then run the following command to apply the changes
pve-efiboot-tool refresh
Edit Modules
Edit the file /etc/modules and add the following lines:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Save and close the file then run the following command to add the kernel modules
update-initramfs -u -k all
Now you can reboot the Proxmox Host
reboot
Now your Proxmox host should be ready for GPU passthrough!
VM Configuration for GPU Passthrough
Create or edit the VM with the following attributes
BIOS: OVMF(UEFI)
Machine: q35
Note: If you did not create the VM with OVMF then you may need to add an EFI disk since it will complain about missing an EFI disk.
Add the PCI Device and make sure that All Functions and PCI-Express is enabled
Now start your VM, and install the OS if you haven't already. I am installing Ubuntu but you can install Windows, etc.
If you received the following error when starting your VM, make sure IOMMU is enabled
kvm: ../hw/pci/pci.c:1613: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
TASK ERROR: start failed: QEMU exited with code 1
Check that the GPU is detected in the VM
lspci -nnv | grep VGA
You should now see the GPU listed!
00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) (prog-if 00 [VGA controller])
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c1) (prog-if 00 [VGA controller])
Update your VM and install the GPU drivers
sudo apt update
sudo apt upgrade -y
Install AMD GPU Drivers for Ubuntu
Download the appropriate DEB file from AMD Graphics Drivers & Support
Run the following commands to install the driver (Replace the filename with the one you have downloaded)
sudo apt install -f ./amdgpu-install_5.7.50700-1_all.deb
amdgpu-install
If you get the following error
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
amdgpu-lib32 : Depends: libdrm2-amdgpu:i386 but it is not installable
Depends: libdrm-amdgpu-amdgpu1:i386 but it is not installable
Depends: libllvm-amdgpu:i386 but it is not installable
Depends: libwayland-amdgpu-client0:i386 but it is not installable or
libwayland-client0:i386 but it is not installable
Depends: libwayland-amdgpu-server0:i386 but it is not installable or
libwayland-server0:i386 but it is not installable
Depends: libwayland-amdgpu-egl1:i386 but it is not installable or
libwayland-egl1:i386 but it is not installable
Depends: libxatracker2-amdgpu:i386 but it is not installable
Depends: libgbm1-amdgpu:i386 but it is not installable
Depends: libegl1-amdgpu-mesa:i386 but it is not installable
Depends: libegl1-amdgpu-mesa-drivers:i386 but it is not installable
Depends: libglapi-amdgpu-mesa:i386 but it is not installable
Depends: libgl1-amdgpu-mesa-glx:i386 but it is not installable
Depends: libgl1-amdgpu-mesa-dri:i386 but it is not installable
Depends: mesa-amdgpu-va-drivers:i386 but it is not installable
Depends: mesa-amdgpu-vdpau-drivers:i386 but it is not installable
E: Unable to correct problems, you have held broken packages.
Try installing the i386 architecture and try again
sudo dpkg --add-architecture i386
Conclusion
You have now successfully passed through your GPU to your Proxmox VM. If you liked this guide, please check out our other homelab guides.
If you have any questions or suggestions, please feel free to leave a comment and I will try to answer them!
