It’s been a few years since I made my post about getting Plex GPU transcoding to work in Docker inside an LXC container running on Proxmox. It’s gotten quite a few updates over the years based on feedbacks and own observations, which fixes different problems and optimized several aspects.
I recently upgraded my Proxmox-host to the latest Debian Bookworm 12.10 (Proxmox 8.4.1), and also wanted to update my NVIDIA-drivers from 550.100 to the latest 570.144.
After the upgrade everything seemed to be fine, but Plex transcoding would not work. All my usual previous fixes did not solve the problem. I decided to “start from scratch”, and see if I could a) get it to work, b) simplify the setup, and c) improve where possible.
My previous guide might still work for you, so feel free to use that one if that suits your setup better. Note, however, that future updates/improvements will probably only be done on this new version of the guide.
As last time, I’ll assume you’ve got Proxmox and LXC set up, ready to go. As of writing this, the Proxmox version was 8.4.1 and LXC was running Debian 12.10 Bookworm. Keep in mind that my host has secure boot disabled in the BIOS, which is a common source of problems when dealing with this setup. There are ways of getting this setup working when using secure boot, but that’s outside the scope of this guide.
In my example I’ll be running a LXC container named docker1 (ID 101) on my Proxmox host. Everything will be headless (i.e. no X involved). The LXC will be privileged with fuse=1,nesting=1 set as features. I’ll use a Nvidia RTX A2000 as the GPU. All commands will be run as root. Note that there might be other steps that needs to be done if you attempt to run this in a rootless/unprivileged LXC container (see here for more information).
The referenced commands in this guide can for the most part be copy-pasted. Some of the steps are interactive and/or requires you to do small changes on your own.
If you followed my previous v1 guide, it’s recommended to “revert” that setup (i.e. do a cleanup), so that you start with a clean slate. See here for more info.
As a sidenote; NVIDIA drivers now includes two kernel module options; the previous proprietary version, and a new open-source alternative. While the open-source version is considered somewhat stable, it still doesn’t have 100% feature-parity with the proprietary kernel module. For this guide, we’ll use the proprietary version.
Proxmox host
First step is to install the NVIDIA driver and relevant dependencies. It’s based on NVIDIA’s own installation guide for Debian, with notes from the headless-section, with some slight modifications.
# modify the apt repo url by adding "contrib non-free non-free-firmware"
# /etc/apt/sources.list should contain something like this
deb http://ftp.no.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://ftp.no.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb http://ftp.no.debian.org/debian bookworm-security main contrib non-free non-free-firmware
# update + install headers
apt update
apt install proxmox-headers-$(uname -r)
# download nvidia keyring and install it
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
# install nvidia drivers
# these are the compute-only (headless) versions of the drivers
# this makes sure we don't install any unnecessary packages (X drivers, etc)
apt install nvidia-driver-cuda nvidia-kernel-dkms
# rebootAfter the reboot, the driver should be installed and working.
root@foobar:~# nvidia-smi
Sun Apr 20 07:37:23 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A2000 On | 00000000:82:00.0 Off | Off |
| 30% 36C P8 5W / 70W | 1MiB / 6138MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@foobar:~# systemctl status nvidia-persistenced.service
● nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; enabled; preset: enabled)
Active: active (running) since Sun 2025-04-20 07:36:51 CEST; 38s ago
Process: 2619 ExecStart=/usr/bin/nvidia-persistenced --verbose (code=exited, status=0/SUCCESS)
Main PID: 2623 (nvidia-persiste)
Tasks: 1 (limit: 154491)
Memory: 752.0K
CPU: 16ms
CGroup: /system.slice/nvidia-persistenced.service
└─2623 /usr/bin/nvidia-persistenced --verbose
Apr 20 07:36:51 foobar systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
Apr 20 07:36:51 foobar nvidia-persistenced[2623]: Verbose syslog connection opened
Apr 20 07:36:51 foobar nvidia-persistenced[2623]: Started (2623)
Apr 20 07:36:51 foobar nvidia-persistenced[2623]: device 0000:82:00.0 - registered
Apr 20 07:36:51 foobar nvidia-persistenced[2623]: device 0000:82:00.0 - persistence mode enabled.
Apr 20 07:36:51 foobar nvidia-persistenced[2623]: device 0000:82:00.0 - NUMA memory onlined.
Apr 20 07:36:51 foobar nvidia-persistenced[2623]: Local RPC services initialized
Apr 20 07:36:51 foobar systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon.
root@foobar:~# ls -alh /dev/nvidia* /dev/dri
crw-rw-rw- 1 root root 195, 0 Apr 20 07:36 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Apr 20 07:36 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Apr 20 07:36 /dev/nvidia-modeset
crw-rw-rw- 1 root root 234, 0 Apr 20 07:37 /dev/nvidia-uvm
crw-rw-rw- 1 root root 234, 1 Apr 20 07:37 /dev/nvidia-uvm-tools
/dev/dri:
total 0
drwxr-xr-x 3 root root 120 Apr 20 07:36 .
drwxr-xr-x 19 root root 5.0K Apr 20 07:37 ..
drwxr-xr-x 2 root root 100 Apr 20 07:36 by-path
crw-rw---- 1 root video 226, 0 Apr 20 07:36 card0
crw-rw---- 1 root video 226, 1 Apr 20 07:36 card1
crw-rw---- 1 root render 226, 128 Apr 20 07:36 renderD128
/dev/nvidia-caps:
total 0
drwxr-xr-x 2 root root 80 Apr 20 07:37 .
drwxr-xr-x 19 root root 5.0K Apr 20 07:37 ..
cr-------- 1 root root 237, 1 Apr 20 07:37 nvidia-cap1
cr--r--r-- 1 root root 237, 2 Apr 20 07:37 nvidia-cap2If the correct GPU shows from nvidia-smi, the persistence service runs fine, and you have at least five files under /dev/nvidia* are available, we’re ready to proceed to the LXC container.
The number of files depend on your setup; if you don’t have any /dev/nvidia-caps folder, you should be fine by adding only the five files listed above. If you also happen to have the /dev/nvidia-caps folder, you should add the two (or more) files within that as well. See here for more info.
Note that the files under /dev/dri is strictly not needed for transcoding, but would be needed for other things like rendering or display applications like VirtualGL. If you’re using Intel or AMD GPUs, this would also be needed. We’re adding them in this guide for completeness.
LXC container
We need to add relevant LXC configuration to our container. Shut down the LXC container, and make the following changes to the LXC configuration file.
# edit /etc/pve/lxc/101.conf and add the following
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 234:* rwm
lxc.cgroup2.devices.allow: c 237:* rwm
lxc.cgroup2.devices.allow: c 238:* rwm
lxc.cgroup2.devices.allow: c 239:* rwm
lxc.cgroup2.devices.allow: c 240:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps/nvidia-cap1 dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps/nvidia-cap2 dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=fileThe numbers on the cgroup2-lines are from the fifth column in the device-lists above. Using the examples above, we would add 195, 226, 234 and 237 as the cgroup-values. These values might differ from your setup, and can also change when you update drivers. You can see that I’ve added more values in my example, as my setup has alternated between these. LXC does not complain if you configure numbers that doesn’t exist, so you can add any new value whenever they change.
We can now turn on the LXC container, and we’ll be ready to install the Nvidia driver.
# modify the apt repo url by adding "contrib non-free non-free-firmware"
# /etc/apt/sources.list should contain something like this
deb http://ftp.no.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://ftp.no.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb http://ftp.no.debian.org/debian bookworm-security main contrib non-free non-free-firmware
# download nvidia keyring and install it
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
# install nvidia drivers
# these are the compute-only (headless) versions of the drivers
# this makes sure we don't install any unnecessary packages (X drivers, etc)
apt install nvidia-driver-cudaThe installation above will install kernel modules and other non-required files. Attempting to remove these, will also remove the driver. We’ll workaround this by doing some small modifications. I have yet found a way to do this in a different manner (i.e. where only the required files are installed). If you find a way, please let me know.
# disable + mask persistence service
systemctl stop nvidia-persistenced.service
systemctl disable nvidia-persistenced.service
systemctl mask nvidia-persistenced.service
# remove kernel config
echo "" > /etc/modprobe.d/nvidia.conf
echo "" > /etc/modprobe.d/nvidia-modeset.conf
# block kernel modules
echo -e "blacklist nvidia\nblacklist nvidia_drm\nblacklist nvidia_modeset\nblacklist nvidia_uvm" > /etc/modprobe.d/blacklist-nvidia.confAfter a reboot, we should see the files we mounted & that the driver works as expected.
root@docker1:~# nvidia-smi
Sun Apr 20 08:26:10 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A2000 Off | 00000000:82:00.0 Off | Off |
| 30% 29C P8 5W / 70W | 1MiB / 6138MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@docker1:~# ls -alh /dev/nvidia* /dev/dri
crw-rw-rw- 1 root root 195, 0 Apr 20 07:36 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Apr 20 07:36 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Apr 20 07:36 /dev/nvidia-modeset
crw-rw-rw- 1 root root 234, 0 Apr 20 07:37 /dev/nvidia-uvm
crw-rw-rw- 1 root root 234, 1 Apr 20 07:37 /dev/nvidia-uvm-tools
/dev/dri:
total 0
drwxr-xr-x 2 root root 100 Apr 20 09:09 .
drwxr-xr-x 9 root root 660 Apr 20 09:09 ..
crw-rw---- 1 root video 226, 0 Apr 20 07:36 card0
crw-rw---- 1 root video 226, 1 Apr 20 07:36 card1
crw-rw---- 1 root input 226, 128 Apr 20 07:36 renderD128
/dev/nvidia-caps:
total 0
drwxr-xr-x 2 root root 80 Apr 20 09:09 .
drwxr-xr-x 9 root root 660 Apr 20 09:09 ..
cr-------- 1 root root 237, 1 Apr 20 07:37 nvidia-cap1
cr--r--r-- 1 root root 237, 2 Apr 20 07:37 nvidia-cap2Docker container
Now we can address the Docker configuration/setup. If you didn’t purge/cleanup these aspects from my v1 guide, and you’re “converting” your system from v1 to v2, you should be able to skip to the next section (testing). If you’re doing a clean install, you need to do these next steps as well.
We’ll be using docker-compose, and we’ll also make sure to have the latest version by removing the Debian-provided docker and docker-compose. We’ll also install the Nvidia-provided Docker runtime. Both these are relevant in terms of making the GPU available within Docker.
# remove debian-provided packages
apt remove docker-compose docker docker.io containerd runc
# install docker from official repository
apt update
apt install ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install docker-ce docker-ce-cli containerd.io
# install docker-compose
curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
# install docker-compose bash completion
curl \
-L https://raw.githubusercontent.com/docker/cli/master/contrib/completion/bash/docker \
-o /etc/bash_completion.d/docker-compose
# install NVIDIA Container Toolkit
apt install -y curl
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install nvidia-container-toolkit
# make sure that docker is configured
# this will modify your existing /etc/docker/daemon.json by adding relevant config
nvidia-ctk runtime configure --runtime=docker
# restart systemd + docker (if you don't reload systemd, it might not work)
systemctl daemon-reload
systemctl restart dockerWe should now be able to run Docker containers with GPU support. Let’s test it.
# nvidia/cuda doesn't support the "latest" tag. they also remove old releases,
# so we need to find the latest one. you can either run the oneliner below,
# or you can find the latest "base-ubuntu" tag manually on this page:
# https://hub.docker.com/r/nvidia/cuda/tags
root@docker1:~# latest_tag="`curl -s https://gitlab.com/nvidia/container-images/cuda/raw/master/doc/supported-tags.md | grep -i "base-ubuntu" | head -1 | perl -wple 's/.+\`(.+?)\`.+/$1/'`"
root@docker1:~# echo $latest_tag
12.8.1-base-ubuntu24.04
root@docker1:~# docker run --rm --gpus all nvidia/cuda:${latest_tag} nvidia-smi
Sun Apr 20 06:31:53 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A2000 Off | 00000000:82:00.0 Off | Off |
| 30% 29C P8 5W / 70W | 1MiB / 6138MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@docker1:~# docker run --rm -it --gpus all --runtime=nvidia linuxserver/ffmpeg -hwaccel nvdec -f lavfi -i testsrc2=duration=300:size=1280x720:rate=90 -c:v hevc_nvenc -qp 18 nvidia-hevc_nvec-90fps-300s.mp4
ffmpeg version 7.1.1 Copyright (c) 2000-2025 the FFmpeg developers
built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
configuration: --disable-debug --disable-doc --disable-ffplay --enable-alsa --enable-cuda-llvm --enable-cuvid --enable-ffprobe --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libfdk_aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libkvazaar --enable-liblc3 --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libplacebo --enable-librav1e --enable-librist --enable-libshaderc --enable-libsrt --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nonfree --enable-nvdec --enable-nvenc --enable-opencl --enable-openssl --enable-stripping --enable-vaapi --enable-vdpau --enable-version3 --enable-vulkan
libavutil 59. 39.100 / 59. 39.100
libavcodec 61. 19.101 / 61. 19.101
libavformat 61. 7.100 / 61. 7.100
libavdevice 61. 3.100 / 61. 3.100
libavfilter 10. 4.100 / 10. 4.100
libswscale 8. 3.100 / 8. 3.100
libswresample 5. 3.100 / 5. 3.100
libpostproc 58. 3.100 / 58. 3.100
Input #0, lavfi, from 'testsrc2=duration=300:size=1280x720:rate=90':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: wrapped_avframe, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 90 fps, 90 tbr, 90 tbn
Stream mapping:
Stream #0:0 -> #0:0 (wrapped_avframe (native) -> hevc (hevc_nvenc))
Press [q] to stop, [?] for help
Output #0, mp4, to 'nvidia-hevc_nvec-90fps-300s.mp4':
Metadata:
encoder : Lavf61.7.100
Stream #0:0: Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 2000 kb/s, 90 fps, 11520 tbn
Metadata:
encoder : Lavc61.19.101 hevc_nvenc
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
[out#0/mp4 @ 0x5cd43105b900] video:569284KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.056810%
frame=27000 fps=720 q=17.0 Lsize= 569607KiB time=00:04:59.96 bitrate=15555.8kbits/s speed= 8x
# while the above is running, you should see the process being run on the GPU
root@foobar:~# nvidia-smi
Sun Apr 20 08:45:04 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A2000 On | 00000000:82:00.0 Off | Off |
| 30% 39C P0 40W / 70W | 162MiB / 6138MiB | 15% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 124937 C /usr/local/bin/ffmpeg 152MiB |
+-----------------------------------------------------------------------------------------+Yay! It’s working!
Let’s add the final pieces together for a fully working Plex docker-compose.yml:
services:
plex:
container_name: plex
hostname: plex
image: linuxserver/plex:latest
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
count: all
environment:
TZ: Europe/Paris
PUID: 0
PGID: 0
VERSION: latest
NVIDIA_VISIBLE_DEVICES: all
NVIDIA_DRIVER_CAPABILITIES: compute,video,utility
network_mode: host
volumes:
- /srv/config/plex:/config
- /storage/media:/data/media
- type: tmpfs
target: /tmp
tmpfs:
size: 8GAnd it’s working! Woho!
root@foobar:~# nvidia-smi
Sun Apr 20 11:08:07 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A2000 On | 00000000:82:00.0 Off | Off |
| 30% 52C P0 40W / 70W | 1078MiB / 6138MiB | 9% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 272015 C ...exmediaserver/Plex Transcoder 516MiB |
| 0 N/A N/A 272569 C ...exmediaserver/Plex Transcoder 546MiB |
+-----------------------------------------------------------------------------------------+Main differences from my original v1 guide
I’ll quickly try to explain the main differences between this and my original v1 guide:
- We use the Debian packages provided by NVIDIA, rather than manually installed drivers. This is the recommended setup from NVIDIA. The downside is that you can’t as-easily cherry-pick a specific driver version, and you won’t always get the latest driver immediately (i.e. when I wrote this guide, driver version 570.144 was released three days ago, but only 570.133 was available via the NVIDIA repository). For our usecase (Plex transcoding) I think that’s perfectly reasonable.
- We don’t have to manually download & install drivers. This will be automatically updated on host+LXC container when doing
apt update+apt upgrade. Yes, we need to upgrade on both the host and within the LXC container at the same time, but that’s completely manageable (and was required even when installing manually anyways). When I wrote my previous guide I could not get this setup working when using distro Debian-packages inside the LXC container (due to multiple errors). By using the packages provided by NVIDIA (rather than from the distro), this now seems to work. - We don’t have to manually fiddle with kernel modules, blocklists, udev-rules, nvidia-persistenced.service, etc. All of these things are handled by the packages automatically.
- Some extra steps needed in the LXC container to avoid potential issues, but should be a one-time configuration.
- Use
tmpfsfor transcodes to make them quicker/snappier. This can of course also be implemented in the v1 setup. - I’m pretty sure I would get my previous setup working if I had tested the
count: allfix, but ultimately I’m glad that I changed the setup/aproach, as it’s simpler/easier to upgrade in the future. If you prefer to have full manual control, you could probably use my v1 guide still.
Upgrading
Upgrading should be a simple apt update followed by apt upgrade on both the Proxmox host and within the LXC container. The below commands should be sufficient, but since I have not yet done this, I can’t yet confirm. Will update this section if there are any changes when I do the next upgrade.
Proxmox host
# upgrade packages
apt update
apt upgrade
# rebootLXC container
# upgrade packages
apt update
apt upgrade
# update docker compose
curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
# update docker bash completion
curl \
-L https://raw.githubusercontent.com/docker/cli/master/contrib/completion/bash/docker \
-o /etc/bash_completion.d/docker-compose
# rebootCleanup from v1 of the guide
If you followed my previous v1 guide/setup, you should “clean up” before attempting to follow this updated guide. The steps below was what I needed to get a clean slate. They might not be suitable to your setup, so consider these as a reference, not a “copy-paste” list of commands.
LXC container
We clean up the LXC container first, since things might behave strangely if we remove stuff on the host first, as the LXC container might be dependent on things from the host.
# uninstall old manually installed driver
# select "no" when asked about xconfig-restore
./NVIDIA-Linux-x86_64-550.100.run --uninstall
# remove nvidia packages
# WARNING: if you *only* have nvidia-container* libnvidia-container* and nvidia-docker2
# you can probably skip this (and avoid having to re-install this)
apt remove --purge -y xserver-xorg-*
apt remove --purge -y glx-alternative-nvidia
apt remove --purge -y nvidia*
apt remove --purge -y cuda*
apt remove --purge -y cudnn*
apt remove --purge -y libnvidia*
apt remove --purge -y libcuda*
apt remove --purge -y libcudnn*
dpkg --list | awk '{print $2}' | grep -E "(xserver-xorg-|glx-alternative-nvidia|nvidia|cuda|cudnn|libnvidia|libcuda|libcudnn)" | xargs dpkg --purgeProxmox host
# remove nvidia kernel modules from /etc/modules-load.d/modules.conf
sed -i'' '/nvidia/d' /etc/modules-load.d/modules.conf
# remove kernel module blacklist of nouveau (as nvidia-packages deals with that)
rm /etc/modprobe.d/blacklist-nouveau.conf
# remove udev-rules
rm /etc/udev/rules.d/70-nvidia.rules
# uninstall nvidia persistence service
cp /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2 .
bunzip2 nvidia-persistenced-init.tar.bz2
tar -xf nvidia-persistenced-init.tar
chmod +x nvidia-persistenced-init/install.sh
./nvidia-persistenced-init/install.sh -r
rm -rf nvidia-persistenced-init*
# uninstall driver
./NVIDIA-Linux-x86_64-550.100.run --uninstall
# remove nvidia packages
apt remove --purge -y xserver-xorg-*
apt remove --purge -y glx-alternative-nvidia
apt remove --purge -y nvidia*
apt remove --purge -y cuda*
apt remove --purge -y cudnn*
apt remove --purge -y libnvidia*
apt remove --purge -y libcuda*
apt remove --purge -y libcudnn*
dpkg --list | awk '{print $2}' | grep -E "(xserver-xorg-|glx-alternative-nvidia|nvidia|cuda|cudnn|libnvidia|libcuda|libcudnn)" | xargs dpkg --purge
# update initramfs & reboot
update-initramfs -u
rebootProblems encountered
Any problems encountered during the setup will be included in separate sections below. Most of them will be updated/incorporated in the sections above.
You can also have a look at the list of problems from the v1 guide, as some of them might be relevant for this guide as well (or your setup).
1. Problems with Plex transcode using Linux kernel 6.8.x
The “new” install method would still not make Plex transcode.
It seems that there has been issues with Plex transcoding using 6.8.x kernel. See this forums.plex.tv discussion. For a while I thought this might be related, as upgrading to the latest Proxmox/Debian introduced 6.8.x kernel. However, these issues seems to be mostly solved by now.
There was specific mentions that Plex would expect /dev/dri devices being passed through (especially if you’ve enabled tone mapping in the Plex transcode settings), but I could not confirm if this was strictly needed when using NVIDIA GPUs or not. Since it’s required for certain other aspects, I decided to add it to be on the safe side. This would anyways be needed if you’re using an Intel or AMD GPU.
This did however not solve the issue. See the next section below for further troubleshooting.
2. Devices under /dev inside Docker not populated
After a lot of troubleshooting, I noticed that a Docker container with the parameter --gpus would work, but not if I ran the same command via Docker Compose. The same docker-compose.yml that I’ve had “forever” would no longer work:
services:
test:
image: tensorflow/tensorflow:latest-gpu
command: nvidia-smi
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]Even running nvidia-smi would fail with the following error:
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "nvidia-smi": executable file not found in $PATH: unknownLooking further into this, I noticed that the relevant NVIDIA devices would not populate under /dev/:
/dev:
total 512
drwxr-xr-x 6 root root 360 Apr 20 08:34 .
drwxr-xr-x 17 root root 2 Apr 20 08:34 ..
lrwxrwxrwx 1 root root 11 Apr 20 08:34 core -> /proc/kcore
drwxr-xr-x 2 root root 100 Apr 20 08:34 dri
lrwxrwxrwx 1 root root 13 Apr 20 08:34 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Apr 20 08:34 full
drwxrwxrwt 2 root root 40 Apr 20 08:34 mqueue
crw-rw-rw- 1 root root 1, 3 Apr 20 08:34 null
lrwxrwxrwx 1 root root 8 Apr 20 08:34 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 Apr 20 08:34 pts
crw-rw-rw- 1 root root 1, 8 Apr 20 08:34 random
lrwxrwxrwx 1 root root 15 Apr 20 08:34 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Apr 20 08:34 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Apr 20 08:34 stdout -> /proc/self/fd/1
crw-rw-rw- 1 root root 5, 0 Apr 20 08:34 tty
crw-rw-rw- 1 root root 1, 9 Apr 20 08:34 urandom
crw-rw-rw- 1 root root 1, 5 Apr 20 08:34 zeroAfter looking into this some more, I found this discussion. It seems that there was/is a bug in Docker Compose. If you haven’t set the count: parameter in the Docker Compose file, it will not load NVIDIA Container Runtime, which in turn doesn’t populate the /dev/ folder with the appropriate files.
Rather than setting a specific driver version or count, we can simply include all GPUs by setting count: all, like this:
services:
test:
image: tensorflow/tensorflow:latest-gpu
command: nvidia-smi
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
count: all
root@docker1:~# docker compose up
[+] Running 1/1
✔ Container test-gpu-test-1 Recreated 0.1s
Attaching to test-1
test-1 | Sun Apr 20 08:48:47 2025
test-1 | +-----------------------------------------------------------------------------------------+
test-1 | | NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
test-1 | |-----------------------------------------+------------------------+----------------------+
test-1 | | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
test-1 | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
test-1 | | | | MIG M. |
test-1 | |=========================================+========================+======================|
test-1 | | 0 NVIDIA RTX A2000 Off | 00000000:82:00.0 Off | Off |
test-1 | | 30% 33C P8 11W / 70W | 4MiB / 6138MiB | 0% Default |
test-1 | | | | N/A |
test-1 | +-----------------------------------------+------------------------+----------------------+
test-1 |
test-1 | +-----------------------------------------------------------------------------------------+
test-1 | | Processes: |
test-1 | | GPU GI CI PID Type Process name GPU Memory |
test-1 | | ID ID Usage |
test-1 | |=========================================================================================|
test-1 | | No running processes found |
test-1 | +-----------------------------------------------------------------------------------------+
test-1 exited with code 0This, however, did not seem to solve the transcoding issue completely. It was a step in the right direction, as the GPU at least now showed up in the “Hardware transcoding device” list within Plex settings:
It did not solve the issues with Tensorflow, though. See the next section for more info regarding that.
3. Problems with the Tensorflow test
One of the tests that I used in my original guide was Tensorflow. After the initial upgrade where things stopped working, I couldn’t get the Tensorflow-test to work either. This is the same issue that James struggled with. He opted to install xorg/X/etc, but that’s not something I wantet to do. This was one of the factors that led me to “start from scratch” on the setup.
After solving the count: all issue described in the previous section above, my old Tensorflow test would still not work. however, there was a difference in output.
Output from before count: all fix:
2025-04-20 08:35:06.689498: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1745138106.835511 1 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1745138106.876932 1 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1745138107.147055 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1745138107.147138 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1745138107.147145 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1745138107.147150 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-04-20 08:35:07.179493: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-04-20 08:35:12.422576: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (34)Output after fixing count: all:
2025-04-20 08:50:57.013493: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1745139057.162458 1 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1745139057.203735 1 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1745139057.483344 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1745139057.483437 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1745139057.483443 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1745139057.483448 1 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-04-20 08:50:57.517940: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
W0000 00:00:1745139063.945881 1 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...I still haven’t solved this issue, but since I don’t need Tensorflow to work, I have not spent anymore time trying to troubleshoot this any further. Maybe a reinstall of nvidia-container-toolkit would work, since NVIDIA recommends that you install it after you install the drivers (and we technically did that when converting from the old setup). If you still encounter this issue with a clean setup, then it’s something else…
4. Problems with environment variables
After still not getting things to work with the above fixes, I did some further troubleshooting. Amidst all the testing/troubleshooting, I had set the following two environmental variables in the Plex container (as there was several references online to these):
NVIDIA_VISIBLE_DEVICES: all
NVIDIA_DRIVER_CAPABILITIES: allIn my previous config, these were set slightly different:
NVIDIA_VISIBLE_DEVICES: all
NVIDIA_DRIVER_CAPABILITIES: compute,video,utilityChanging it back to the latter works (i.e. NVIDIA_DRIVER_CAPABILITIES: all will NOT work with Plex).
5. Use tmpfs/RAM for faster transcoding
In my previous setup I discovered that transcodes would be slow if the transcode directory was using fuse-overlayfs. I also discovered that even if you set a transcoder temporary directory in Plex settings, it’s not used for everything (i.e. the “Detecting intros” jobs would also transcode, and used /tmp). Both of these were fixed by mounting folders from the LXC container into the Docker container.
However, there’s an even better way to do this. If you have enough RAM, we can use a tmpfs mount, which is a temporary storage in memory/RAM. You can adjust the size of this in the docker-compose.yml file to something suitable for your available memory. There seems to be several people that has success using 8GB as the size, which seems to support multiple simultaneous transcodes (but you might have to increase it if you expect many simultaneous transcodes). If you don’t have a sufficient amount of memory/RAM available, this might not be the solution for you, and you should consider using the local storage method I had in my v1 guide.
By using tmpfs, the transcodes should be even faster/snappier, especially skipping within a movie/episode. The final docker-compose.yml for Plex takes both of these into account. You’ll have to set /tmp as the value for “Transcoder temporary directory” in the Plex settings. This way both transcodes and “Detecting intros” jobs would use the same path.
Tremendous updated guide – just in time for me nuking my entrie proxmox box and starting from scratch!
Thanks a million.