Hardware Accelerated Video on Linux

Why is my CPU at 100% at 2560x1440x60 YouTube videos? And even gives framedrops? Same and worse problems at 3840x2160x30, and higher.
Shouldn't there be GPU acceleration? Why is radeontop showing me, that my GPU is basically idling?


Because…
As always, there is not just one, but many.
Steeling information from archwiki:

• Video Acceleration API (VA-API)
• Video Decode and Presentation API for Unix (VDPAU)
• NVDECODE/NVENCODE

Which one is for you? All and none, probably. As every vendor, every driver, every generation seems to use something else. And there is also a translation layer for VA-API on top of VDPAU. Running on an AMD GPU I can use VA-API and VDPAU.
vainfo and vdpauinfo are both giving me good results back about supported codecs, like:

• MPEG2
• H264
• VC1
• MPEG4
• HVEC
• VP9

Looking at YouTubes "Statistics for Nerds" it shows:


So it should work. Right?

Following the links to Chromium and Firefox are leading to

A) you need a custom chromium with va-api, or
B) wayland to get acceleration with firefox.

Not installing the special chromium, and running Xorg, I don't currently have acceleration inside the browser.

But what about vlc and mpv?
Thanks to youtube-dl you can download the videos from YouTube in lots of different formats and sizes, e.g.:

$ youtube-dl -F https://www.youtube.com/watch?v=N1-Jmq7BLFE
[youtube] N1-Jmq7BLFE: Downloading webpage
[info] Available formats for N1-Jmq7BLFE:
format code  extension  resolution note
249          webm       audio only tiny   66k , opus @ 50k (48000Hz), 2.09MiB
250          webm       audio only tiny   84k , opus @ 70k (48000Hz), 2.77MiB
140          m4a        audio only tiny  130k , m4a_dash container, mp4a.40.2@128k (44100Hz), 5.24MiB
251          webm       audio only tiny  155k , opus @160k (48000Hz), 5.43MiB
394          mp4        256x144    144p   91k , av01.0.00M.10.0.110.09.16.09.0, 30fps, video only, 2.62MiB
278          webm       256x144    144p   99k , webm container, vp9, 30fps, video only, 3.37MiB
160          mp4        256x144    144p  111k , avc1.4d400c, 30fps, video only, 2.23MiB
395          mp4        426x240    240p  201k , av01.0.00M.10.0.110.09.16.09.0, 30fps, video only, 4.79MiB
242          webm       426x240    240p  228k , vp9, 30fps, video only, 6.57MiB
330          webm       256x144    144p60 HDR  243k , vp9.2, 60fps, video only, 6.71MiB
133          mp4        426x240    240p  245k , avc1.4d4015, 30fps, video only, 4.46MiB
396          mp4        640x360    360p  363k , av01.0.01M.10.0.110.09.16.09.0, 30fps, video only, 9.26MiB
243          webm       640x360    360p  450k , vp9, 30fps, video only, 12.89MiB
331          webm       426x240    240p60 HDR  502k , vp9.2, 60fps, video only, 15.98MiB
134          mp4        640x360    360p  633k , avc1.4d401e, 30fps, video only, 13.34MiB
397          mp4        854x480    480p  687k , av01.0.04M.10.0.110.09.16.09.0, 30fps, video only, 18.22MiB
244          webm       854x480    480p  782k , vp9, 30fps, video only, 23.93MiB
332          webm       640x360    360p60 HDR 1065k , vp9.2, 60fps, video only, 37.82MiB
135          mp4        854x480    480p 1157k , avc1.4d401f, 30fps, video only, 28.26MiB
398          mp4        1280x720   720p60 1359k , av01.0.08M.10.0.110.09.16.09.0, 60fps, video only, 42.11MiB
247          webm       1280x720   720p 1624k , vp9, 30fps, video only, 49.54MiB
333          webm       854x480    480p60 HDR 1990k , vp9.2, 60fps, video only, 75.45MiB
136          mp4        1280x720   720p 2314k , avc1.4d401f, 30fps, video only, 57.56MiB
399          mp4        1920x1080  1080p60 2428k , av01.0.09M.10.0.110.09.16.09.0, 60fps, video only, 73.81MiB
302          webm       1280x720   720p60 2682k , vp9, 60fps, video only, 80.53MiB
248          webm       1920x1080  1080p 2835k , vp9, 30fps, video only, 90.95MiB
298          mp4        1280x720   720p60 4055k , avc1.4d4020, 60fps, video only, 102.17MiB
137          mp4        1920x1080  1080p 4338k , avc1.640028, 30fps, video only, 110.15MiB
303          webm       1920x1080  1080p60 4473k , vp9, 60fps, video only, 147.45MiB
334          webm       1280x720   720p60 HDR 4527k , vp9.2, 60fps, video only, 177.12MiB
299          mp4        1920x1080  1080p60 6841k , avc1.64002a, 60fps, video only, 194.13MiB
335          webm       1920x1080  1080p60 HDR 6920k , vp9.2, 60fps, video only, 274.03MiB
400          mp4        2560x1440  1440p60 8521k , av01.0.12M.10.0.110.09.16.09.0, 60fps, video only, 249.60MiB
271          webm       2560x1440  1440p 9023k , vp9, 30fps, video only, 289.62MiB
308          webm       2560x1440  1440p60 13329k , vp9, 60fps, video only, 451.32MiB
336          webm       2560x1440  1440p60 HDR 16982k , vp9.2, 60fps, video only, 656.49MiB
401          mp4        3840x2160  2160p60 17006k , av01.0.13M.10.0.110.09.16.09.0, 60fps, video only, 504.35MiB
313          webm       3840x2160  2160p 18006k , vp9, 30fps, video only, 662.93MiB
402          mp4        7680x4320  4320p60 21504k , av01.0.17M.10.0.110.09.16.09.0, 60fps, video only, 605.22MiB
315          webm       3840x2160  2160p60 26636k , vp9, 60fps, video only, 1015.53MiB
337          webm       3840x2160  2160p60 HDR 30403k , vp9.2, 60fps, video only, 1.14GiB
272          webm       7680x4320  4320p60 30412k , vp9, 60fps, video only, 855.23MiB
571          mp4        7680x4320  4320p60 33459k , av01.0.17M.10.0.110.09.16.09.0, 60fps, video only, 960.11MiB
18           mp4        640x360    360p  617k , avc1.42001E, mp4a.40.2@ 96k (44100Hz), 24.99MiB
22           mp4        1280x720   720p 1551k , avc1.64001F, mp4a.40.2@192k (44100Hz) (best)

Let's get the best this video has to offer:

$ youtube-dl -f "bestvideo+bestaudio" https://www.youtube.com/watch?v=N1-Jmq7BLFE

Running it with mpv yields information about the format:

av1 7680x4320 59.940fps

And also with -v option:

[vd] Codec list:
[vd]     libaom-av1 (av1) - libaom AV1
[vd] Opening decoder libaom-av1
[vd] No hardware decoding requested.
[vd] Using software decoding.

AV1 is currently not supported for hardware acceleration.
VP9 should work.
Disallowing AV1 would be nice, but as you can see in the list, it is not just av01, but lots of different av01.something. They all have in commen, that they are all inside the mp4 container, and the others inside webm.

VP9 decoding is only available with va-api, so I request this from mpv:

$ mpv -v --hwdec=vaapi Bulgaria\ 8K\ HDR\ 60P\ \(FUHD\)-N1-Jmq7BLFE.webm

It tries:

[vd] Codec list:
[vd]     vp9 - Google VP9
[vd]     vp9_v4l2m2m (vp9) - V4L2 mem2mem VP9 decoder wrapper
[vd]     libvpx-vp9 (vp9) - libvpx VP9
[vd] Opening decoder vp9
[vd] Looking at hwdec vp9-vaapi...

But then gives up:

[ffmpeg/video] vp9: Hardware does not support image size 7680x4320 (constraints: width 0-4096 height 0-4096).

So, we download another video. This time with smaller size:

$ youtube-dl -f "bestvideo[ext=webm][width<=4096][height<=4096]+bestaudio" https://www.youtube.com/watch?v=N1-Jmq7BLFE

And now mpv is saying:

[vd] Using hardware decoding (vaapi).

In vlc it doesn't work for me and just gives errors:

glconv_vaapi_x11 gl error: vaDeriveImage: operation failed
main video output error: video output creation failed
main decoder error: failed to create video output

Back to mpv, which can directly take YouTube-URLs, and also has an option, which will be forwarded to youtube-dl. In that we can directly specify the format we want. So that we will get a video, which directly can be hardware decoded:

$ mpv --hwdec=yes --ytdl-format="bestvideo[width<=4096][height<=4096][ext=webm]+bestaudio" https://www.youtube.com/watch?v=N1-Jmq7BLFE