Firefox 81 on Fedora with VA-API, WebRTC and X11

Mozilla Firefox continues to implement more and more HW acceleration features on Linux and Fedora comes with extra integration patches to enable it.

Firefox 78 introduced VA-API video decoding on Wayland. Firefox 81 (a latest stable release) comes with VA-API for WebRTC streams decoding and enables VA-API on X11 for supported platforms.

Let’s start with X11. Firefox implements VA-API on top of DMABuf where particular video frames are exported by vaExportSurfaceHandle from libva and imported by EGL_EXT_image_dma_buf_import extension to OpenGL/EGL. That’s the default OpenGL sequence on Wayland where the VA-API was implemented first.

There are two options how to bring VA-API on X11 – vaPutSurface/GLX or vaExportSurfaceHandle/EGL. GLX is the default Firefox OpenGL backend on X11, it’s well supported on all devices including NVIDIA proprietary drivers and it’s generally well tested. On the contrary Firefox does not implement native X11 pixmap textures in any HW accelerated backend, shm based textures are used everywhere. There was an attempt to implement VA-API/vaPutSurface with Basic compositor but it was rejected.

VaExportSurfaceHandle/EGL is the second option for X11. EGL is well supported on Intel/AMD drivers but almost completely missing on NVIDIA, both proprietary and free ones (a partial solution may be GLX/EGL wrapper by Adam Jackson). But with EGL we can recycle most of the work done for Wayland and use the same code path which led to final EGL/X11 implementation by Robert Mader.

The EGL backend is not used by default yet, we’re working to enable it for Mesa drivers. Until that happens you need to run Firefox with set MOZ_X11_EGL env variable to switch from GLX to EGL, so run Firefox as:

MOZ_X11_EGL=1 firefox

As a consequence the dmabuf/vaapi preferences has been changed. Enable VA-API decoding by media.ffmpeg.vaapi.enabled set to true at about:config. That’s a single option for both Wayland and X11 backends as well as for WebRTC decoding (see bellow). Also make sure WebRender (Firefox HW accelerated backend) is enabled.

Users of Mozilla stock builds or other distros than Fedora need also set media.ffvpx.enabled to false until VA-API decoding is implemented in in-tree ffvpx library.

And now the WebRTC VA-API decoding. I tested the implementation on BlueJeans video conferencing system as it’s used by Red Hat internally. BlueJeans uses VP8 video streams and Intel HW decoder produces YV12 video frames which have to be handled by DMABuf surfaces. And we also want to route WebRTC video through external decoder by default [1][2].

If you use Fedora Firefox builds you are settled here, the patches [1][2] are included in the builds.

Users of other distros or Mozilla Firefox stock builds must set media.ffmpeg.low-latency.enabled and media.navigator.mediadatadecoder_vpx_enabled to true at about:config.