User Agent: Mozilla/5.0 (Android 11; Mobile; rv:98.0) Gecko/98.0 Firefox/98.0

Steps to reproduce:

https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/chap39.html
https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/chap39.html
https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/chap39.html
Vulkan has an extension to video decoding, it can be used to implement hardware decoding in Firefox as an optional method.
Nvidia driver (Windows 458.17, Linux 455.50.12, since 2021-04-03) has implemented Video Decoding API.
https://developer.nvidia.com/vulkan-driver
It can be a better universal replacement than proprietary Nvidia vdpau API.
This method can be used for intel, AMD and Nvidia graphic Card once they implement this extension, as NVIDIA has done. And it can be used in both Windows, Linux and macOS, because Vulkan is available in all three platforms.

Actual results:

There's no hardware decoding way through Vulkan Video Decoding API extension.

Expected results:

Make hardware decoding through Vulkan Video Decoding API implemented.

ffmpeg is working on it, this is their branch https://github.com/cyanreg/FFmpeg/commits/vulkan_decode . It maybe a example to refer to.
And as MPV developer said ( https://github.com/mpv-player/mpv/issues/9675#issuecomment-1005623063 ), this feature is only added in Nvidia Vulkan Beta driver. It was my mistake to say it has been added to the stable version.

And there is an example to use this feature. https://github.com/nvpro-samples/vk_video_samples

Related(Chromium's feature request): https://bugs.chromium.org/p/chromium/issues/detail?id=1200559

Mesa's Intel Vulkan driver is working on implement this extension, this is their PR. https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13940

Mesa has merged PR that make AMD support Vulkan video decode. It will be a part of mesa 23.1.
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20388

I'd like to highlight the recent release of FFmpeg version 6.1, which introduces support for Vulkan Video Decoding/Encoding. This could significantly enhance the multimedia capabilities of Firefox.
https://ffmpeg.org/index.html#pr6.1

It wasn't mentioned here that only H.264 and H.265 were supported.
Recently AV1 was introduced too:
https://www.khronos.org/blog/khronos-releases-vulkan-video-av1-decode-extension-vulkan-sdk-now-supports-h.264-h.265-encode

In case the blog link breaks eventually, the extension is:
https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_KHR_video_decode_av1.html

Mesa support:

• Common: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27423
• AMD: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27424
• Intel: ?

mesa mainlined VK_KHR_video_decode_av1 extension support in radv (amd vulkan driver). Would be available for mesa 24.1.

MESA v24.0.4- ANV(Intel) supports these extensions as of now:

❯ vulkaninfo | grep video
MESA-INTEL: warning: cannot initialize blitter engine
	VK_KHR_video_decode_h264                    : extension revision 9
	VK_KHR_video_decode_h265                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 8
	VK_KHR_video_queue                          : extension revision 8
			videoCodecOperations:
			videoCodecOperations: count = 2

What we need is to implement dmabuf surface export from vulkan video frame. Then it can be used by rest of gecko.

Would this possible to use ffmpeg vulkan decode ? I think we don't need to directly support VK_KHR_video, but we could just support ffmpeg lib that use VK_KHR_video, thus maybe the codechanges a lot smaller (rather than doing fully zero copy VK_KHR_video_*) ?

The problem is that VAAPI api is kinda cyclic (between mesa and vaapi) and hacky in term of integration in Linux ecosystem, sometimes some distro integration fails just because dlsym error when loading / casting function from dri (it could be version missmatch between libva & .dri.so driver, etc).

I think with VK_KHR_video, even if this won't be quite effecient as expected (?), I think the integration would be a lot of smoother across Linux distro rather than using VAAPI. It doesn't mean dropping support for VAAPI but I think the future will be brighter rather than relying on VAAPI.

(In reply to zerosign from comment #12)

I think with VK_KHR_video, even if this won't be quite effecient as expected (?), I think the integration would be a lot of smoother across Linux distro rather than using VAAPI. It doesn't mean dropping support for VAAPI but I think the future will be brighter rather than relying on VAAPI.

You will have to keep VAAPI anyway because the hardware support for Vulkan Video isn't there for everyone. I have an RDNA4 GPU which unfortunately nobody has contributed support yet (https://gitlab.freedesktop.org/mesa/mesa/-/issues/13118), but VAAPI works fine. There are also cases like NVIDIA though, which does not support VAAPI support at all (there is a best-effort community driver though which is often tied to NVIDIA driver internals and breaks a lot) but they do have working Vulkan Video so this support could be really helpful for them.