Can't enable hardware decoders for H.264/AVC and H.265/HEVC without enabling software decoders

[Neal Gompa]

Summary of the bug:
In Fedora, we're trying to ship a build of FFmpeg that's maximally useful without shipping encumbered stuff. To that end, we want to be able to rely on built-in hardware codecs available on the user's system for codecs we can't ship (such as H264 (AVC) and HEVC (H.265)). Unfortunately, it seems that it doesn't work for all decoders but the V4L2-M2M and QSV ones. That means people who have NVDEC, VDPAU, and VAAPI offload for decoding cannot use them through ffmpeg.

A solution for this would require source level separation so that just enough stuff is available to leverage the hardware decoders, since we have to strip ffmpeg sources of stuff we can't ship before we upload it for build. My understanding is that the AV1 decoder has this separation (though we have AV1 fully turned on in FFmpeg in Fedora), so it's not entirely without precedent.

How to reproduce:

1. Download the ffmpeg-5.0 release tarball
2. Build ffmpeg with the following configure line:

   % ./configure --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 '--optflags=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' '--extra-cflags=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --enable-version3 --enable-libsmbclient --disable-openssl --enable-gnutls --enable-ladspa --enable-vulkan --disable-cuda-sdk --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librubberband --enable-libsvtav1 --enable-libsoxr --enable-libspeex --enable-libssh --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libv4l2 --enable-libvpx --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libzvbi --enable-lto --enable-libmfx --enable-vaapi --enable-vdpau --enable-muxers --enable-demuxers --disable-encoders --disable-decoders --disable-decoder=mpeg4,h263,h264,hevc,vc1 --enable-encoder=,libfdk_aac,ac3,apng,ass,ayuv,bmp,ffv1,ffvhuff,flac,gif,h263_v4l2m2m,h264_amf,h264_nvenc,h264_qsv,h264_v4l2m2m,h264_vaapi,hevc_amf,hevc_nvenc,hevc_qsv,hevc_v4l2m2m,hevc_vaapi,huffyuv,jpegls,jpeg2000,libaom,libaom_av1,libcodec2,libgsm,libmp3lame,libopenjpeg,libopus,librav1e,libschroedinger,libspeex,libsvtav1,libtheora,libtwolame,libvorbis,libvpx_vp8,libvpx_vp9,libwebp,libwebp_anim,mjpeg,mjpeg_qsv,mp2,mp2fixed,mpeg1video,mpeg2video,mpeg2_qsv,mpeg4_v4l2m2m,opus,pam,pbm,pcm_alaw,pcm_f32be,pcm_f32le,pcm_f64be,pcm_f64le,pcm_mulaw,pcm_s16be,pcm_s16be_planar,pcm_s16le,pcm_s16le_planar,pcm_s24be,pcm_s24le,pcm_s24le_planar,pcm_s32be,pcm_s32le,pcm_s32le_planar,pcm_s8,pcm_s8_planar,pcm_u16be,pcm_u16le,pcm_u24be,pcm_u24le,pcm_u32be,pcm_u32le,pcm_u8,pcx,pgm,pgmyuv,png,ppm,rawvideo,sgi,srt,ssa,sunrast,targa,text,tiff,v210,v308,v408,v410,vc1_qsv,vc1_v4l2m2m,vorbis,vp8_qsv,vp8_v4l2m2m,vp9_qsv,webvtt,wrapped_avframe,xbm,xwd,y41p,yuv4,zlib, --enable-decoder=,libfdk_aac,ac3,ansi,apng,ass,av1_qsv,ayuv,bmp,dirac,exr,ffv1,ffvhuff,ffwavesynth,flac,gif,gsm,h263_v4l2m2m,h264_nvdec,h264_vdpau,h264_vaapi,h264_v4l2m2m,h264_qsv,hevc_nvdec,hevc_vdpau,hevc_vaapi,hevc_v4l2m2m,hevc_qsv,huffyuv,ilbc,jpeg2000,libaom,libaom_av1,libcelt,libcodec2,libdav1d,libgsm,libopenjpeg,libopus,libschroedinger,libspeex,libvorbis,libvpx_vp8,libvpx_vp9,mjpeg,mjpeg_qsv,mp1,mp1float,mp2,mp2float,mp3,mp3float,mpeg1video,mpeg1_v4l2m2m,mpeg2video,mpeg2_qsv,mpeg2_v4l2m2m,mpeg4_v4l2m2m,opus,pam,pbm,pcm_alaw,pcm_bluray,pcm_dvd,pcm_f32be,pcm_f32le,pcm_f64be,pcm_f64le,pcm_mulaw,pcm_s16be,pcm_s16be_planar,pcm_s16le,pcm_s16le_planar,pcm_s24be,pcm_s24le,pcm_s24le_planar,pcm_s32be,pcm_s32le,pcm_s32le_planar,pcm_s8,pcm_s8_planar,pcm_u16be,pcm_u16le,pcm_u24be,pcm_u24le,pcm_u32be,pcm_u32le,pcm_u8,pcx,pgm,pgmyuv,pgssub,pgx,png,ppm,rawvideo,sgi,srt,ssa,sunrast,targa,text,theora,tiff,v210,v210x,v308,v408,v410,vc1_qsv,vc1_v4l2m2m,vorbis,vp3,vp5,vp6,vp6a,vp6f,vp8,vp8_qsv,vp8_v4l2m2m,vp9,vp9_qsv,vp9_v4l2m2m,webp,webvtt,wrapped_avframe,xbm,xwd,y41p,yuv4,zlib,
   

3. Note that the resulting ffmpeg lacks a number of hardware decoders enabled at configure time:

   % ffmpeg -decoders | grep h264
   [...]
    V..... h264_v4l2m2m         V4L2 mem2mem H.264 decoder wrapper (codec h264)
    V....D h264_qsv             H264 video (Intel Quick Sync Video acceleration) (codec h264)
   % ffmpeg -decoders | grep hevc
   [...]
    V....D hevc_qsv             HEVC video (Intel Quick Sync Video acceleration) (codec hevc)
    V..... hevc_v4l2m2m         V4L2 mem2mem HEVC decoder wrapper (codec hevc)
   

[Balling]

codecs we can't ship (such as H264 (AVC) and

Of course you can ship h264, you cannot ship libx264. NVDEC is not going to work without h264, just like the encoder. NEVER. And maybe cuvid decoders/encoders may allow this, but I doubt it and they are deprecated anyway.

[Neal Gompa]

Replying to Balling:

codecs we can't ship (such as H264 (AVC) and

Of course you can ship h264, you cannot ship libx264. NVDEC is not going to work without h264, just like the encoder. NEVER. And maybe cuvid decoders/encoders may allow this, but I doubt it and they are deprecated anyway.

The hardware encoders work just fine with having the ffmpeg h264 and hevc software stuff disabled and stripped.

% ffmpeg -encoders | grep h264
[...]
 V....D h264_amf             AMD AMF H.264 Encoder (codec h264)
 V....D h264_nvenc           NVIDIA NVENC H.264 encoder (codec h264)
 V..... h264_qsv             H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration) (codec h264)
 V..... h264_v4l2m2m         V4L2 mem2mem H.264 encoder wrapper (codec h264)
 V....D h264_vaapi           H.264/AVC (VAAPI) (codec h264)
% ffmpeg -encoders | grep hevc
[...]
 V....D hevc_amf             AMD AMF HEVC encoder (codec hevc)
 V....D hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)
 V..... hevc_qsv             HEVC (Intel Quick Sync Video acceleration) (codec hevc)
 V..... hevc_v4l2m2m         V4L2 mem2mem HEVC encoder wrapper (codec hevc)
 V....D hevc_vaapi           H.265/HEVC (VAAPI) (codec hevc)

[mkver]

AV1 is not a real precedent for this; we just don't have a native AV1 decoder yet.

[Balling]

Okay, but the nvdec hwaccel is using h264 to parse everything... And as for nvenc you are being very funny right now.

​https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/nvenc_h264.c

[Balling]

Replying to mkver:

AV1 is not a real precedent for this; we just don't have a native AV1 decoder yet.

We do have an OBU parser though. And h.274 film grain and specific to av1 too (?). And what not.

[Carl Eugen Hoyos]

If the described behaviour changes (which is not so unlikely), it will not be related to the reasoning of this ticket.