av1_nvenc output frames sometimes cannot be decoded

[nikomo]

Summary of the bug: av1_nvenc output with some inputs will occasionally produce unplayable files. Attached files result in latest stable VLC decoding blocks, but not any frames. Windows's built-in Media Player plays back sample fine, but with full file as input, audio in output file will not be played back in Media Player but video will, while VLC will play audio but not video.
How to reproduce:

PS E:\encode_testing\av1_nvenc_bug> .\ffmpeg.exe -v 9 -loglevel 99 -i .\test_input.mkv
ffmpeg version 2024-12-27-git-5f38c82536-essentials_build-www.gyan.dev Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14.2.0 (Rev1, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
  libavutil      59. 53.100 / 59. 53.100
  libavcodec     61. 28.100 / 61. 28.100
  libavformat    61.  9.102 / 61.  9.102
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  6.101 / 10.  6.101
  libswscale      8. 13.100 /  8. 13.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input url with argument '.\test_input.mkv'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input url .\test_input.mkv.
Successfully parsed a group of options.
Opening an input file: .\test_input.mkv.
[AVFormatContext @ 000001d618d5f100] Opening '.\test_input.mkv' for reading
[file @ 000001d618d5f780] Setting default whitelist 'file,crypto,data'
Probing matroska,webm score:100 size:2048
[matroska,webm @ 000001d618d5f100] Format matroska,webm probed with size=2048 and score=100
[matroska,webm @ 000001d618d5f100] Unknown entry 0x22B59D at pos. 4472
[matroska,webm @ 000001d618d5f100] Unknown entry 0x22B59D at pos. 4569
[matroska,webm @ 000001d618d5f100] Unknown entry 0x22B59D at pos. 6296
[matroska,webm @ 000001d618d5f100] Unknown entry 0x437D at pos. 7547
st:0 removing common factor 1000000 from timebase
st:1 removing common factor 1000000 from timebase
st:2 removing common factor 1000000 from timebase
[matroska,webm @ 000001d618d5f100] Before avformat_find_stream_info() pos: 7660 bytes read:33772 seeks:2 nb_streams:3
[h264 @ 000001d618d65a80] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] nal_unit_type: 8(PPS), nal_ref_idc: 3
[matroska,webm @ 000001d618d5f100] parser not found for codec pcm_s24le, packets or times may be invalid.
[h264 @ 000001d618d65a80] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] ct_type:1 pic_struct:0
[h264 @ 000001d618d65a80] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] nal_unit_type: 6(SEI), nal_ref_idc: 0
    Last message repeated 2 times
[h264 @ 000001d618d65a80] nal_unit_type: 5(IDR), nal_ref_idc: 3
    Last message repeated 3 times
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] Format yuv420p chosen by get_format().
[h264 @ 000001d618d65a80] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 000001d618d65a80] ct_type:1 pic_struct:0
[h264 @ 000001d618d65a80] no picture
[matroska,webm @ 000001d618d5f100] parser not found for codec pcm_s24le, packets or times may be invalid.
[h264 @ 000001d618d65a80] ct_type:1 pic_struct:0
    Last message repeated 4 times
[matroska,webm @ 000001d618d5f100] All info found
[matroska,webm @ 000001d618d5f100] stream 0: start_time: 0 duration: NOPTS
[matroska,webm @ 000001d618d5f100] stream 1: start_time: 0 duration: NOPTS
[matroska,webm @ 000001d618d5f100] stream 2: start_time: 0 duration: 5.256
[matroska,webm @ 000001d618d5f100] format: start_time: 0 duration: 5.256 (estimate from stream) bitrate=31915 kb/s
[matroska,webm @ 000001d618d5f100] After avformat_find_stream_info() pos: 90112 bytes read:132076 seeks:2 frames:62
[aist#0:1/pcm_s24le @ 000001d618dbd8c0] Guessed Channel Layout: stereo
Input #0, matroska,webm, from '.\test_input.mkv':
  Metadata:
    creation_time   : 2024-12-31T23:06:15.000000Z
    ENCODER         : Lavf61.9.102
  Duration: 00:00:05.26, start: 0.000000, bitrate: 31915 kb/s
  Chapters:
    Chapter #0:0: start 0.033000, end 5.000000
      Metadata:
        title           : Intro
  Stream #0:0(jpn), 6, 1/1000: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1920x1080 [SAR 1:1 DAR 16:9], 0/1, 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      title           : JPNBD
      BPS             : 29707337
      DURATION        : 00:00:05.256000000
      NUMBER_OF_FRAMES: 124
      NUMBER_OF_BYTES : 19517721
      _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-12-31 23:06:15
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:1(jpn), 56, 1/1000: Audio: pcm_s24le, 48000 Hz, stereo, s32 (24 bit), 2304 kb/s (default)
    Metadata:
      title           : PCM 2.0
      BPS             : 2304000
      DURATION        : 00:00:05.000000000
      NUMBER_OF_FRAMES: 1000
      NUMBER_OF_BYTES : 1440000
      _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-12-31 23:06:15
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng), 0, 1/1000: Subtitle: ass (ssa) (default)
    Metadata:
      title           : Subtitles
      BPS             : 0
      DURATION        : 00:00:00.000000000
      NUMBER_OF_FRAMES: 0
      NUMBER_OF_BYTES : 0
      _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-12-31 23:06:15
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Successfully opened the file.
At least one output file must be specified
[AVIOContext @ 000001d618d5fbc0] Statistics: 132076 bytes read, 2 seeks

@echo off
.\ffmpeg.exe -y -report -i test_input.mkv -c:v h264_nvenc debug_h264_nvenc.mkv
.\ffmpeg.exe -y -report -i test_input.mkv -c:v hevc_nvenc debug_hevc_nvenc.mkv
.\ffmpeg.exe -y -report -i test_input.mkv -c:v libsvtav1 debug_libsvtav1.mkv
.\ffmpeg.exe -y -report -i test_input.mkv -c:v av1_nvenc debug_av1_nvenc.mkv

[nikomo]

Test input file

[nikomo]

h264_nvenc output

[nikomo]

hevc_nvenc output

[nikomo]

libstvav1 output (had to substitute in another FFmpeg build due to original one not having this encoder included)

[nikomo]

av1_nvenc output

[nikomo]

VLC showing blocks were decoded, but no frames

[nikomo]

I tried this.

@echo off
.\ffmpeg.exe -y -hwaccel cuda -hwaccel_output_format cuda -i test_input.mkv -c:v hevc_nvenc -preset:v p1 -tune:v lossless -profile:v main -tier:v high -rc:v constqp -rc-lookahead:v 0 -2pass:v 0 -qp:v 0 -multipass:v 0 -lookahead_level:v 0 -split_encode_mode:v 1 -c:a copy -c:s copy -c:d copy tmp.mkv
.\ffmpeg.exe -y -hwaccel cuda -hwaccel_output_format cuda -i tmp.mkv -c:v av1_nvenc -preset:v p7 -tune:v uhq -rc:v vbr -b:v 0 -multipass:v 2 -rc-lookahead:v 30 -lookahead_level:v 15 -split_encode_mode:v 1 -cq:v 30 -c:a copy -c:s copy -c:d copy debug.mkv

To my utter confusion, the output file still has the issue. I literally re-encoded it to HEVC in between, and it still has the same issue. I have never been this confused in my life.

[Balling]

while VLC will play audio but not video.

This is obviously a bug in VLC. My new Intel Arrow lake has no issues decoding after a couple of frames that fail to decode.

mpv.com   --target-colorspace-hint=yes -vo=gpu-next -gpu-api=d3d11 -hwdec=d3d11va debug_av1_nvenc.mkv
● Video  --vid=1  --vlang=jpn  'JPNBD' (av1 1920x1080 23.976 fps) [default]
● Audio  --aid=1  --alang=jpn  'PCM 2.0' (vorbis 2ch 48000 Hz) [default]
● Subs   --sid=1  --slang=eng  'Full Subtitles [Sunraku]' (ass) [default]
[ffmpeg/video] av1: trailing_zero_bit out of range: 1, but must be in [0,0].
[ffmpeg/video] av1: Failed to read unit 1 (type 5).
[ffmpeg/video] av1: Failed to read packet.
Error while decoding frame (hardware decoding)!
[ffmpeg/video] av1: trailing_one_bit out of range: 0, but must be in [1,1].
[ffmpeg/video] av1: Failed to read unit 0 (type 5).
[ffmpeg/video] av1: Failed to read packet.
Error while decoding frame (hardware decoding)!
[ffmpeg/video] av1: Missing reference frame needed for show_existing_frame (frame_to_show_map_idx = 6).
[ffmpeg/video] av1: Failed to read unit 0 (type 3).
[ffmpeg/video] av1: Failed to read packet.
Error while decoding frame (hardware decoding)!
AO: [wasapi] 48000Hz stereo 2ch float
VO: [gpu-next] 1920x1080 yuv420p
AV: 00:00:01 / 00:00:01 (89%) A-V:  0.000
Exiting... (End of file)

[nikomo]

That did remind me, I forgot to test something.
I disabled hardware accelerated decode in VLC so it would be forced to use dav1d, and that decodes it successfully. I'll have to chase down that rabbit hole.

[Balling]

Those AV1 warnings come from software decoder. Counter to popular belief software decoder is still very much used to preparse the data to send it to HW decoder
Can you try instead of -hwaccel cuda -hwaccel_output_format cuda -i

Legacy way

-c:v hevc_cuvid -i

And maybe av1_cuvid tmif that exists.