av1_nvenc output frames sometimes cannot be decoded

[nikomo]

Summary of the bug: av1_nvenc output with some inputs will occasionally produce unplayable files. Attached files result in latest stable VLC decoding blocks, but not any frames. Windows's built-in Media Player plays back sample fine, but with full file as input, audio in output file will not be played back in Media Player but video will, while VLC will play audio but not video.
How to reproduce:

PS E:\encode_testing\av1_nvenc_bug> .\ffmpeg.exe -v 9 -loglevel 99 -i .\test_input.mkv
ffmpeg version 2024-12-27-git-5f38c82536-essentials_build-www.gyan.dev Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14.2.0 (Rev1, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
  libavutil      59. 53.100 / 59. 53.100
  libavcodec     61. 28.100 / 61. 28.100
  libavformat    61.  9.102 / 61.  9.102
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  6.101 / 10.  6.101
  libswscale      8. 13.100 /  8. 13.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input url with argument '.\test_input.mkv'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input url .\test_input.mkv.
Successfully parsed a group of options.
Opening an input file: .\test_input.mkv.
[AVFormatContext @ 000001d618d5f100] Opening '.\test_input.mkv' for reading
[file @ 000001d618d5f780] Setting default whitelist 'file,crypto,data'
Probing matroska,webm score:100 size:2048
[matroska,webm @ 000001d618d5f100] Format matroska,webm probed with size=2048 and score=100
[matroska,webm @ 000001d618d5f100] Unknown entry 0x22B59D at pos. 4472
[matroska,webm @ 000001d618d5f100] Unknown entry 0x22B59D at pos. 4569
[matroska,webm @ 000001d618d5f100] Unknown entry 0x22B59D at pos. 6296
[matroska,webm @ 000001d618d5f100] Unknown entry 0x437D at pos. 7547
st:0 removing common factor 1000000 from timebase
st:1 removing common factor 1000000 from timebase
st:2 removing common factor 1000000 from timebase
[matroska,webm @ 000001d618d5f100] Before avformat_find_stream_info() pos: 7660 bytes read:33772 seeks:2 nb_streams:3
[h264 @ 000001d618d65a80] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] nal_unit_type: 8(PPS), nal_ref_idc: 3
[matroska,webm @ 000001d618d5f100] parser not found for codec pcm_s24le, packets or times may be invalid.
[h264 @ 000001d618d65a80] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] ct_type:1 pic_struct:0
[h264 @ 000001d618d65a80] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 000001d618d65a80] nal_unit_type: 6(SEI), nal_ref_idc: 0
    Last message repeated 2 times
[h264 @ 000001d618d65a80] nal_unit_type: 5(IDR), nal_ref_idc: 3
    Last message repeated 3 times
[h264 @ 000001d618d65a80] Decoding VUI
[h264 @ 000001d618d65a80] Format yuv420p chosen by get_format().
[h264 @ 000001d618d65a80] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 000001d618d65a80] ct_type:1 pic_struct:0
[h264 @ 000001d618d65a80] no picture
[matroska,webm @ 000001d618d5f100] parser not found for codec pcm_s24le, packets or times may be invalid.
[h264 @ 000001d618d65a80] ct_type:1 pic_struct:0
    Last message repeated 4 times
[matroska,webm @ 000001d618d5f100] All info found
[matroska,webm @ 000001d618d5f100] stream 0: start_time: 0 duration: NOPTS
[matroska,webm @ 000001d618d5f100] stream 1: start_time: 0 duration: NOPTS
[matroska,webm @ 000001d618d5f100] stream 2: start_time: 0 duration: 5.256
[matroska,webm @ 000001d618d5f100] format: start_time: 0 duration: 5.256 (estimate from stream) bitrate=31915 kb/s
[matroska,webm @ 000001d618d5f100] After avformat_find_stream_info() pos: 90112 bytes read:132076 seeks:2 frames:62
[aist#0:1/pcm_s24le @ 000001d618dbd8c0] Guessed Channel Layout: stereo
Input #0, matroska,webm, from '.\test_input.mkv':
  Metadata:
    creation_time   : 2024-12-31T23:06:15.000000Z
    ENCODER         : Lavf61.9.102
  Duration: 00:00:05.26, start: 0.000000, bitrate: 31915 kb/s
  Chapters:
    Chapter #0:0: start 0.033000, end 5.000000
      Metadata:
        title           : Intro
  Stream #0:0(jpn), 6, 1/1000: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1920x1080 [SAR 1:1 DAR 16:9], 0/1, 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      title           : JPNBD
      BPS             : 29707337
      DURATION        : 00:00:05.256000000
      NUMBER_OF_FRAMES: 124
      NUMBER_OF_BYTES : 19517721
      _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-12-31 23:06:15
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:1(jpn), 56, 1/1000: Audio: pcm_s24le, 48000 Hz, stereo, s32 (24 bit), 2304 kb/s (default)
    Metadata:
      title           : PCM 2.0
      BPS             : 2304000
      DURATION        : 00:00:05.000000000
      NUMBER_OF_FRAMES: 1000
      NUMBER_OF_BYTES : 1440000
      _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-12-31 23:06:15
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng), 0, 1/1000: Subtitle: ass (ssa) (default)
    Metadata:
      title           : Subtitles
      BPS             : 0
      DURATION        : 00:00:00.000000000
      NUMBER_OF_FRAMES: 0
      NUMBER_OF_BYTES : 0
      _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2024-12-31 23:06:15
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Successfully opened the file.
At least one output file must be specified
[AVIOContext @ 000001d618d5fbc0] Statistics: 132076 bytes read, 2 seeks

@echo off
.\ffmpeg.exe -y -report -i test_input.mkv -c:v h264_nvenc debug_h264_nvenc.mkv
.\ffmpeg.exe -y -report -i test_input.mkv -c:v hevc_nvenc debug_hevc_nvenc.mkv
.\ffmpeg.exe -y -report -i test_input.mkv -c:v libsvtav1 debug_libsvtav1.mkv
.\ffmpeg.exe -y -report -i test_input.mkv -c:v av1_nvenc debug_av1_nvenc.mkv

[nikomo]

Test input file

[nikomo]

h264_nvenc output

[nikomo]

hevc_nvenc output

[nikomo]

libstvav1 output (had to substitute in another FFmpeg build due to original one not having this encoder included)

[nikomo]

av1_nvenc output

[nikomo]

VLC showing blocks were decoded, but no frames

[nikomo]

I tried this.

@echo off
.\ffmpeg.exe -y -hwaccel cuda -hwaccel_output_format cuda -i test_input.mkv -c:v hevc_nvenc -preset:v p1 -tune:v lossless -profile:v main -tier:v high -rc:v constqp -rc-lookahead:v 0 -2pass:v 0 -qp:v 0 -multipass:v 0 -lookahead_level:v 0 -split_encode_mode:v 1 -c:a copy -c:s copy -c:d copy tmp.mkv
.\ffmpeg.exe -y -hwaccel cuda -hwaccel_output_format cuda -i tmp.mkv -c:v av1_nvenc -preset:v p7 -tune:v uhq -rc:v vbr -b:v 0 -multipass:v 2 -rc-lookahead:v 30 -lookahead_level:v 15 -split_encode_mode:v 1 -cq:v 30 -c:a copy -c:s copy -c:d copy debug.mkv

To my utter confusion, the output file still has the issue. I literally re-encoded it to HEVC in between, and it still has the same issue. I have never been this confused in my life.

[Balling]

while VLC will play audio but not video.

This is obviously a bug in VLC. My new Intel Arrow lake has no issues decoding after a couple of frames that fail to decode.

mpv.com   --target-colorspace-hint=yes -vo=gpu-next -gpu-api=d3d11 -hwdec=d3d11va debug_av1_nvenc.mkv
● Video  --vid=1  --vlang=jpn  'JPNBD' (av1 1920x1080 23.976 fps) [default]
● Audio  --aid=1  --alang=jpn  'PCM 2.0' (vorbis 2ch 48000 Hz) [default]
● Subs   --sid=1  --slang=eng  'Full Subtitles [Sunraku]' (ass) [default]
[ffmpeg/video] av1: trailing_zero_bit out of range: 1, but must be in [0,0].
[ffmpeg/video] av1: Failed to read unit 1 (type 5).
[ffmpeg/video] av1: Failed to read packet.
Error while decoding frame (hardware decoding)!
[ffmpeg/video] av1: trailing_one_bit out of range: 0, but must be in [1,1].
[ffmpeg/video] av1: Failed to read unit 0 (type 5).
[ffmpeg/video] av1: Failed to read packet.
Error while decoding frame (hardware decoding)!
[ffmpeg/video] av1: Missing reference frame needed for show_existing_frame (frame_to_show_map_idx = 6).
[ffmpeg/video] av1: Failed to read unit 0 (type 3).
[ffmpeg/video] av1: Failed to read packet.
Error while decoding frame (hardware decoding)!
AO: [wasapi] 48000Hz stereo 2ch float
VO: [gpu-next] 1920x1080 yuv420p
AV: 00:00:01 / 00:00:01 (89%) A-V:  0.000
Exiting... (End of file)

[nikomo]

That did remind me, I forgot to test something.
I disabled hardware accelerated decode in VLC so it would be forced to use dav1d, and that decodes it successfully. I'll have to chase down that rabbit hole.

[Balling]

Those AV1 warnings come from software decoder. Counter to popular belief software decoder is still very much used to preparse the data to send it to HW decoder
Can you try instead of -hwaccel cuda -hwaccel_output_format cuda -i

Legacy way

-c:v hevc_cuvid -i

And maybe av1_cuvid tmif that exists.

[cheppizzadrakon]

Hello, I have reported the same issue in ​https://trac.ffmpeg.org/ticket/10909
This is still an issue on the latest release (ffmpeg version 2025-08-14-git-cdbb5f1b93-full_build-www.gyan.dev)

Instead of relying on external players, I have used ffmpeg directly to test the decoding

ffmpeg.exe -y -hwaccel cuda -i debug_av1_nvenc.mkv -f null -

This produces the same error as seen by you in mpv

[av1 @ 0000027989a5a240] trailing_zero_bit out of range: 1, but must be in [0,0].
[av1 @ 0000027989a5a240] Failed to read unit 1 (type 5).
[av1 @ 0000027989a5a240] Failed to read packet.
[vist#0:0/av1 @ 0000027989a802c0] [dec:av1 @ 0000027989a7a580] Error submitting packet to decoder: Invalid data found when processing input
[av1 @ 0000027989a5a240] trailing_one_bit out of range: 0, but must be in [1,1].
[av1 @ 0000027989a5a240] Failed to read unit 0 (type 5).
[av1 @ 0000027989a5a240] Failed to read packet.

Which ultimately fails

[vist#0:0/av1 @ 0000027989a802c0] [dec:av1 @ 0000027989a7a580] Decode error rate 1 exceeds maximum 0.666667
[vist#0:0/av1 @ 0000027989a802c0] [dec:av1 @ 0000027989a7a580] Task finished with error code: -1145393733 (Error number -1145393733 occurred)
[vist#0:0/av1 @ 0000027989a802c0] [dec:av1 @ 0000027989a7a580] Terminating thread with return code -1145393733 (Error number -1145393733 occurred)
[vost#0:0/wrapped_avframe @ 0000027989a7e600] No filtered frames for output stream, trying to initialize anyway.
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf61.1.100
  Chapters:
    Chapter #0:0: start 0.033000, end 1.250000
      Metadata:
        title           : Intro
  Stream #0:0(jpn): Video: wrapped_avframe, yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 1k tbn (default)
      Metadata:
        title           : JPNBD
        DURATION        : 00:00:01.505000000
        BPS             : 34918000
        NUMBER_OF_FRAMES: 34141
        NUMBER_OF_BYTES : 6215251315
        _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-11-10 11:06:16
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
        encoder         : Lavc61.3.100 wrapped_avframe
  Stream #0:1(jpn): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
      Metadata:
        title           : PCM 2.0
        DURATION        : 00:00:01.253000000
        BPS             : 2304000
        NUMBER_OF_FRAMES: 284793
        NUMBER_OF_BYTES : 410101920
        _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-11-10 11:06:16
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
        encoder         : Lavc61.3.100 pcm_s16le
[out#0/null @ 00000279899ef840] video:0KiB audio:238KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=N/A bitrate=N/A speed=N/A
Conversion failed!

I think the difference in behavior between mpv and VLC is because mpv automatically falls back to the software decoder if the hardware one fails: opening the Stats for nerds menu, a working av1 video shows

Video: Alliance for Open Media AV1 [Main] HW: d3d11va

while the broken video shows

Video: Alliance for Open Media AV1 [Main] [libdav1d]

The cuvid decoders solve this issue: the av1 video will be decoded correctly when running

ffmpeg -c:v av1_cuvid -i debug_av1_nvenc.mkv -f null -

Also, using hevc_cuvid in the two-step example from above will produce an av1 video which can be played with -hwaccel cuda without needing av1_cuvid (i had to remove -tune:v uhq from the command since it produced a 0-bytes file)

.\ffmpeg.exe -y -c:v hevc_cuvid -i tmp.mkv -c:v av1_nvenc -preset:v p7 -rc:v vbr -b:v 0 -multipass:v 2 -rc-lookahead:v 30 -lookahead_level:v 15 -split_encode_mode:v 1 -cq:v 30 -c:a copy -c:s copy -c:d copy debug.mkv

As I wrote in my issue, also reencoding with a software codec like libx264 or libx265 before encoding again with av1_nvenc will produce a playable result

[Sesse]

I took a look at some of these streams. What seems to be going on is that the OBU padding (in the form of trailing bits) is wrong when there's more than one byte to be padded.

AV1 pads all of its OBUs to at least one byte granularity, with the pattern 100000... (a single one, then as many zeros as are needed). Generally, this works fine with the NVENC output. However, when there's more than eight padding bits, in one case here 113 bits, the pattern should have been

100(...100 more zeros...)0000000000

but is instead

000(...100 more zeros...)0010000000

The one is simply placed in the wrong byte. It should have been in the first, but is in the last.

The “trailing bits” reader/writer in libavcodec/cbs_av1_syntax_template.c actually enforces this padding, and thus, the stream is seen as illegal and the hardware decoder almost immediately fails. (This also goes for my Intel decoder, with VAAPI. So the problem is not NVDEC.) It then typically falls back to libdav1d, which simply does not bother to check the contents of the padding, and thus accepts the stream just fine.

I don't know what writes this padding; whether it's something in FFmpeg or in the NVENC drivers. But it is clearly an illegal bitstream -- although one FFmpeg could gladly choose to ignore if it so wishes. If I hack out the error returns in FFmpeg and just accept whatever bytes here, the streams appear to decode just fine on my Alder Lake, although with some dropped frames (possibly for unrelated reasons).

[Sesse]

As an aside: I believe this also explains why seemingly random solutions like “transcode via x264 first” or “use a hardware decoder instead of software decoder” fixes the problem for some; it changes the side data stuck into the AV1 stream (e.g., some of these operations will make you lose color management information), which changes the amount of padding needed and thus works around the bug if you're lucky.

[Timo R.]

FFmpeg does not generate any padding or OBUs itself, the driver does that.
So you need to raise this with Nvidia.

[Sesse]

I agree that this sounds like an NVENC bug, and someone should report it to NVIDIA (I don't have an AV1-capable NV GPU, unfortunately). At the same time, FFmpeg could also work around it by being less restrictive in parsing the padding, like e.g. dav1d seemingly is.

[Sesse]

This seems to be sufficient as a workaround; it still outputs the error, but effectively ignores it:

diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c
index d05352c738..8bf44e1a28 100644
--- a/libavcodec/cbs_av1.c
+++ b/libavcodec/cbs_av1.c
@@ -1014,7 +1014,7 @@ static int cbs_av1_read_unit(CodedBitstreamContext *ctx,
             return AVERROR_INVALIDDATA;
 
         err = cbs_av1_read_trailing_bits(ctx, &gbc, nb_bits);
-        if (err < 0)
+        if (err < 0 && err != AVERROR_INVALIDDATA)
             return err;
     }

If one wants to ignore the error entirely, I guess it's sufficient to just remove the entire cbs_av1_read_trailing_bits() call.

[Timo R.]

James looked into this yesterday, and the result was that nvenc does not generate broken trailing bits.
It instead writes a HEVC time_code SEI message into the AV1 bitstream.
So something is definitely messed up there in the driver.