Regression: hevcdec with videotoolbox hwaccel aborts on broken frames after commit 79afc45c03abfb7f1d9dc2cc758ec162bfade1a0

[gnattu]

Summary of the bug:

Since commit 79afc45c03abfb7f1d9dc2cc758ec162bfade1a0 (lavc/hevcdec: use a ContainerFifo to hold frames scheduled for output), the behavior of the HEVC decoder (hevcdec) with videotoolbox hardware acceleration has changed. When decoding input files that contain broken frames, the decoder now aborts early instead of continuing to decode subsequent good frames, as it did before.

This is verified by bisecting from 7.0.2 to 7.1.1:

79afc45c03abfb7f1d9dc2cc758ec162bfade1a0 is the first bad commit
commit 79afc45c03abfb7f1d9dc2cc758ec162bfade1a0
Author: Anton Khirnov <anton@khirnov.net>
Date:   Thu Jun 20 21:01:49 2024 +0200

    lavc/hevcdec: use a ContainerFifo to hold frames scheduled for output

    Instead of a single AVFrame.

    Will be useful in future commits, where we will want to produce multiple
    output frames for a single coded frame.

 libavcodec/Makefile       |  2 +-
 libavcodec/hevc/hevcdec.c | 31 +++++++++++++++++++++----------
 libavcodec/hevc/hevcdec.h |  4 ++--
 libavcodec/hevc/refs.c    | 12 ++++--------
 4 files changed, 28 insertions(+), 21 deletions(-)

How to reproduce:

1. Download this sample, which contains broken frames at the beginning:

​https://drive.usercontent.google.com/download?id=1z5dRzJ1PMSTphUqXa379Ourm1OdARIO_&export=download&authuser=0

2. Run the following command:

./ffmpeg -hide_banner -hwaccel videotoolbox -hwaccel_output_format videotoolbox_vld -i './LG Cymatic Jazz 4K Demo.ts' -vcodec h264_videotoolbox -an -sn -dn test.mp4

You should observe the bug, where decoding aborts after encountering broken frames instead of continuing:

[hevc @ 0x142e1e860] No frame decoded?
[vist#0:0/hevc @ 0x144004b90] [dec:hevc @ 0x142e17820] Error submitting packet to decoder: Generic error in an external library
[vist#0:0/hevc @ 0x144004b90] [dec:hevc @ 0x142e17820] Error submitting packet to decoder: Unknown error occurred
[vist#0:0/hevc @ 0x144004b90] [dec:hevc @ 0x142e17820] A decoder returned an unexpected error code. This is a bug, please report it.
[vist#0:0/hevc @ 0x144004b90] [dec:hevc @ 0x142e17820] Error processing packet in decoder: Internal bug, should not have happened
[vist#0:0/hevc @ 0x144004b90] [dec:hevc @ 0x142e17820] Task finished with error code: -558323010 (Internal bug, should not have happened)
[vist#0:0/hevc @ 0x144004b90] [dec:hevc @ 0x142e17820] Terminating thread with return code -558323010 (Internal bug, should not have happened)

In 7.0.2, it will continue decoding past the broken frames, despite the log is also flooded with decode errors:

[hevc @ 0x150e1efa0] Could not find ref with POC 38
[hevc @ 0x150e175b0] vt decoder cb: output image buffer is null: -17694
[hevc @ 0x150e1efa0] hardware accelerator failed to decode picture
[vist#0:0/hevc @ 0x150e08d30] [dec:hevc @ 0x150e16020] Error submitting packet to decoder: Unknown error occurred
    Last message repeated 7 times  

A temporary workaround is to change line 127 of videotoolbox.c to not return AVERROR_EXTERNAL if no frame was decoded, but to return 0 after unreffing the frame. This allows decoding to continue with invalid frame warning:

[hevc @ 0x11a604f40] An invalid frame was output by a decoder. This is a bug, please report it.

But this is probably not the best way to restore the old behavior, and therefore I created this ticket.