Segfault when using hardware vaapi encoding

[Patrick M]

Summary of the bug: segfault within pick_format of avfiltergraph
How to reproduce: {{{ ffmpeg -vaapi_device /dev/dri/renderD129 -i test.mp4 -vf 'format=nv12,hwupload' -c:v h264_vaapi output.mp4
built on gentoo ffmpeg version N-113120-g545686e49e Copyright (c) 2000-2023 the FFmpeg developers

built with gcc 13 (Gentoo Hardened 13.2.1_p20230826 p7)

}}}
Older git version or released versions 6.* work fine one same machines. Tried on two machines.

[Patrick M]

gdb content with backtrace

[Patrick M]

valgrind output

[Charlie Turner]

This isn't specific to encoding either, I have been experiencing this problem for decode too. I'm on yesterday's Mesa main commit 49378bc3cda770493c656e0de1b2e2a1e35d79f4 with an integrated AMD Radeon Graphics (RADV GFX1036)

$ gdb --args ~/cts/root/bin/ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.h264 -f null -
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/cturner/cts/root/bin/ffmpeg...
(gdb) r
Starting program: /home/cturner/cts/root/bin/ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.h264 -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
ffmpeg version git-2024-01-03-c5bcf868c4 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/home/cturner/cts/root --disable-doc --disable-shared --enable-static --enable-gpl --enable-nonfree --enable-libx264 --enable-libx265 --enable-vulkan --enable-vaapi --disable-optimizations --extra-cflags=-Og --extra-cflags=-fno-omit-frame-pointer --enable-debug=3 --extra-cflags=-fno-inline
  libavutil      58. 36.100 / 58. 36.100
  libavcodec     60. 37.100 / 60. 37.100
  libavformat    60. 20.100 / 60. 20.100
  libavdevice    60.  4.100 / 60.  4.100
  libavfilter     9. 16.100 /  9. 16.100
  libswscale      7.  6.100 /  7.  6.100
  libswresample   4. 13.100 /  4. 13.100
  libpostproc    57.  4.100 / 57.  4.100
Input #0, h264, from 'input.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 176x144, 25 fps, 60 tbr, 1200k tbn
[New Thread 0x7fffe99ff6c0 (LWP 4975)]
[New Thread 0x7fffe91fe6c0 (LWP 4976)]
[New Thread 0x7fffe89fd6c0 (LWP 4977)]
[New Thread 0x7fffdbfff6c0 (LWP 4978)]
[New Thread 0x7fffdb1fe6c0 (LWP 4979)]
[New Thread 0x7fffda9fd6c0 (LWP 4980)]
[New Thread 0x7fffda1fc6c0 (LWP 4981)]
[New Thread 0x7fffd99fb6c0 (LWP 4982)]
[New Thread 0x7fffd91fa6c0 (LWP 4983)]
[New Thread 0x7fffd89f96c0 (LWP 4984)]
[New Thread 0x7fffd3fff6c0 (LWP 4985)]
[New Thread 0x7fffd37fe6c0 (LWP 4986)]
[New Thread 0x7fffd2ffd6c0 (LWP 4987)]
[New Thread 0x7fffd27fc6c0 (LWP 4988)]
[New Thread 0x7fffd1ffb6c0 (LWP 4989)]
[New Thread 0x7fffd17fa6c0 (LWP 4990)]
[New Thread 0x7fffd0ff96c0 (LWP 4991)]
[New Thread 0x7fffd07f86c0 (LWP 4992)]
[New Thread 0x7fffcfff76c0 (LWP 4993)]
[New Thread 0x7fffcf7f66c0 (LWP 4994)]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
[New Thread 0x7fffceff56c0 (LWP 4995)]
[New Thread 0x7fffce7f46c0 (LWP 4996)]
[New Thread 0x7fffcdff36c0 (LWP 4997)]
[New Thread 0x7fffcd7f26c0 (LWP 4998)]
Press [q] to stop, [?] for help
[New Thread 0x7fffccbf16c0 (LWP 4999)]
[New Thread 0x7fff7ffff6c0 (LWP 5000)]
[New Thread 0x7fff7f7fe6c0 (LWP 5001)]
[New Thread 0x7fff7effd6c0 (LWP 5002)]
[New Thread 0x7fff7e7fc6c0 (LWP 5003)]
[New Thread 0x7fff7dffb6c0 (LWP 5004)]
[New Thread 0x7fff7d7fa6c0 (LWP 5005)]
[New Thread 0x7fff7cff96c0 (LWP 5006)]
[New Thread 0x7fff7c7f86c0 (LWP 5007)]
[New Thread 0x7fff7bff76c0 (LWP 5008)]
[New Thread 0x7fff7b7f66c0 (LWP 5009)]
[New Thread 0x7fff7aff56c0 (LWP 5010)]
[New Thread 0x7fff7a7f46c0 (LWP 5011)]
[New Thread 0x7fff79ff36c0 (LWP 5012)]
[New Thread 0x7fff797f26c0 (LWP 5013)]

Thread 23 "vf#0:0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffce7f46c0 (LWP 4996)]
0x00005555556d9108 in pick_format (link=0x7fffc0004cc0, ref=ref@entry=0x0) at libavfilter/avfiltergraph.c:671
671	            swfmt = ((AVHWFramesContext *) link->hw_frames_ctx->data)->sw_format;
(gdb) bt full
#0  0x00005555556d9108 in pick_format (link=0x7fffc0004cc0, ref=ref@entry=0x0) at libavfilter/avfiltergraph.c:671
        swfmt = AV_PIX_FMT_VAAPI
#1  0x00005555556d9363 in pick_formats (graph=graph@entry=0x7fffc00012c0) at libavfilter/avfiltergraph.c:1204
        filter = 0x7fffc0003780
        i = 0
        j = 0
        ret = <optimized out>
        change = 0
#2  0x00005555556d954a in graph_config_formats (graph=graph@entry=0x7fffc00012c0, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:1273
        ret = <optimized out>
#3  0x00005555556d957e in avfilter_graph_config (graphctx=0x7fffc00012c0, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:1326
        ret = <optimized out>
#4  0x000055555568aa5a in configure_filtergraph (fg=fg@entry=0x555557f51180, fgt=fgt@entry=0x7fffce7f3d50) at fftools/ffmpeg_filter.c:1765
        fgp = 0x555557f51180
        hw_device = <optimized out>
        inputs = 0x0
        outputs = 0x0
        cur = 0x0
        ret = 0
        i = 1
        simple = <optimized out>
        have_input_eof = 0
        graph_desc = <optimized out>
#5  0x000055555568af60 in send_frame (fg=fg@entry=0x555557f51180, fgt=fgt@entry=0x7fffce7f3d50, ifilter=ifilter@entry=0x555557f16000, frame=frame@entry=0x7fffc0000900)
    at fftools/ffmpeg_filter.c:2667
        tmp = 0x0
        ifp = 0x555557f16000
        fd = <optimized out>
        sd = <optimized out>
        need_reinit = <optimized out>
        ret = 0
#6  0x000055555568b3c3 in filter_thread (arg=0x555557f51180) at fftools/ffmpeg_filter.c:2828
        ifilter = 0x555557f16000
        ifp = <optimized out>
        o = 0
        input_idx = 0
        fgp = 0x555557f51180
        fg = 0x555557f51180
        fgt = {graph = 0x7fffc00012c0, frame = 0x7fffc0000900, frame_queue_out = 0x7fffc0000f40, next_in = 0, got_frame = 0, eof_in = 0x7fffc0000b40 "", 
          eof_out = 0x7fffc0000c00 ""}
        ret = <optimized out>
        input_status = <optimized out>
#7  0x000055555569fcde in task_wrapper (arg=0x555557f51768) at fftools/ffmpeg_sched.c:2200
        task = 0x555557f51768
        sch = 0x555557f13f80
        ret = <optimized out>
        err = 0
#8  0x00007ffff64a8044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736657835712, -972534296838396404, -136, 0, 140737488347840, 140736649445376, 972568388114064908, 972550613396235788}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#9  0x00007ffff652861c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.

[Charlie Turner]

I bisected this issue to

commit 8c7934f73ab6c568acaa47c821a6833f9145fdbb
Author: Niklas Haas <git@haasn.dev>
Date:   Sun Dec 31 13:35:03 2023 -0800

    avfilter: add negotiation API for color space/range
    
    Motivated by YUVJ removal. This change will allow full negotiation
    between color ranges and matrices as needed. By default, all ranges and
    matrices are marked as supported.
    
    Because grayscale formats are currently handled very inconsistently (and
    in particular, assumed as forced full-range by swscale), we exclude them
    from negotiation altogether for the time being, to get this API merged.
    
    After filter negotiation is available, we can relax the
    grayscale-is-forced-jpeg restriction again, when it will be more
    feasible to do so without breaking a million test cases.
    
    Note that this commit updates one FATE test as a consequence of the
    sanity fallback for non-YUV formats. In particular, the test case now
    writes rgb24(pc, gbr/unspecified/unspecified) to the matroska file,
    instead of rgb24(unspecified/unspecified/unspecified) as before.

It would be nice to have some automated CI for the h/w acceleration. We'll hopefully have something like this at my company soon. Not sure if FATE is suitable for tests requiring a GPU.

[bermond]

This is not limited to vaapi encoding, but also affects any hardware accelerated filters that I tested: overlay_cuda, bilateral_cuda, overlay_opencl, overlay_qsv, overlay_vaapi and overlay_vulkan.

My git bisect also points to 8c7934f73ab6c568acaa47c821a6833f9145fdbb as being the offending commit.

[Ruijing Dong]

seems commented out this part, the issue has gone.

---

diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c
index 727eff81ee..324f178cdd 100644
--- a/libavfilter/avfiltergraph.c
+++ b/libavfilter/avfiltergraph.c
@@ -666,11 +666,12 @@ static int pick_format(AVFilterLink *link, AVFilterLink *ref)

if (link->type == AVMEDIA_TYPE_VIDEO) {

enum AVPixelFormat swfmt = link->format;

+#if 0

if (av_pix_fmt_desc_get(swfmt)->flags & AV_PIX_FMT_FLAG_HWACCEL) {

av_assert1(link->hw_frames_ctx);
swfmt = ((AVHWFramesContext *) link->hw_frames_ctx->data)->sw_format;

}

-
+#endif

if (!ff_fmt_is_regular_yuv(swfmt)) {

const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(swfmt);
/* These fields are explicitly documented as affecting YUV only,

[Fei Wang]

A merged hack commit already fixed this:
​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/8b76bae8961c3045fa73aa2da47345c70a93d599

[Patrick M]

Thanks for the fix, works for me on a quick test using vaapi.

[Ruijing Dong]

Thanks for the information!

[Charlie Turner]

The hack works for my testing as well, thanks!