Opened 10 months ago
Last modified 10 months ago
#10777 new defect
Segfault when using hardware vaapi encoding
| Reported by: | Patrick M | Owned by: | |
|---|---|---|---|
| Priority: | normal | Component: | undetermined |
| Version: | unspecified | Keywords: | |
| Cc: | Blocked By: | ||
| Blocking: | Reproduced by developer: | no | |
| Analyzed by developer: | no |
Description
Summary of the bug: segfault within pick_format of avfiltergraph
How to reproduce: {{{ ffmpeg -vaapi_device /dev/dri/renderD129 -i test.mp4 -vf 'format=nv12,hwupload' -c:v h264_vaapi output.mp4
built on gentoo ffmpeg version N-113120-g545686e49e Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13 (Gentoo Hardened 13.2.1_p20230826 p7)
}}}
Older git version or released versions 6.* work fine one same machines. Tried on two machines.
Attachments (2)
Change History (10)
by , 10 months ago
comment:1 by , 10 months ago
This isn't specific to encoding either, I have been experiencing this problem for decode too. I'm on yesterday's Mesa main commit 49378bc3cda770493c656e0de1b2e2a1e35d79f4 with an integrated AMD Radeon Graphics (RADV GFX1036)
$ gdb --args ~/cts/root/bin/ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.h264 -f null -
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/cturner/cts/root/bin/ffmpeg...
(gdb) r
Starting program: /home/cturner/cts/root/bin/ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.h264 -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
ffmpeg version git-2024-01-03-c5bcf868c4 Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 12 (Debian 12.2.0-14)
configuration: --prefix=/home/cturner/cts/root --disable-doc --disable-shared --enable-static --enable-gpl --enable-nonfree --enable-libx264 --enable-libx265 --enable-vulkan --enable-vaapi --disable-optimizations --extra-cflags=-Og --extra-cflags=-fno-omit-frame-pointer --enable-debug=3 --extra-cflags=-fno-inline
libavutil 58. 36.100 / 58. 36.100
libavcodec 60. 37.100 / 60. 37.100
libavformat 60. 20.100 / 60. 20.100
libavdevice 60. 4.100 / 60. 4.100
libavfilter 9. 16.100 / 9. 16.100
libswscale 7. 6.100 / 7. 6.100
libswresample 4. 13.100 / 4. 13.100
libpostproc 57. 4.100 / 57. 4.100
Input #0, h264, from 'input.h264':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 176x144, 25 fps, 60 tbr, 1200k tbn
[New Thread 0x7fffe99ff6c0 (LWP 4975)]
[New Thread 0x7fffe91fe6c0 (LWP 4976)]
[New Thread 0x7fffe89fd6c0 (LWP 4977)]
[New Thread 0x7fffdbfff6c0 (LWP 4978)]
[New Thread 0x7fffdb1fe6c0 (LWP 4979)]
[New Thread 0x7fffda9fd6c0 (LWP 4980)]
[New Thread 0x7fffda1fc6c0 (LWP 4981)]
[New Thread 0x7fffd99fb6c0 (LWP 4982)]
[New Thread 0x7fffd91fa6c0 (LWP 4983)]
[New Thread 0x7fffd89f96c0 (LWP 4984)]
[New Thread 0x7fffd3fff6c0 (LWP 4985)]
[New Thread 0x7fffd37fe6c0 (LWP 4986)]
[New Thread 0x7fffd2ffd6c0 (LWP 4987)]
[New Thread 0x7fffd27fc6c0 (LWP 4988)]
[New Thread 0x7fffd1ffb6c0 (LWP 4989)]
[New Thread 0x7fffd17fa6c0 (LWP 4990)]
[New Thread 0x7fffd0ff96c0 (LWP 4991)]
[New Thread 0x7fffd07f86c0 (LWP 4992)]
[New Thread 0x7fffcfff76c0 (LWP 4993)]
[New Thread 0x7fffcf7f66c0 (LWP 4994)]
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
[New Thread 0x7fffceff56c0 (LWP 4995)]
[New Thread 0x7fffce7f46c0 (LWP 4996)]
[New Thread 0x7fffcdff36c0 (LWP 4997)]
[New Thread 0x7fffcd7f26c0 (LWP 4998)]
Press [q] to stop, [?] for help
[New Thread 0x7fffccbf16c0 (LWP 4999)]
[New Thread 0x7fff7ffff6c0 (LWP 5000)]
[New Thread 0x7fff7f7fe6c0 (LWP 5001)]
[New Thread 0x7fff7effd6c0 (LWP 5002)]
[New Thread 0x7fff7e7fc6c0 (LWP 5003)]
[New Thread 0x7fff7dffb6c0 (LWP 5004)]
[New Thread 0x7fff7d7fa6c0 (LWP 5005)]
[New Thread 0x7fff7cff96c0 (LWP 5006)]
[New Thread 0x7fff7c7f86c0 (LWP 5007)]
[New Thread 0x7fff7bff76c0 (LWP 5008)]
[New Thread 0x7fff7b7f66c0 (LWP 5009)]
[New Thread 0x7fff7aff56c0 (LWP 5010)]
[New Thread 0x7fff7a7f46c0 (LWP 5011)]
[New Thread 0x7fff79ff36c0 (LWP 5012)]
[New Thread 0x7fff797f26c0 (LWP 5013)]
Thread 23 "vf#0:0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffce7f46c0 (LWP 4996)]
0x00005555556d9108 in pick_format (link=0x7fffc0004cc0, ref=ref@entry=0x0) at libavfilter/avfiltergraph.c:671
671 swfmt = ((AVHWFramesContext *) link->hw_frames_ctx->data)->sw_format;
(gdb) bt full
#0 0x00005555556d9108 in pick_format (link=0x7fffc0004cc0, ref=ref@entry=0x0) at libavfilter/avfiltergraph.c:671
swfmt = AV_PIX_FMT_VAAPI
#1 0x00005555556d9363 in pick_formats (graph=graph@entry=0x7fffc00012c0) at libavfilter/avfiltergraph.c:1204
filter = 0x7fffc0003780
i = 0
j = 0
ret = <optimized out>
change = 0
#2 0x00005555556d954a in graph_config_formats (graph=graph@entry=0x7fffc00012c0, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:1273
ret = <optimized out>
#3 0x00005555556d957e in avfilter_graph_config (graphctx=0x7fffc00012c0, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:1326
ret = <optimized out>
#4 0x000055555568aa5a in configure_filtergraph (fg=fg@entry=0x555557f51180, fgt=fgt@entry=0x7fffce7f3d50) at fftools/ffmpeg_filter.c:1765
fgp = 0x555557f51180
hw_device = <optimized out>
inputs = 0x0
outputs = 0x0
cur = 0x0
ret = 0
i = 1
simple = <optimized out>
have_input_eof = 0
graph_desc = <optimized out>
#5 0x000055555568af60 in send_frame (fg=fg@entry=0x555557f51180, fgt=fgt@entry=0x7fffce7f3d50, ifilter=ifilter@entry=0x555557f16000, frame=frame@entry=0x7fffc0000900)
at fftools/ffmpeg_filter.c:2667
tmp = 0x0
ifp = 0x555557f16000
fd = <optimized out>
sd = <optimized out>
need_reinit = <optimized out>
ret = 0
#6 0x000055555568b3c3 in filter_thread (arg=0x555557f51180) at fftools/ffmpeg_filter.c:2828
ifilter = 0x555557f16000
ifp = <optimized out>
o = 0
input_idx = 0
fgp = 0x555557f51180
fg = 0x555557f51180
fgt = {graph = 0x7fffc00012c0, frame = 0x7fffc0000900, frame_queue_out = 0x7fffc0000f40, next_in = 0, got_frame = 0, eof_in = 0x7fffc0000b40 "",
eof_out = 0x7fffc0000c00 ""}
ret = <optimized out>
input_status = <optimized out>
#7 0x000055555569fcde in task_wrapper (arg=0x555557f51768) at fftools/ffmpeg_sched.c:2200
task = 0x555557f51768
sch = 0x555557f13f80
ret = <optimized out>
err = 0
#8 0x00007ffff64a8044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736657835712, -972534296838396404, -136, 0, 140737488347840, 140736649445376, 972568388114064908, 972550613396235788},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#9 0x00007ffff652861c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
comment:2 by , 10 months ago
I bisected this issue to
commit 8c7934f73ab6c568acaa47c821a6833f9145fdbb
Author: Niklas Haas <git@haasn.dev>
Date: Sun Dec 31 13:35:03 2023 -0800
avfilter: add negotiation API for color space/range
Motivated by YUVJ removal. This change will allow full negotiation
between color ranges and matrices as needed. By default, all ranges and
matrices are marked as supported.
Because grayscale formats are currently handled very inconsistently (and
in particular, assumed as forced full-range by swscale), we exclude them
from negotiation altogether for the time being, to get this API merged.
After filter negotiation is available, we can relax the
grayscale-is-forced-jpeg restriction again, when it will be more
feasible to do so without breaking a million test cases.
Note that this commit updates one FATE test as a consequence of the
sanity fallback for non-YUV formats. In particular, the test case now
writes rgb24(pc, gbr/unspecified/unspecified) to the matroska file,
instead of rgb24(unspecified/unspecified/unspecified) as before.
It would be nice to have some automated CI for the h/w acceleration. We'll hopefully have something like this at my company soon. Not sure if FATE is suitable for tests requiring a GPU.
comment:3 by , 10 months ago
This is not limited to vaapi encoding, but also affects any hardware accelerated filters that I tested: overlay_cuda, bilateral_cuda, overlay_opencl, overlay_qsv, overlay_vaapi and overlay_vulkan.
My git bisect also points to 8c7934f73ab6c568acaa47c821a6833f9145fdbb as being the offending commit.
comment:4 by , 10 months ago
seems commented out this part, the issue has gone.
diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c
index 727eff81ee..324f178cdd 100644
--- a/libavfilter/avfiltergraph.c
+++ b/libavfilter/avfiltergraph.c
@@ -666,11 +666,12 @@ static int pick_format(AVFilterLink *link, AVFilterLink *ref)
if (link->type == AVMEDIA_TYPE_VIDEO) {
enum AVPixelFormat swfmt = link->format;
+#if 0
if (av_pix_fmt_desc_get(swfmt)->flags & AV_PIX_FMT_FLAG_HWACCEL) {
av_assert1(link->hw_frames_ctx);
swfmt = ((AVHWFramesContext *) link->hw_frames_ctx->data)->sw_format;
}
-
+#endif
if (!ff_fmt_is_regular_yuv(swfmt)) {
const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(swfmt);
/* These fields are explicitly documented as affecting YUV only,
comment:5 by , 10 months ago
A merged hack commit already fixed this:
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/8b76bae8961c3045fa73aa2da47345c70a93d599
gdb content with backtrace