HandBrake 1.0.3 segfaults or encodes 0 frames with FFmpeg 3.1.7 (works with 3.2.4)

[Dominik Mierzejewski]

Summary of the bug:
HandBrake 1.0.3 segfaults in decomb filter (when encoding anything using default settings) or encodes 0 frames (if I disable the filter in video filters tab) when built with FFmpeg 3.1.7 (works with 3.2.4)
How to reproduce:

Case 1:

1. (On Fedora 25,) build HandBrake 1.0.3 against FFmpeg 3.1.7.
2. In ghb, open any video file and click "start encoding".
3. ghb segfaults in decomb filter:

[...]
Thread 27 "ghb" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe1016700 (LWP 31173)]
detect_gamma_combed_segment (segment_stop=204, segment_start=0, pv=0x7fffd80008c0) at ../libhb/comb_detect.c:443
443	        int height  = pv->ref[0]->plane[pp].height;
(gdb) bt full
#0  detect_gamma_combed_segment (segment_stop=204, segment_start=0, pv=0x7fffd80008c0) at ../libhb/comb_detect.c:443
        y = <optimized out>
        x = <optimized out>
        stride = <optimized out>
        width = <optimized out>
        height = -671059792
        mthresh = <optimized out>
        athresh = <optimized out>
        athresh6 = <optimized out>
        pp = <optimized out>
#1  decomb_filter_thread (thread_args_v=0x7fffd801a870) at ../libhb/comb_detect.c:1061
        pp = 0
        pv = <optimized out>
        segment = 0
        segment_start = 0
        segment_stop = 204
        thread_args = 0x7fffd801a870
#2  0x000000000046887b in hb_thread_func (_t=0x7fffd8001070) at ../libhb/ports.c:860
        t = 0x7fffd8001070
#3  0x00007ffff25006ca in start_thread (arg=0x7fffe1016700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fffe1016700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736968353536, 3440324452558995082, 0, 140736250525967, 
                140736968354240, 140736968353536, -3440258779170570614, -3440300979176309110}, mask_was_saved = 0}}, 
          priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fffed607f7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
No locals.

Case 2:

1. (On Fedora 25,) build HandBrake 1.0.3 against FFmpeg 3.1.7.
2. In ghb, open any video file.
3. Disable deinterlacing filters in video filtering tab.
4. Click "start encoding".
5. ghb doesn't crash, but encodes 0 video frames.

All the above works fine if built against FFmpeg 3.2.4.

HandBrake upstream has already said using FFmpeg is an unsupported configuration as they support libav only: ​https://github.com/HandBrake/HandBrake/issues/631 . However, I'd appreciate any help in identifying where the bug is.

[Carl Eugen Hoyos]

Please bisect to find out which commit fixed the crash.

[Carl Eugen Hoyos]

Please reopen this ticket if you can provide us with the relevant commit.