pullup crashes with gray8

[Timothy Gu]

Summary of the bug

pullup filter crashes with gray8 pixel format. I discovered this bug while trying to add FATE test for pullup. The input doesn't seem to matter -- it always crashes.

How to reproduce

timothy_gu@ubuntu-lenovo:~/ffmpeg$ ./ffmpeg -f lavfi -i testsrc -vf pullup -pix_fmt gray -f null -
ffmpeg version N-61480-g6c47a4e Copyright (c) 2000-2014 the FFmpeg developers
  built on Mar 15 2014 19:21:43 with gcc 4.8 (Ubuntu/Linaro 4.8.1-10ubuntu9)
  configuration: --enable-gpl
  libavutil      52. 66.101 / 52. 66.101
  libavcodec     55. 52.102 / 55. 52.102
  libavformat    55. 34.101 / 55. 34.101
  libavdevice    55. 11.100 / 55. 11.100
  libavfilter     4.  3.100 /  4.  3.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf55.34.101
    Stream #0:0: Video: rawvideo (Y800 / 0x30303859), gray, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
Segmentation fault (core dumped)

GDB output

timothy_gu@ubuntu-lenovo:~/ffmpeg$ gdb ffmpeg_g
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/timothy_gu/ffmpeg/ffmpeg_g...done.
(gdb) r -loglevel 99 -f lavfi -i testsrc -vf pullup -pix_fmt gray -f null -
Starting program: /home/timothy_gu/ffmpeg/ffmpeg_g -loglevel 99 -f lavfi -i testsrc -vf pullup -pix_fmt gray -f null -
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
ffmpeg version N-61480-g6c47a4e Copyright (c) 2000-2014 the FFmpeg developers
  built on Mar 15 2014 19:21:43 with gcc 4.8 (Ubuntu/Linaro 4.8.1-10ubuntu9)
  configuration: --enable-gpl
  libavutil      52. 66.101 / 52. 66.101
  libavcodec     55. 52.102 / 55. 52.102
  libavformat    55. 34.101 / 55. 34.101
  libavdevice    55. 11.100 / 55. 11.100
  libavfilter     4.  3.100 /  4.  3.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'lavfi'.
Reading option '-i' ... matched as input file with argument 'testsrc'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'pullup'.
Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'gray'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'null'.
Reading option '-' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument 99.
Successfully parsed a group of options.
Parsing a group of options: input file testsrc.
Applying option f (force format) with argument lavfi.
Successfully parsed a group of options.
Opening an input file: testsrc.
detected 4 logical cores
[New Thread 0x7ffff27e9700 (LWP 14267)]
[New Thread 0x7ffff1fe8700 (LWP 14268)]
[New Thread 0x7ffff17e7700 (LWP 14269)]
[New Thread 0x7ffff0fe6700 (LWP 14270)]
[New Thread 0x7ffff07e5700 (LWP 14271)]
[Parsed_testsrc_0 @ 0x17d3e40] size:320x240 rate:25/1 duration:-1.000000 sar:1/1
[AVFilterGraph @ 0x17d3500] query_formats: 2 queried, 1 merged, 0 already done, 0 delayed
[lavfi @ 0x17d2900] All info found
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0, 1, 1/25: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 1/25, 25 tbr, 25 tbn, 25 tbc
Successfully opened the file.
Parsing a group of options: output file -.
Applying option vf (set video filters) with argument pullup.
Applying option pix_fmt (set pixel format) with argument gray.
Applying option f (force format) with argument null.
Successfully parsed a group of options.
Opening an output file: -.
Successfully opened the file.
[New Thread 0x7fffeffe4700 (LWP 14272)]
[New Thread 0x7fffef7e3700 (LWP 14273)]
[New Thread 0x7fffeefe2700 (LWP 14274)]
[New Thread 0x7fffee7e1700 (LWP 14275)]
[New Thread 0x7fffedfe0700 (LWP 14276)]
[graph 0 input from stream 0:0 @ 0x17bcd60] Setting 'video_size' to value '320x240'
[graph 0 input from stream 0:0 @ 0x17bcd60] Setting 'pix_fmt' to value '2'
[graph 0 input from stream 0:0 @ 0x17bcd60] Setting 'time_base' to value '1/25'
[graph 0 input from stream 0:0 @ 0x17bcd60] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0x17bcd60] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0x17bcd60] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0x17bcd60] w:320 h:240 pixfmt:rgb24 tb:1/25 fr:25/1 sar:1/1 sws_param:flags=2
[format @ 0x17bd860] compat: called with args=[gray]
[format @ 0x17bd860] Setting 'pix_fmts' to value 'gray'
[auto-inserted scaler 0 @ 0x17beb60] Setting 'flags' to value '0x4'
[auto-inserted scaler 0 @ 0x17beb60] w:iw h:ih flags:'0x4' interl:0
[Parsed_pullup_0 @ 0x17bc5e0] auto-inserting filter 'auto-inserted scaler 0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_pullup_0'
[AVFilterGraph @ 0x17cafe0] query_formats: 4 queried, 2 merged, 1 already done, 0 delayed
[auto-inserted scaler 0 @ 0x17beb60] w:320 h:240 fmt:rgb24 sar:1/1 -> w:320 h:240 fmt:gray sar:1/1 flags:0x4
[Parsed_pullup_0 @ 0x17bc5e0] w: 38 h: 28
[Parsed_pullup_0 @ 0x17bc5e0] offset: 2568 length: 1064
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf55.34.101
    Stream #0:0, 0, 1/90000: Video: rawvideo (Y800 / 0x30303859), gray, 320x240 [SAR 1:1 DAR 4:3], 1/25, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
0x0000000000bf80ee in memcpy (__len=1024, __src=<optimized out>, 
    __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:51
51	  return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
(gdb) bt
#0  0x0000000000bf80ee in memcpy (__len=1024, __src=<optimized out>, 
    __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#1  av_image_copy (dst_data=dst_data@entry=0x17bc778, 
    dst_linesizes=dst_linesizes@entry=0x17bc738, src_data=<optimized out>, 
    src_linesizes=<optimized out>, pix_fmt=<optimized out>, 
    width=<optimized out>, height=240) at libavutil/imgutils.c:272
#2  0x00000000004c17d5 in filter_frame (inlink=inlink@entry=0x17bf460, 
    in=0x17ee0a0) at libavfilter/vf_pullup.c:676
#3  0x000000000048ac7e in ff_filter_frame_framed (link=link@entry=0x17bf460, 
    frame=frame@entry=0x17ee0a0) at libavfilter/avfilter.c:1081
#4  0x000000000048bb5c in ff_filter_frame (link=link@entry=0x17bf460, 
    frame=frame@entry=0x17ee0a0) at libavfilter/avfilter.c:1161
#5  0x00000000004c4adf in filter_frame (link=link@entry=0x17bd720, in=0x0)
    at libavfilter/vf_scale.c:524
#6  0x000000000048ac7e in ff_filter_frame_framed (link=link@entry=0x17bd720, 
    frame=frame@entry=0x17ede00) at libavfilter/avfilter.c:1081
#7  0x000000000048bb5c in ff_filter_frame (link=link@entry=0x17bd720, 
    frame=0x17ede00) at libavfilter/avfilter.c:1161
#8  0x0000000000490382 in request_frame (link=0x17bd720)
    at libavfilter/buffersrc.c:500
#9  0x0000000000490098 in av_buffersrc_add_frame_internal (
    ctx=ctx@entry=0x17bcd60, frame=frame@entry=0x17ed860, flags=flags@entry=4)
    at libavfilter/buffersrc.c:181
---Type <return> to continue, or q <return> to quit---
#10 0x00000000004905ad in av_buffersrc_add_frame_flags (ctx=0x17bcd60, 
    frame=frame@entry=0x17ed860, flags=flags@entry=4)
    at libavfilter/buffersrc.c:106
#11 0x000000000047b60a in decode_video (ist=ist@entry=0x17d2fa0, 
    pkt=pkt@entry=0x7fffffffdb10, got_output=got_output@entry=0x7fffffffdaac)
    at ffmpeg.c:1835
#12 0x000000000046537b in output_packet (pkt=0x7fffffffdab0, ist=0x17d2fa0)
    at ffmpeg.c:1966
#13 process_input (file_index=<optimized out>) at ffmpeg.c:3296
#14 transcode_step () at ffmpeg.c:3390
#15 transcode () at ffmpeg.c:3442
#16 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3622

[Carl Eugen Hoyos]

Reproducible with:

$ ffmpeg -f lavfi -i testsrc -vf format=gray,pullup -f null -

[Carl Eugen Hoyos]

Not a regression.

[Michael Niedermayer]

Fixed in e818ee090ac53d1b333a7d6a45274f75cf1a71a1