H.261 encoding crashes with trellis

[Maik Merten]

Summary of the bug:
How to reproduce:

% ffmpeg -i /tmp/test.y4m -f h261 -vb 256k -trellis 2 /tmp/test.h261
ffmpeg version N-58112-g5592d1b Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 15 2013 18:16:18 with gcc 4.8 (Ubuntu/Linaro 4.8.1-10ubuntu8)

There's some revived interest in H.261 as possible guaranteed-patent-free baseline codec for videocommunication. While ffmpeg's H.261 usually works fine, it does not work with advanced techniques such as trellis quantization (or most rate/distortion options, for that matter). This is about trellis.

It would be awesome if some of the cool advanced encoder features that are available for MPEG-1 (e.g., trellis) would also work for H.261. Think "pig with rockets attached". If this is not possible the encoder at least should not crash.

Program received signal SIGSEGV, Segmentation fault.
0x000000000083536a in dct_quantize_trellis_c (s=0x16342a0, block=0x1626d60, 
    n=1, qscale=4, overflow=<optimized out>) at libavcodec/mpegvideo_enc.c:3619
3619	                    int score= distortion + length[UNI_AC_ENC_INDEX(run, level)]*lambda;
(gdb) bt
#0  0x000000000083536a in dct_quantize_trellis_c (s=0x16342a0, 
    block=0x1626d60, n=1, qscale=4, overflow=<optimized out>)
    at libavcodec/mpegvideo_enc.c:3619
#1  0x0000000000842d13 in encode_mb_internal (mb_block_count=6, 
    mb_block_width=8, mb_block_height=8, motion_y=0, motion_x=0, s=0x16342a0)
    at libavcodec/mpegvideo_enc.c:2060
#2  encode_mb (motion_y=0, motion_x=0, s=0x16342a0)
    at libavcodec/mpegvideo_enc.c:2168
#3  encode_thread (c=<optimized out>, arg=<optimized out>)
    at libavcodec/mpegvideo_enc.c:3042
#4  0x000000000090d8bf in avcodec_default_execute (c=0x1633b20, 
    func=0x83f8b0 <encode_thread>, arg=<optimized out>, ret=<optimized out>, 
    count=1, size=8) at libavcodec/utils.c:1016
#5  0x000000000083a2c7 in encode_picture (picture_number=0, s=0x16342a0)
    at libavcodec/mpegvideo_enc.c:3435
#6  ff_MPV_encode_picture (avctx=0x1633b20, pkt=0x7fffffffd910, 
    pic_arg=<optimized out>, got_packet=0x7fffffffd90c)
    at libavcodec/mpegvideo_enc.c:1494
#7  0x000000000090e6a5 in avcodec_encode_video2 (avctx=avctx@entry=0x1633b20, 
    avpkt=avpkt@entry=0x7fffffffd910, frame=frame@entry=0x1628700, 
    got_packet_ptr=got_packet_ptr@entry=0x7fffffffd90c)
    at libavcodec/utils.c:1861
#8  0x000000000046da85 in do_video_out (in_picture=0x1628700, ost=0x1633f80, 
---Type <return> to continue, or q <return> to quit---
    s=0x1633200) at ffmpeg.c:953
#9  reap_filters () at ffmpeg.c:1098
#10 0x000000000045d1df in transcode_step () at ffmpeg.c:3223
#11 transcode () at ffmpeg.c:3266
#12 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3444

[Carl Eugen Hoyos]

Is this reproducible with -f lavfi -i testsrc (or tests/lena.pnm) as input or does it need a specific file to reproduce the crash?
Could you test a very old version (0.5 or 0.7)?

[Maik Merten]

I'm testing with uncompressed test sequences stored in YUV4MPEG format. Suitable input is e.g.

​http://media.xiph.org/video/derf/y4m/mad900_cif.y4m (this sequence is so totally 90ies).

Will try to test an old version.

[Maik Merten]

Version 0.5.13 behaves in a similar way, crashes at the same statement in the corresponding file.

Starting program: /home/maik/builds/ffmpeg-0.5.13/ffmpeg_g -i /tmp/test.y4m -f h261 -vb 256k -trellis 2 /tmp/test.h261
FFmpeg version 0.5.13, Copyright (c) 2000-2013 Fabrice Bellard, et al.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000521add in dct_quantize_trellis_c (s=0xe9d6e0, block=0xeadeb0, n=1, qscale=4, overflow=<optimized out>) at libavcodec/mpegvideo_enc.c:3100
3100	                    int score= distortion + length[UNI_AC_ENC_INDEX(run, level)]*lambda;
(gdb) bt
#0  0x0000000000521add in dct_quantize_trellis_c (s=0xe9d6e0, block=0xeadeb0, n=1, qscale=4, overflow=<optimized out>) at libavcodec/mpegvideo_enc.c:3100
#1  0x0000000000535f6f in encode_mb_internal (mb_block_count=6, mb_block_height=8, motion_y=0, motion_x=0, s=0xe9d6e0) at libavcodec/mpegvideo_enc.c:1635
#2  encode_mb (motion_y=0, motion_x=0, s=0xe9d6e0) at libavcodec/mpegvideo_enc.c:1731
#3  encode_thread (c=<optimized out>, arg=<optimized out>) at libavcodec/mpegvideo_enc.c:2554
#4  0x00000000004d623f in avcodec_default_execute (c=0xe94f70, func=0x532ad0 <encode_thread>, arg=<optimized out>, ret=<optimized out>, count=1, size=8)
    at libavcodec/utils.c:399
#5  0x0000000000524e51 in encode_picture (picture_number=0, s=0xe9d6e0) at libavcodec/mpegvideo_enc.c:2912
#6  MPV_encode_picture (avctx=<optimized out>, buf=<optimized out>, buf_size=<optimized out>, data=<optimized out>) at libavcodec/mpegvideo_enc.c:1214
#7  0x00000000004d66f6 in avcodec_encode_video (avctx=avctx@entry=0xe94f70, buf=<optimized out>, buf_size=<optimized out>, pict=pict@entry=0x7fffffffd340)
    at libavcodec/utils.c:515
#8  0x00000000004378c5 in do_video_out (ist=0x0, frame_size=<synthetischer Zeiger>, in_picture=0x7fffffffcfe0, ost=0xe9d510, s=<optimized out>) at ffmpeg.c:974
#9  output_packet (ist=ist@entry=0xe9d4a0, ist_index=ist_index@entry=0, ost_table=ost_table@entry=0xe9d4f0, nb_ostreams=nb_ostreams@entry=1, 
    pkt=pkt@entry=0x7fffffffd560) at ffmpeg.c:1358
#10 0x000000000043a5fc in av_encode (nb_output_files=nb_output_files@entry=1, nb_input_files=nb_input_files@entry=1, nb_stream_maps=<optimized out>, 
    stream_maps=0xb23420 <stream_maps>, input_files=0xb27420 <input_files>, output_files=0xb259c0 <output_files>) at ffmpeg.c:2153
#11 0x0000000000435018 in main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3928

[Carl Eugen Hoyos]

(gdb) r -i tests/lena.pnm -s 176x144 -trellis 2 out.h261
Starting program: ffmpeg_g -i tests/lena.pnm -s 176x144 -trellis 2 out.h261
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
ffmpeg version N-58200-g92cbd77 Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 17 2013 04:21:45 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      52. 53.100 / 52. 53.100
  libavcodec     55. 43.100 / 55. 43.100
  libavformat    55. 21.100 / 55. 21.100
  libavdevice    55.  5.100 / 55.  5.100
  libavfilter     3. 91.100 /  3. 91.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, image2, from 'tests/lena.pnm':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: ppm, rgb24, 256x256, 25 tbr, 25 tbn, 25 tbc
[New Thread 0x7ffff59eb700 (LWP 23636)]
[New Thread 0x7ffff51ea700 (LWP 23637)]
[New Thread 0x7ffff49e9700 (LWP 23638)]
[New Thread 0x7ffff41e8700 (LWP 23639)]
[New Thread 0x7ffff39e7700 (LWP 23640)]
[New Thread 0x7ffff31e6700 (LWP 23641)]
[New Thread 0x7ffff29e5700 (LWP 23642)]
[New Thread 0x7ffff21e4700 (LWP 23643)]
[New Thread 0x7ffff19e3700 (LWP 23644)]
Output #0, h261, to 'out.h261':
  Metadata:
    encoder         : Lavf55.21.100
    Stream #0:0: Video: h261, yuv420p, 176x144, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ppm -> h261)
Press [q] to stop, [?] for help

Program received signal SIGSEGV, Segmentation fault.
0x000000000093d2f9 in dct_quantize_trellis_c (s=0x17cb5e0, block=0x17ef2a0, n=0, qscale=3,
    overflow=0x7fff0000001a) at libavcodec/mpegvideo_enc.c:3619
3619                        int score= distortion + length[UNI_AC_ENC_INDEX(run, level)]*lambda;
(gdb) bt
#0  0x000000000093d2f9 in dct_quantize_trellis_c (s=0x17cb5e0, block=0x17ef2a0, n=0, qscale=3,
    overflow=0x7fff0000001a) at libavcodec/mpegvideo_enc.c:3619
#1  0x00000000009488c1 in encode_mb_internal (mb_block_count=6, mb_block_width=8,
    mb_block_height=8, motion_y=0, motion_x=0, s=0x17cb5e0) at libavcodec/mpegvideo_enc.c:2060
#2  encode_mb (motion_y=0, motion_x=0, s=0x17cb5e0) at libavcodec/mpegvideo_enc.c:2168
#3  encode_thread (c=<optimized out>, arg=<optimized out>) at libavcodec/mpegvideo_enc.c:3042
#4  0x0000000000a2fda7 in avcodec_default_execute (c=0x17cafa0, func=0x945310 <encode_thread>,
    arg=<optimized out>, ret=<optimized out>, count=1, size=8) at libavcodec/utils.c:1016
#5  0x000000000094fc21 in encode_picture (picture_number=0, s=0x17cb5e0)
    at libavcodec/mpegvideo_enc.c:3435
#6  ff_MPV_encode_picture (avctx=0x17cafa0, pkt=0x7fffffffda60, pic_arg=<optimized out>,
    got_packet=0x7fffffffd91c) at libavcodec/mpegvideo_enc.c:1494
#7  0x0000000000a30ff7 in avcodec_encode_video2 (avctx=avctx@entry=0x17cafa0,
    avpkt=avpkt@entry=0x7fffffffda60, frame=frame@entry=0x182d3a0,
    got_packet_ptr=got_packet_ptr@entry=0x7fffffffd91c) at libavcodec/utils.c:1863
#8  0x0000000000471552 in do_video_out (in_picture=0x182d3a0, ost=0x17cb400, s=0x17caa00)
    at ffmpeg.c:965
#9  reap_filters () at ffmpeg.c:1110
#10 0x0000000000461548 in transcode_step () at ffmpeg.c:3235
#11 transcode () at ffmpeg.c:3278
#12 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3456
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0x93d2d9 to 0x93d319:
   0x000000000093d2d9 <dct_quantize_trellis_c+1177>:    jmp    0x93d2eb <dct_quantize_trellis_c+1195>
   0x000000000093d2db <dct_quantize_trellis_c+1179>:    nopl   0x0(%rax,%rax,1)
   0x000000000093d2e0 <dct_quantize_trellis_c+1184>:    movslq %edx,%rax
   0x000000000093d2e3 <dct_quantize_trellis_c+1187>:    movslq 0x4e0(%rsp,%rax,4),%rsi
   0x000000000093d2eb <dct_quantize_trellis_c+1195>:    mov    %r9d,%edi
   0x000000000093d2ee <dct_quantize_trellis_c+1198>:    sub    %esi,%edi
   0x000000000093d2f0 <dct_quantize_trellis_c+1200>:    mov    %edi,%eax
   0x000000000093d2f2 <dct_quantize_trellis_c+1202>:    shl    $0x7,%eax
   0x000000000093d2f5 <dct_quantize_trellis_c+1205>:    add    %ebx,%eax
   0x000000000093d2f7 <dct_quantize_trellis_c+1207>:    cltq
=> 0x000000000093d2f9 <dct_quantize_trellis_c+1209>:    movzbl 0x0(%r13,%rax,1),%eax
   0x000000000093d2ff <dct_quantize_trellis_c+1215>:    imul   %r15d,%eax
   0x000000000093d303 <dct_quantize_trellis_c+1219>:    add    %r10d,%eax
   0x000000000093d306 <dct_quantize_trellis_c+1222>:    add    0x3d0(%rsp,%rsi,4),%eax
   0x000000000093d30d <dct_quantize_trellis_c+1229>:    cmp    %ecx,%eax
   0x000000000093d30f <dct_quantize_trellis_c+1231>:    jge    0x93d323 <dct_quantize_trellis_c+1251>
   0x000000000093d311 <dct_quantize_trellis_c+1233>:    mov    %edi,0x1b0(%rsp,%r8,4)
End of assembler dump.
(gdb) info register
rax            0x41     65
rbx            0x41     65
rcx            0x78000000       2013265920
rdx            0x0      0
rsi            0x1      1
rdi            0x0      0
rbp            0x0      0x0
rsp            0x7fffffff0bc0   0x7fffffff0bc0
r8             0x2      2
r9             0x1      1
r10            0xfffffe20       4294966816
r11            0x1      1
r12            0x0      0
r13            0x0      0
r14            0x0      0
r15            0x24f    591
rip            0x93d2f9 0x93d2f9 <dct_quantize_trellis_c+1209>
eflags         0x10206  [ PF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

[Michael Niedermayer]

Implemented in 331a90cec42f600c8b63231bb0d5be6df6bf1717