Encoding an image only movie fails when used with the movie overlay filter

[Jörgen Isaksson]

I'm trying to overlay a QuickTime? movie that consists of PNG images on top of another movie. I have noticed that FFMPEG doesn't seem to handle such an image-only movie correctly when used in combination with the movie overlay filter.

This is my command line:

ffmpeg -y -i /Users/jogga/Desktop/skiing.mov -acodec copy -s 640x360 -vcodec libx264 -vpre slow -vpre ipod640 -crf 22 -threads 0 -vf "movie=/Users/jogga/Desktop/skiing_subtitles.mov:f=mov, pad=640:360:0:286:0x00000000[subtitles];[in][subtitles] overlay=0:0:1[out]" /Users/jogga/Desktop/skiing_subtitled.m4v

And this is the uncut output of the operation:

ffmpeg version N-32576-g872655f, Copyright (c) 2000-2011 the FFmpeg developers

built on Sep 14 2011 10:39:13 with clang 2.1 (tags/Apple/clang-163.7.1)
configuration: --arch=x86_64 --cc=clang --disable-debug --disable-shared --enable-static --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-libvpx --enable-libtheora --enable-libvorbis --disable-ffplay --disable-ffserver --enable-gpl --enable-swscale --enable-avfilter --enable-filter=movie
libavutil 51. 16. 0 / 51. 16. 0
libavcodec 53. 14. 0 / 53. 14. 0
libavformat 53. 12. 0 / 53. 12. 0
libavdevice 53. 3. 0 / 53. 3. 0
libavfilter 2. 40. 0 / 2. 40. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0

Seems stream 0 codec frame rate differs from container frame rate: 5000.00 (5000/1) -> 25.00 (25/1)

Seems stream 2 codec frame rate differs from container frame rate: 5000.00 (5000/1) -> 25.00 (25/1)

Seems stream 4 codec frame rate differs from container frame rate: 5000.00 (5000/1) -> 25.00 (25/1)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/jogga/Desktop/skiing.mov':

Metadata:

major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2009-08-30 06:30:04
date : 2009
title : skiing
encoder : Subtitled using iSubtitle (​http://www.bitfield.se/isubtitle)

Duration: 00:00:10.98, start: 0.000000, bitrate: 2623 kb/s

Stream #0.0(): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 1493 kb/s, 25 fps, 25 tbr, 2500 tbn, 5k tbc
Metadata:

creation_time : 2009-08-30 06:30:04

Stream #0.1(sve): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, s16, 131 kb/s
Metadata:

creation_time : 2009-10-06 11:45:11

Stream #0.2(): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 1493 kb/s, 25 fps, 25 tbr, 2500 tbn, 5k tbc
Metadata:

creation_time : 2009-08-30 06:30:04

Stream #0.3(sve): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, s16, 131 kb/s
Metadata:

creation_time : 2009-10-06 11:45:11

Stream #0.4(): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 1493 kb/s, 25 fps, 25 tbr, 2500 tbn, 5k tbc
Metadata:

creation_time : 2009-08-30 06:30:04

Stream #0.5(sve): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, s16, 131 kb/s
Metadata:

creation_time : 2009-10-06 11:45:11

Stream #0.6(eng): Data: none (tmcd / 0x64636D74)
Metadata:

creation_time : 2010-10-18 06:35:59

[buffer @ 0x10b703a60] w:640 h:360 pixfmt:yuv420p tb:1/1000000 sar:1/1 sws_param:
[movie @ 0x10b7048a0] seek_point:0 format_name:mov file_name:/Users/jogga/Desktop/skiing_subtitles.mov stream_index:0
[overlay @ 0x10b704140] auto-inserting filter 'auto-inserted scaler 0' between the filter 'Parsed_pad_1' and the filter 'Parsed_overlay_2'
[pad @ 0x10b704800] w:640 h:74 -> w:640 h:360 x:0 y:286 color:0x00000000[rgba]
[scale @ 0x10b7192a0] w:640 h:360 fmt:bgra -> w:640 h:360 fmt:yuva420p flags:0x4
[overlay @ 0x10b704140] main w:640 h:360 fmt:yuv420p overlay x:0 y:0 w:640 h:360 fmt:yuva420p
[overlay @ 0x10b704140] main_tb:1/1000000 overlay_tb:1/600 -> tb:1/3000000 exact:1
[libx264 @ 0x7f7f58827e00] using SAR=1/1
[libx264 @ 0x7f7f58827e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
[libx264 @ 0x7f7f58827e00] profile Constrained Baseline, level 3.0
[libx264 @ 0x7f7f58827e00] 264 - core 115 r1937 aa21558 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - ​http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=1:0:0 analyse=0x1:0x111 me=umh subme=8 psy=0 mixed_ref=0 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=6 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=50 rc=crf mbtree=0 crf=22.0 qcomp=0.60 qpmin=10 qpmax=51 qpstep=4 vbv_maxrate=10000 vbv_bufsize=10000 crf_max=0.0 nal_hrd=none ip_ratio=1.40 aq=1:1.00
Output #0, ipod, to '/Users/jogga/Desktop/skiing_subtitled.m4v':

Metadata:

major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2009-08-30 06:30:04
date : 2009
title : skiing
encoder : Lavf53.12.0
Stream #0.0(): Video: h264 (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], q=10-51, 25 tbn, 25 tbc
Metadata:

creation_time : 2009-08-30 06:30:04

Stream #0.1(sve): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, 131 kb/s
Metadata:

creation_time : 2009-10-06 11:45:11

Stream mapping:

Stream #0.0 -> #0.0 (h264 -> libx264)
Stream #0.1 -> #0.1 (copy)

Press [q] to stop, ? for help
frame= 86 fps= 0 q=23.0 size= 178kB time=00:00:01.16 bitrate=1256.0kbits/frame= 137 fps=136 q=26.0 size= 509kB time=00:00:03.20 bitrate=1302.3kbits/frame= 177 fps=117 q=22.0 size= 756kB time=00:00:04.80 bitrate=1290.0kbits/frame= 222 fps=110 q=26.0 size= 1089kB time=00:00:06.60 bitrate=1351.8kbits/frame= 263 fps=104 q=26.0 size= 1396kB time=00:00:08.24 bitrate=1387.8kbits/frame= 274 fps= 84 q=-1.0 Lsize= 1905kB time=00:00:10.96 bitrate=1423.8kbits/s
video:1722kB audio:176kB global headers:0kB muxing overhead 0.373441%
frame I:11 Avg QP:15.74 size: 15903
[libx264 @ 0x7f7f58827e00] frame P:263 Avg QP:23.39 size: 6036
[libx264 @ 0x7f7f58827e00] mb I I16..4: 78.5% 0.0% 21.5%
[libx264 @ 0x7f7f58827e00] mb P I16..4: 13.4% 0.0% 6.5% P16..4: 52.5% 9.3% 1.2% 0.0% 0.0% skip:17.1%
[libx264 @ 0x7f7f58827e00] coded y,uvDC,uvAC intra: 43.9% 32.4% 1.0% inter: 34.0% 21.6% 0.1%
[libx264 @ 0x7f7f58827e00] i16 v,h,dc,p: 34% 26% 27% 14%
[libx264 @ 0x7f7f58827e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 29% 29% 3% 3% 2% 4% 2% 4%
[libx264 @ 0x7f7f58827e00] i8c dc,h,v,p: 74% 14% 11% 1%
[libx264 @ 0x7f7f58827e00] kb/s:1286.40

The first sample in the overlay movie "skiing_subtitles.mov" starts at exactly 4 seconds. When encoded using the command line above FFMPEG skips the first 4 seconds of empty samples and the first sample is shown on frame 1 instead of frame 100 in the resulting output movie.

I have attached all the sample files and a screenshot.

/jorgen

[Jörgen Isaksson]

The sample files are available in my dropbox at:

"skiing.mov" at ​http://dl.dropbox.com/u/659266/skiing.mov

"skiing_subtitles.mov" at ​http://dl.dropbox.com/u/659266/skiing_subtitles.mov

and the resulting file of the command line

"skiing_subtitled.m4v" at ​http://dl.dropbox.com/u/659266/skiing_subtitled.m4v

Let me know if you need anything more

[Carl Eugen Hoyos]

Do you believe this is a regression? (You set the priority to important, that usually indicates a regression.)
If yes, please provide the version introducing it.

Is the problem only reproducible with libx264?
Is scaling (-s) required to reproduce the problem?

Please consider using "Code block" to post ffmpeg output.

[Jörgen Isaksson]

Hi cehoyos,

Do you believe this is a regression? (You set the priority to important, that usually indicates a regression.) If yes, please provide the version introducing it.

Yes, I believe it's a regression. As I pointed out in ticket #137 it stopped working in FFmpeg version SVN-r25526, Copyright (c) 2000-2010 the FFmpeg developers

Is the problem only reproducible with libx264?

It's reproducable with any codec

Is scaling (-s) required to reproduce the problem?

No

Please consider using "Code block" to post ffmpeg output.

Will do.

Please check ticket #137. This is the same bug that was fixed in #137 but it still doesn't work with the movie overlay filter.

[Carl Eugen Hoyos]

Was there really a working filter system before r25526?

Please add a minimal command line (without external libraries) and the output.

[Jörgen Isaksson]

Was there really a working filter system before r25526?

Yes, but libavfilter was in it's own branch back then. I use this version that's still working. svn co ​svn://svn.ffmpeg.org/soc/libavfilter@r5940

Please add a minimal command line (without external libraries) and the output.

This as simple as I can make it

ffmpeg -y -i /Users/jogga/Desktop/skiing.m4v -vf "movie=/Users/jogga/Desktop/skiing_subtitles.mov:f=mov, pad=640:360:0:286:0x00000000[subtitles];[in][subtitles] overlay=0:0:1[out]" /Users/jogga/Desktop/skiing_subtitled.m4v

and this is the uncut output

ffmpeg version N-32576-g872655f, Copyright (c) 2000-2011 the FFmpeg developers
  built on Sep 14 2011 10:39:13 with clang 2.1 (tags/Apple/clang-163.7.1)
  configuration: --arch=x86_64 --cc=clang --disable-debug --disable-shared --enable-static --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-libvpx --enable-libtheora --enable-libvorbis --disable-ffplay --disable-ffserver --enable-gpl --enable-swscale --enable-avfilter --enable-filter=movie
  libavutil    51. 16. 0 / 51. 16. 0
  libavcodec   53. 14. 0 / 53. 14. 0
  libavformat  53. 12. 0 / 53. 12. 0
  libavdevice  53.  3. 0 / 53.  3. 0
  libavfilter   2. 40. 0 /  2. 40. 0
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0

Seems stream 0 codec frame rate differs from container frame rate: 5000.00 (5000/1) -> 25.00 (25/1)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/jogga/Desktop/skiing.m4v':
  Metadata:
    major_brand     : M4V 
    minor_version   : 1
    compatible_brands: M4V M4A mp42isom
    creation_time   : 2011-09-15 08:04:40
    encoder         : Mac OS X v10.7.1 (CMA 889, CM 705.35, i386)
  Duration: 00:00:09.96, start: 0.000000, bitrate: 1531 kb/s
    Stream #0.0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 1529 kb/s, 25.09 fps, 25 tbr, 2500 tbn, 5k tbc
    Metadata:
      creation_time   : 2011-09-15 08:04:40
[buffer @ 0x108e01420] w:640 h:360 pixfmt:yuv420p tb:1/1000000 sar:1/1 sws_param:
[movie @ 0x108e02460] seek_point:0 format_name:mov file_name:/Users/jogga/Desktop/skiing_subtitles.mov stream_index:0
[overlay @ 0x108e03760] auto-inserting filter 'auto-inserted scaler 0' between the filter 'Parsed_pad_1' and the filter 'Parsed_overlay_2'
[pad @ 0x108e03500] w:640 h:74 -> w:640 h:360 x:0 y:286 color:0x00000000[rgba]
[scale @ 0x108e01a40] w:640 h:360 fmt:bgra -> w:640 h:360 fmt:yuva420p flags:0x4
[overlay @ 0x108e03760] main w:640 h:360 fmt:yuv420p overlay x:0 y:0 w:640 h:360 fmt:yuva420p
[overlay @ 0x108e03760] main_tb:1/1000000 overlay_tb:1/600 -> tb:1/3000000 exact:1
[libx264 @ 0x7f9eab807600] using SAR=1/1
[libx264 @ 0x7f9eab807600] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
[libx264 @ 0x7f9eab807600] profile High, level 3.0
[libx264 @ 0x7f9eab807600] 264 - core 115 r1937 aa21558 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, ipod, to '/Users/jogga/Desktop/skiing_subtitled.m4v':
  Metadata:
    major_brand     : M4V 
    minor_version   : 1
    compatible_brands: M4V M4A mp42isom
    creation_time   : 2011-09-15 08:04:40
    encoder         : Lavf53.12.0
    Stream #0.0(und): Video: h264 (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], q=-1--1, 25 tbn, 25 tbc
    Metadata:
      creation_time   : 2011-09-15 08:04:40
Stream mapping:
  Stream #0.0 -> #0.0 (h264 -> libx264)
Press [q] to stop, [?] for help
frame=   73 fps=  0 q=28.0 size=      91kB time=00:00:00.84 bitrate= 886.2kbits/frame=  113 fps=108 q=28.0 size=     190kB time=00:00:02.44 bitrate= 636.6kbits/frame=  156 fps=101 q=28.0 size=     326kB time=00:00:04.16 bitrate= 642.0kbits/frame=  198 fps= 96 q=28.0 size=     447kB time=00:00:05.84 bitrate= 627.7kbits/frame=  231 fps= 90 q=28.0 size=     684kB time=00:00:07.16 bitrate= 782.3kbits/frame=  250 fps= 73 q=-1.0 Lsize=    1070kB time=00:00:09.92 bitrate= 883.4kbits/s    
video:1065kB audio:0kB global headers:0kB muxing overhead 0.412756%
frame I:7     Avg QP:20.72  size: 17157
[libx264 @ 0x7f9eab807600] frame P:132   Avg QP:23.97  size:  5745
[libx264 @ 0x7f9eab807600] frame B:111   Avg QP:27.70  size:  1908
[libx264 @ 0x7f9eab807600] consecutive B-frames: 23.2% 46.4% 19.2% 11.2%
[libx264 @ 0x7f9eab807600] mb I  I16..4: 65.8%  3.3% 30.9%
[libx264 @ 0x7f9eab807600] mb P  I16..4: 17.9%  7.7%  3.5%  P16..4: 48.2%  6.5%  3.4%  0.0%  0.0%    skip:12.8%
[libx264 @ 0x7f9eab807600] mb B  I16..4:  2.7%  1.8%  0.7%  B16..8: 42.4%  3.1%  0.6%  direct: 2.4%  skip:46.3%  L0:71.0% L1:20.5% BI: 8.4%
[libx264 @ 0x7f9eab807600] 8x8 transform intra:24.2% inter:46.0%
[libx264 @ 0x7f9eab807600] coded y,uvDC,uvAC intra: 42.7% 29.2% 1.0% inter: 24.5% 14.0% 0.0%
[libx264 @ 0x7f9eab807600] i16 v,h,dc,p: 35% 25% 26% 14%
[libx264 @ 0x7f9eab807600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 24% 44%  3%  2%  2%  2%  2%  5%
[libx264 @ 0x7f9eab807600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 29% 25%  3%  3%  2%  3%  2%  5%
[libx264 @ 0x7f9eab807600] i8c dc,h,v,p: 73% 15% 11%  1%
[libx264 @ 0x7f9eab807600] Weighted P-Frames: Y:25.0% UV:19.7%
[libx264 @ 0x7f9eab807600] ref P L0: 72.8% 15.0% 10.0%  2.2%  0.1%
[libx264 @ 0x7f9eab807600] ref B L0: 97.2%  2.4%  0.3%
[libx264 @ 0x7f9eab807600] ref B L1: 97.9%  2.1%
[libx264 @ 0x7f9eab807600] kb/s:872.20

The two input files:

​http://dl.dropbox.com/u/659266/skiing.m4v
​http://dl.dropbox.com/u/659266/skiing_subtitles.mov

And the resulting output file:

​http://dl.dropbox.com/u/659266/skiing_subtitled.m4v

[Stefano Sabatini]

It took more time than desired, but should be fixed in:

commit 7bdefc0f1281c2c18375197985fcd5109e711829
Author: Stefano Sabatini <stefasab@gmail.com>
Date:   Tue Jan 10 01:21:17 2012 +0100

    lavfi/overlay: add logic for avoiding overlaying frames with PTS > main frame PTS
    
    Also add debug logging messages for helping tracking down similar
    issues.
    
    Fix trac ticket #467.

Thanks for the accurate report.

[Jörgen Isaksson]

Hi Stefano,

Thx a lot for finally fixing this bug :-)

I will check out the fix ASAP.

/Jörgen