x264 encoding with overlay fails ( moov atom not found)

[houbahop69]

Summary of the bug:I am using ffmpeg to re-encode old realmedia videos (rv10). I have writtent a script that produces in the same run mp4 (x264 aac) ogg and webm files. I have taken the opportunity to add a logo on the files.

Everything works perfectly for ogv and webm but the mp4 file cannot be played with this error:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0xdb61a0] moov atom not found
TagLib: MP4: Invalid atom size
[0xdd1c88] mp4 demux error: MP4 plugin discarded (no moov,foov,moof box)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xdd2280] moov atom not found
[0xdd1c88] avformat demux error: Could not open : Unknown error 1094995529

I have found a trick : if I add at least 1 ms to the "duration parameter" value the encoding works perfectly. There is also no problem if I remove the overlay. The following copy paste may have errors...

How to reproduce:

ffmpeg -y -ss 00:00:00.00 -v warning -threads 0 -i test.rm -i logoWhiteSmall.png -t 00:00:30.01 -s 768x576 -aspect 4:3 -r 25.0 -metadata title=Pathogenicity and genomic variability of HHV-6 -metadata author=Henri Agut -metadata copyright=ENS Lyon -metadata:s:a language=fra -vcodec libx264 -vpre medium -async 1 -pass 1 -b:v 700k -bt:v 50k -maxrate 750k -bufsize 1500k -map 0:0 -acodec libfaac -ac 2 -ar 48000 -b:a 64k -map 0:1 test.mp4 -t 00:00:30.01 -s 768x576 -aspect 4:3 -r 25.0 -metadata
 title=Pathogenicity and genomic variability of HHV-6 -metadata author=Henri Agut -metadata copyright=Lyon -metadata:s:a language=fra -vcodec libvpx -vpre ipod -async 1 -pass 1 -b:v 700k -bt:v 50k -maxrate 750k -bufsize 1500k -map 0:0 -acodec libvorbis -ac 2 -ar 48000 -b:a 64k -map 0:1 test.webm -t 00:00:30.01 -s 768x576 -aspect 4:3 -r 25.0 -metadata title=Pathogenicity and genomic variability of HHV-6 -metadata author=Henri Agut -metadata copyright=Lyon -metadata:s:a language=fra -vcodec libtheora -vpre ipod -async 1 -pass 1 -b:v 700k -bt:v 50k -maxrate 750k -bufsize 1500k -map 0:0 -acodec libvorbis -ac 2 -ar 48000 -b:a 64k -map 0:1 test.ogv
ffmpeg -y -ss 00:00:00.00 -v warning -threads 0 -i test.rm -i logoWhiteSmall.png -t 00:00:30.01 -filter_complex [0:v[1:v]overlay=0:main_h-overlay_h -s 768x576 -aspect 4:3 -r 25.0 -metadata title=Pathogenicity and genomic variability of HHV-6 -metadata author=Henri Agut -metadata copyright=ENS Lyon -metadata:s:a language=fra -vcodec libx264 -vpre medium -async 1 -pass 2 -b:v 700k -bt:v 50k -maxrate 750k -bufsize 1500k -map 0:0 -acodec libfaac -ac 2 -ar 48000 -b:a 64k -map 0:1 -map_channel 0.1.0 -map_channel 0.1.0 test.mp4 -t 00:00:30.01 -filter_complex [0:v][1:v]overlay=0:main_h-overlay_h -s 768x576 -aspect 4:3 -r 25.0 -metadata title=Pathogenicity and genomic variability of HHV-6 metadata author=Henri Agut -metadata copyright=ENS Lyon metadata:s:a language=fra -vcodec libvpx -vpre ipod -async 1 -pass 2 -b:v 700k -bt:v 50k -maxrate 750k -bufsize 1500k -map 0:0 -acodec libvorbis -ac 2 -ar 48000 -b:a 64k -map 0:1 -map_channel 0.1.0 -map_channel 0.1.0 test.webm -t 00:00:30.01 -filter_complex [0:v][1:v]overlay=0:main_h-overlay_h -s 768x576 -aspect 4:3 -r 25.0 -metadata title=Pathogenicity and genomic variability of HHV-6 -metadata author=Henri Agut -metadata copyright=Lyon -metadata:s:a language=fra -vcodec libtheora -vpre ipod -async 1 -pass 2 -b:v 700k -bt:v 50k -maxrate 750k -bufsize 1500k -map 0:0 -acodec libvorbis -ac 2 -ar 48000 -b:a 64k -map 0:1 -map_channel 0.1.0 -map_channel 0.1.0 test.ogv

[Carl Eugen Hoyos]

To make this a valid ticket, please add complete, uncut console output together with the minimal command line of the failing encode.

Please test current git head.

Is the problem only reproducible if you encode to webm, ogv and mp4 at the same time or also if you only encode to mp4?

[houbahop69]

logfiles of encodings as described in the comment

[houbahop69]

Thank you for considering my remarks.

To answer you quetion you will find attached a bzip2 tar file containing 5 logs :

1. *./logFail3coding.txt* obtained by executing 3 encoding (mp4 webm and ogv). Encoding succeeds but playing fails (see file logFfplayFail.txt)
2. *./logSuccess1coding.txt* obtained by executing mp4 encoding only. Encoding and playing succeed (see file logFfplaySuccess.txt)
3. *./logSuccess3coding+1ms.txt* obtained by executing 3 encoding (mp4 webm and ogv) with 1ms cheat on. Encoding and playing succeed (see file logFfplaySuccess.txt)
4. *./logFfplayFail.txt* log of unsuccessfull playing
5. *./logFfplaySuccess.txt* log of successfull playing

The encoding is not failing as the file produced has the same size (human readable) in all cases, the problem seems to be linked with writing the atom at the end of the file when multiple encoding. I have tried options to write the atom at the beginning with no success but I am not familiar with these questions and I may have misused the parameter.

I am using scripts to produce ffmpeg command and it is rather difficult to remove parameters without getting into trouble while constructing the command. It is really time-consuming... another solution might be to remove by hand the parameters but I don't know what parameters can be removed without breaking the command.

I am using ffmpeg from debian ​http://www.debian-multimedia.org/, I do not compile any more from git after some trouble with ekiga dependencies, if possible I would try to avoid compiling git source.

To answer you last question YES the problem occurs only when encoding the three formats at the same time.

As the problem can be easily bypassed either by adding 1ms in duration or separating the Three encodings I understand that it is not major, nevertheless it is very annoying when you try to encode a small part of a long file.

Thanks again for the great job.

[Carl Eugen Hoyos]

Is the problem also reproducible if you use -vcodec mpeg4 for the mp4 output file?

[houbahop69]

Replying to cehoyos:

Is the problem also reproducible if you use -vcodec mpeg4 for the mp4 output file?

The behavior is exactly the same with -vcodec mpeg4. I had to remove the presets but after that with the correct duration I get the atom problem if I add extra time in duration parameter it works just like libx264

[Carl Eugen Hoyos]

It is very difficult to read the log you provided, but am I correct that the problem disappears if you remove the ogv output stream but it still fails if you only remove the webm stream?
Is the problem still reproducible if you completely remove audio (that might need three times -an)?

Is the original problem (that encoding fails and aborts) also reproducible if you only encode ogv? That would make the analysis significantly easier afaict.

[houbahop69]

Replying to cehoyos:

It is very difficult to read the log you provided, but am I correct that the problem disappears if you remove the ogv output stream but it still fails if you only remove the webm stream?

If I remove only webm it fails
If I remove only ogv it works

Is the problem still reproducible if you completely remove audio (that might need three times -an)?

If I remove audio (3 times in both pass1 and 2) it fails

Is the original problem (that encoding fails and aborts)

Encoding fails (the resulting file connot be played) but I am not sure it aborts there is no message stating that at the end of mp4 encoding

also reproducible if you only encode ogv? That would make the analysis significantly easier afaict.

If I encode only one format it works perfectly with any of the formats including libx264/mp4

To push up your idea I have tried to change the sequence of encoding from

mp4 -> webm -> ogv to mp4 -> ogv -> webm

It does not work either

[Carl Eugen Hoyos]

Is the original problem (that encoding fails and aborts) also reproducible if you only encode ogv? That would make the analysis significantly easier afaict.

You misunderstood this paragraph.
In the output you attached, it can be seen that theora encoding fails with an error message, ffmpeg exits and no moov atom is written to the mp4 file making the mp4 output file invalid.

[libtheora @ 0x13cb6a0] theora_encode_YUVin failed (encoder is not ready or is finished) [-10]

Do you see the same error message if you only encode to ogv?

[Carl Eugen Hoyos]

I was finally able to reproduce this problem, fixed in 2fc354f

$ ffmpeg -i tests/lena.pnm -s 128x128 logo.png

$ ffmpeg -y -i fate-suite/svq3/Vertical400kbit.sorenson3.mov -t 5 -pass 1 -an -vcodec mpeg4 -f mov /dev/null -t 5 -pass 1 -an -f ogg /dev/null
ffmpeg version N-42203-g7beeea8 Copyright (c) 2000-2012 the FFmpeg developers
  built on Feb 22 2013 01:22:51 with gcc 4.7.1 20120723 [gcc-4_7-branch revision 189773]
  configuration: --enable-libtheora
  libavutil      51. 64.100 / 51. 64.100
  libavcodec     54. 32.100 / 54. 32.100
  libavformat    54. 14.100 / 54. 14.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     3.  0.101 /  3.  0.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33b8160] max_analyze_duration 5000000 reached at 5000998
Guessed Channel Layout for  Input Stream #0.1 : mono
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'fate-suite/svq3/Vertical400kbit.sorenson3.mov':
  Metadata:
    creation_time   : 2001-03-20 16:17:18
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Sorenson Video 3
    encoder-eng     : Sorenson Video 3
  Duration: 00:00:43.57, start: 0.000000, bitrate: 580 kb/s
    Stream #0:0(eng): Video: svq3 (SVQ3 / 0x33515653), yuvj420p, 320x240, 391 kb/s, 30.02 fps, 30 tbr, 600 tbn, 600 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
    Stream #0:1(eng): Audio: adpcm_ima_qt (ima4 / 0x34616D69), 44100 Hz, mono, s16, 176 kb/s
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
[graph 0 input from stream 0:0 @ 0x33daca0] w:320 h:240 pixfmt:yuvj420p tb:1/600 fr:30/1 sar:0/1 sws_param:flags=2
[output stream 0:0 @ 0x33b4020] No opaque field provided
[auto-inserted scaler 0 @ 0x33b42e0] w:320 h:240 fmt:yuvj420p sar:0/1 -> w:320 h:240 fmt:yuv420p sar:0/1 flags:0x4
[graph 1 input from stream 0:0 @ 0x34111c0] w:320 h:240 pixfmt:yuvj420p tb:1/600 fr:30/1 sar:0/1 sws_param:flags=2
[output stream 1:0 @ 0x349d100] No opaque field provided
[auto-inserted scaler 0 @ 0x349d060] w:320 h:240 fmt:yuvj420p sar:0/1 -> w:320 h:240 fmt:yuv420p sar:0/1 flags:0x4
Output #0, mov, to '/dev/null':
  Metadata:
    encoder-eng     : Sorenson Video 3
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Lavf54.14.100
    Stream #0:0(eng): Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 320x240, q=2-31, pass 1, 200 kb/s, 30 tbn, 30 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
Output #1, ogg, to '/dev/null':
  Metadata:
    encoder-eng     : Sorenson Video 3
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Lavf54.14.100
    Stream #1:0(eng): Video: theora, yuv420p, 320x240, q=2-31, pass 1, 200 kb/s, 30 tbn, 30 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
Stream mapping:
  Stream #0:0 -> #0:0 (svq3 -> mpeg4)
  Stream #0:0 -> #1:0 (svq3 -> libtheora)
Press [q] to stop, [?] for help
frame=  150 fps=0.0 q=13.3 Lq=0.0 size=       0kB time=00:00:05.00 bitrate=   0.0kbits/s
video:214kB audio:0kB subtitle:0 global headers:3kB muxing overhead -100.000000%

$ ffmpeg -y -i fate-suite/svq3/Vertical400kbit.sorenson3.mov -i logo.png -t 5 -pass 2 -an -vcodec mpeg4 test.mov -t 5 -pass 2 -an -filter_complex [0:0][1:0]overlay test.ogv
ffmpeg version N-42203-g7beeea8 Copyright (c) 2000-2012 the FFmpeg developers
  built on Feb 22 2013 01:22:51 with gcc 4.7.1 20120723 [gcc-4_7-branch revision 189773]
  configuration: --enable-libtheora
  libavutil      51. 64.100 / 51. 64.100
  libavcodec     54. 32.100 / 54. 32.100
  libavformat    54. 14.100 / 54. 14.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     3.  0.101 /  3.  0.101
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2b49160] max_analyze_duration 5000000 reached at 5000998
Guessed Channel Layout for  Input Stream #0.1 : mono
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'fate-suite/svq3/Vertical400kbit.sorenson3.mov':
  Metadata:
    creation_time   : 2001-03-20 16:17:18
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Sorenson Video 3
    encoder-eng     : Sorenson Video 3
  Duration: 00:00:43.57, start: 0.000000, bitrate: 580 kb/s
    Stream #0:0(eng): Video: svq3 (SVQ3 / 0x33515653), yuvj420p, 320x240, 391 kb/s, 30.02 fps, 30 tbr, 600 tbn, 600 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
    Stream #0:1(eng): Audio: adpcm_ima_qt (ima4 / 0x34616D69), 44100 Hz, mono, s16, 176 kb/s
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
Input #1, image2, from 'logo.png':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: png, rgb24, 128x128, 25 tbr, 25 tbn, 25 tbc
[graph 0 input from stream 0:0 @ 0x2b6bf60] w:320 h:240 pixfmt:yuvj420p tb:1/600 fr:30/1 sar:0/1 sws_param:flags=2
[graph 0 input from stream 1:0 @ 0x2b46fa0] w:128 h:128 pixfmt:rgb24 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
-t does not work with -filter_complex (yet).
[output stream 1:0 @ 0x2b6c300] No opaque field provided
[auto-inserted scaler 0 @ 0x2b6be20] w:320 h:240 fmt:yuvj420p sar:0/1 -> w:320 h:240 fmt:yuv420p sar:0/1 flags:0x2
[auto-inserted scaler 1 @ 0x2b451e0] w:128 h:128 fmt:rgb24 sar:0/1 -> w:128 h:128 fmt:yuva420p sar:0/1 flags:0x2
[graph 1 input from stream 0:0 @ 0x2c55440] w:320 h:240 pixfmt:yuvj420p tb:1/600 fr:30/1 sar:0/1 sws_param:flags=2
[output stream 0:0 @ 0x2c56460] No opaque field provided
[auto-inserted scaler 0 @ 0x2c56b20] w:320 h:240 fmt:yuvj420p sar:0/1 -> w:320 h:240 fmt:yuv420p sar:0/1 flags:0x4
Output #0, mov, to 'test.mov':
  Metadata:
    encoder-eng     : Sorenson Video 3
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Lavf54.14.100
    Stream #0:0(eng): Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 320x240, q=2-31, pass 2, 200 kb/s, 30 tbn, 30 tbc
    Metadata:
      creation_time   : 2001-03-20 16:17:18
      handler_name    : Apple Alias Data Handler
Output #1, ogg, to 'test.ogv':
  Metadata:
    encoder-eng     : Sorenson Video 3
    title           : Vertical Online SV3 Demo
    title-eng       : Vertical Online SV3 Demo
    artist          : Logan Kelsey
    artist-eng      : Logan Kelsey
    copyright       : © Vertical Online 2001
    copyright-eng   : © Vertical Online 2001
    encoder         : Lavf54.14.100
    Stream #1:0: Video: theora, yuv420p, 320x240, q=2-31, pass 2, 200 kb/s, 30 tbn, 30 tbc
Stream mapping:
  Stream #0:0 (svq3) -> overlay:main (graph 0)
  Stream #1:0 (png) -> overlay:overlay (graph 0)
  Stream #0:0 -> #0:0 (svq3 -> mpeg4)
  overlay (graph 0) -> Stream #1:0 (libtheora)
Press [q] to stop, [?] for help
theora_encode_YUVin failed (encoder is not ready or is finished) [-10]ate= 166.2kbits/s
Video encoding failed

When an encoding (or decoding) error occurs while muxing mov, current FFmpeg still fails to write the moov atom (see for example ticket #1918), I don't know if this is a bug / can be changed.