Dataloss because ffmpeg manpage about -passlogfile is wrong

[Jon Bendtsen]

ffmpeg manpage about -passlogfile is wrong, it says:

"-passlogfile prefix
Set two-pass log file name prefix to prefix, the default file name prefix is ffmpeg2pass. The complete file name will be PREFIX-N.log, where N is a number specific to the output stream."

So based on that I felt it was safe to use the input filename as -passlogfile, but nooo, it was not safe, and my input files was overwritten, causing a dataloss :-(

Command lines and ffmpeg output is attached as files.

System is debian stable.
ii ffmpeg 5:0.7.13-dmo2 audio/video encoder, streaming server & audio/video file conve

[Jon Bendtsen]

The command lines for ffmpeg I was running

[Jon Bendtsen]

The output from some of the ffmpeg commands before I interrupted it.

[Jon Bendtsen]

ls -la of the directory where the input and passlogfiles was stored.

[Carl Eugen Hoyos]

Could you confirm that the problem is not reproducible with -vcodec mpeg4 ?

[Jon Bendtsen]

Excuse me, but dataloss is NEVER EVER minor. Dataloss is a serious problem and the documentation and/or code should be changed quickly.

I can confirm that I did not overwrite the input source file with these commands:

ffmpeg -i 00004.MTS -vcodec mpeg4 -sameq -acodec copy -deinterlace -r 25 -s 1440x1080 -t 1537 -pass 1 -an -passlogfile 00004.MTS -f rawvideo -y /dev/null
ffmpeg -i 00004.MTS -vcodec mpeg4 -sameq -acodec copy -deinterlace -r 25 -s 1440x1080 -t 1537 -pass 2 -an -passlogfile 00004.MTS -f rawvideo -y /dev/null
ffmpeg -i 00004.MTS -vcodec mpeg4 -sameq -acodec copy -deinterlace -r 25 -s 1440x1080 -t 1537 -pass 3 -passlogfile 00004.MTS 00004.MTS.t1537.mp4

[Carl Eugen Hoyos]

This may have been fixed several months ago.
Could you test current git head?

[Jon Bendtsen]

If it has been fixed, and it is a dataloss bug, how come the online documentation was not updated?

Trying to follow https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide running ./configure --enable-static

Found yasm 0.8.0.2194
Minimum version is yasm-1.0.0
If you really want to compile without asm, configure with --disable-asm.

So no I can not just test against current git head.

[Carl Eugen Hoyos]

Replying to jonb:

If it has been fixed, and it is a dataloss bug, how come the online documentation was not updated?

The code was fixed to match the actual documentation.

Trying to follow https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide running ./configure --enable-static
Found yasm 0.8.0.2194
Minimum version is yasm-1.0.0
If you really want to compile without asm, configure with --disable-asm.

(It took me quite some time to find out you failed building x264 and not FFmpeg, did you test using the system-installed x264 instead? Version 118 from September 2011 is needed at least.)
yasm is a standalone binary that takes a few minutes to compile, you can either put it into /usr/local/bin or overwrite your current yasm executable.

[Jon Bendtsen]

What? No warning in the actual documentation that versions prior to $fixed_version has a dataloss bug.

I am sorry for not writting that I was trying to compile x264. Which I am trying to compile because this issue is only with -vcodec libx264 so naturally I need x264, right?

[Carl Eugen Hoyos]

Replying to jonb:

I am sorry for not writting that I was trying to compile x264. Which I am trying to compile because this issue is only with -vcodec libx264 so naturally I need x264, right?

Yes, but you have already a version of x264 on your system, and if it is >= 118 (FFmpeg configure will tell you), you do not have to reinstall.

[Jon Bendtsen]

I am sorry, but the compilation turns out to be complicated, I need lots of libs and utilities. How come you guys can not test with any video file and turn it into a libx264?

[Carl Eugen Hoyos]

Replying to jonb:

I am sorry, but the compilation turns out to be complicated, I need lots of libs

Could you elaborate? FFmpeg is known for being mostly self-contained, except for libx264 if you want to encode h264 no other external libraries should be needed.

[Jon Bendtsen]

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3
ERROR: libfdk_aac not found

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
​ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.
jonbendtsen@media:/jon_compiles_ffmpeg/fdk-aac$ autoreconf -fiv
-bash: autoreconf: command not found

okay, so I install the Debian package autoconf2.13

jonbendtsen@media:/jon_compiles_ffmpeg/fdk-aac$ autoreconf -fiv
autoreconf2.50: Entering directory `.'
autoreconf2.50: configure.ac: not using Gettext
autoreconf2.50: running: aclocal --force -I m4
autoreconf2.50: configure.ac: tracing
autoreconf2.50: configure.ac: not using Libtool
autoreconf2.50: running: /usr/bin/autoconf --force
autoreconf2.50: configure.ac: not using Autoheader
autoreconf2.50: running: automake --add-missing --copy --force-missing
configure.ac:7: installing `./install-sh'
configure.ac:7: installing `./missing'
Makefile.am:30: Libtool library used but `LIBTOOL' is undefined
Makefile.am:30: The usual way to define LIBTOOL' is to add AC_PROG_LIBTOOL'
Makefile.am:30: to configure.ac' and run aclocal' and `autoconf' again.
Makefile.am:30: If AC_PROG_LIBTOOL' is in configure.ac', make sure
Makefile.am:30: its definition is in aclocal's search path.
Makefile.am: installing `./depcomp'
autoreconf2.50: automake failed with exit status: 1

but it is still complaining about missing stuff.

[Jon Bendtsen]

oh and before ffmpeg configure complained about libfdk_aac it complained about

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-libfaac --enable-libfdk-aac --enable-libmp3lame \

--enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora \
--enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3

ERROR: libfaac not found

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
​ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.

which I also installed from debian package manager.

the machine is a server and naturally it does not have lots of stuff for compiling

[reimar]

You don't need those to test this issue.
./configure --enable-gpl --enable-x264
should be enough.
And even though we can/could test it, it is always most reliable for the person who reported an issue to verify it is fixed.

[Jon Bendtsen]

onbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-x264
Unknown option "--enable-x264".
See ./configure --help for available options.

[reimar]

I obviously meant the --enable-libx264 you had in there originally.

[Jon Bendtsen]

I just did cut'n'move

[Jon Bendtsen]

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-libx264
ERROR: libx264 not found

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
​ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.

installing the debian package now.

okay, so it compiled, but the same cmdlines doesnt work.

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg/overwritetest$ cat ffmpeg_cmdlines.sh
ls -la
../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 1 -an -passlogfile 00000.MTS -f rawvideo -y /dev/null
ls -la
../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 2 -an -passlogfile 00000.MTS -f rawvideo -y /dev/null
ls -la
../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 3 -passlogfile 00000.MTS 00000.MTS.ss438.h264

that does not overwrite any files, but now there is another problem: No sound?

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg/overwritetest$ ffmpeg -i *.h264
ffmpeg version 0.7.13, Copyright (c) 2000-2011 the FFmpeg developers

built on Jun 13 2012 14:14:09 with gcc 4.4.5
configuration: --enable-libdc1394 --prefix=/usr --extra-cflags='-Wall -g ' --cc='ccache cc' --enable-shared --enable-libmp3lame --enable-gpl --enable-libvorbis --enable-pthreads --enable-libfaac --enable-libxvid --enable-postproc --enable-x11grab --enable-libgsm --enable-libtheora --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libx264 --enable-libspeex --enable-nonfree --disable-stripping --enable-avfilter --enable-libdirac --disable-decoder=libdirac --enable-libfreetype --enable-libschroedinger --disable-encoder=libschroedinger --enable-version3 --enable-libopenjpeg --enable-libvpx --enable-librtmp --extra-libs=-lgcrypt --disable-altivec --disable-armv5te --disable-armv6 --disable-vis
libavutil 50. 43. 0 / 50. 43. 0
libavcodec 52.123. 0 / 52.123. 0
libavformat 52.111. 0 / 52.111. 0
libavdevice 52. 5. 0 / 52. 5. 0
libavfilter 1. 80. 0 / 1. 80. 0
libswscale 0. 14. 1 / 0. 14. 1
libpostproc 51. 2. 0 / 51. 2. 0

[h264 @ 0x24ed2e0] max_analyze_duration 5000000 reached at 5000000
[h264 @ 0x24ed2e0] Estimating duration from bitrate, this may be inaccurate

Seems stream 0 codec frame rate differs from container frame rate: 50.00 (50/1) -> 25.00 (50/2)
Input #0, h264, from '00000.MTS.ss438.h264':

Duration: N/A, bitrate: N/A

Stream #0.0: Video: h264 (High), yuv420p, 1440x1080 [PAR 4:3 DAR 16:9], 25 fps, 25 tbr, 1200k tbn, 50 tbc

At least one output file must be specified

[Carl Eugen Hoyos]

Replying to jonb:

../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 3 -passlogfile 00000.MTS 00000.MTS.ss438.h264

that does not overwrite any files, but now there is another problem: No sound?

(Complete, uncut console output missing.)

A h264 file (that is what your last line produces) cannot contain sound, the format only contains raw H264 video.

Carl Eugen

[Jon Bendtsen]

Replying to cehoyos:

A h264 file (that is what your last line produces) cannot contain sound, the format only contains raw H264 video.

okay, then I want to complain that FFmpeg continues regardless of conflicting options.

-acodec copy tells FFmpeg I want it to copy the sound. That means I want sound output. And if the output choice 00000.MTS.ss438.h264 indicates that I dont want sound, then there is a conflict which means FFmpeg should not continue. There is no reason to waste CPU cycles (and time) encoding something which is useless.

[Carl Eugen Hoyos]

Replying to jonb:

There is no reason to waste CPU cycles (and time) encoding something which is useless.

No CPU cycles are wasted in the command line you use.

[Carl Eugen Hoyos]

Replying to cehoyos:

Replying to jonb:

There is no reason to waste CPU cycles (and time) encoding something which is useless.

No CPU cycles are wasted in the command line you use.

(Maybe except for the fact that some sources claim 3-pass encoding makes no sense.)

[Jon Bendtsen]

Yeah I noticed that when I reconnected my screen as well. The 3 pass must be a left over from when I experimented with mencoder. And the no sound must be an output from one of those previous runs when I did specify no sound.

[Jon Bendtsen]

Anyway, back to the topic. Why havent ​http://ffmpeg.org/ffmpeg.html been updated to warn people that prior to version $insert_correct_version_here$ the passlogfile when choosing libx264 has a different name. Surely avoiding a dataloss should be high priority.

[Carl Eugen Hoyos]

Do you know the value for $insert_correct_version_here$ ?

[Jon Bendtsen]

Replying to cehoyos:

Do you know the value for $insert_correct_version_here$ ?

18 hours ago you did and wrote:

Version changed from unspecified to 0.7.13

This may have been fixed several months ago.
Could you test current git head?

And we found out it looks like it has been fixed. So did you guys publish any new versions between then and now ? If so that should be the value. Else you might pick the next version number.

[Carl Eugen Hoyos]

Thanks to your work, we know now that the original bug was fixed between the time 0.7 was branched (June 2011 iirc) and yesterday's git head.
Unfortunately, several thousand changes were in-between, so we don't know (yet) if this is reproducible with 0.8, 0.9, 0.10 or 0.11...
If you want to help, please consider using git bisect to find the change fixing the original problem.

[Jon Bendtsen]

Replying to cehoyos:

Thanks to your work, we know now that the original bug was fixed between the time 0.7 was branched (June 2011 iirc) and yesterday's git head.
Unfortunately, several thousand changes were in-between, so we don't know (yet) if this is reproducible with 0.8, 0.9, 0.10 or 0.11...
If you want to help, please consider using git bisect to find the change fixing the original problem.

Does that mean I have to read code. I'm not so good at that. Couldnt we just start to write in the documentation that it has been discovered that there in some versions is a possibility for a dataloss when using libx264 and passlogfile?

Then we can look for the versions that overwrite or not.

I might script something that checks out every version from git between 0.7.13 and head and finds out when it was fixed.

[Carl Eugen Hoyos]

Replying to jonb:

I might script something that checks out every version from git between 0.7.13 and head and finds out when it was fixed.

It will probably be enough to test 6968a7d and c2e3b56 (git checkout 6968a7d)

(Even if that surprisingly turns out to be not enough, git bisect would only require you to test ~12 revisions.)

[Jon Bendtsen]

Replying to cehoyos:

Replying to jonb:

I might script something that checks out every version from git between 0.7.13 and head and finds out when it was fixed.

It will probably be enough to test 6968a7d and c2e3b56 (git checkout 6968a7d)

trying

[Jon Bendtsen]

jonbendtsen@media:/jon_compiles_ffmpeg$ mkdir c2e3b56
jonbendtsen@media:/jon_compiles_ffmpeg$ cd c2e3b56/
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ git clone --depth 1 ​git://source.ffmpeg.org/ffmpeg
Cloning into ffmpeg...
remote: Counting objects: 13044, done.
remote: Compressing objects: 100% (8272/8272), done.
remote: Total 13044 (delta 9107), reused 7128 (delta 4547)
Receiving objects: 100% (13044/13044), 13.49 MiB | 2.71 MiB/s, done.
Resolving deltas: 100% (9107/9107), done.
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ git checkout c2e3b56
fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ ls
ffmpeg
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ cd ffmpeg/
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56/ffmpeg$ git checkout c2e3b56
error: pathspec 'c2e3b56' did not match any file(s) known to git.
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56/ffmpeg$ git-checkout c2e3b56
-bash: git-checkout: command not found
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56/ffmpeg$ git git-checkout c2e3b56
git: 'git-checkout' is not a git command. See 'git --help'.

[Carl Eugen Hoyos]

Replying to jonb:

jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ git clone --depth 1 ​git://source.ffmpeg.org/ffmpeg

Remove --depth 1

[Jon Bendtsen]

The results are in: c2e3b56 is bad and it overwrites files. 6968a7d is good, it does not overwrite files.

[Jon Bendtsen]

Replying to jonb:

Yeah I noticed that when I reconnected my screen as well. The 3 pass must be a left over from when I experimented with mencoder. And the no sound must be an output from one of those previous runs when I did specify no sound.

No, I found the source for my experiment with 3 pass encoding and ffmpeg, ​http://sonnati.wordpress.com/2011/08/19/ffmpeg-%E2%80%93-the-swiss-army-knife-of-internet-streaming-%E2%80%93-part-iii/

[Carl Eugen Hoyos]

I committed a documentation update for the next 0.7, 0.8, 0.9, 0.10 and 0.11 releases to clarify that there are two -passlogfile options with different syntax (depending on the used encoder).

I don't think we normally mention old behaviour in the current documentation, if you think a change is needed, change your local doc/ffmpeg.texi file and send the changes as unified diff to ffmpeg-devel.
See ​http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/150405 for the discussion so far.