Opened 6 years ago

Closed 6 years ago

#1686 closed defect (fixed)

Dataloss because ffmpeg manpage about -passlogfile is wrong

Reported by: jonb Owned by:
Priority: minor Component: documentation
Version: 0.7.13 Keywords: passlogfile
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

ffmpeg manpage about -passlogfile is wrong, it says:

"-passlogfile prefix
Set two-pass log file name prefix to prefix, the default file name prefix is ffmpeg2pass. The complete file name will be PREFIX-N.log, where N is a number specific to the output stream."

So based on that I felt it was safe to use the input filename as -passlogfile, but nooo, it was not safe, and my input files was overwritten, causing a dataloss :-(

Command lines and ffmpeg output is attached as files.

System is debian stable.
ii ffmpeg 5:0.7.13-dmo2 audio/video encoder, streaming server & audio/video file conve

Attachments (5)

ffmpeg.cmdlines.txt (2.7 KB) - added by jonb 6 years ago.
The command lines for ffmpeg I was running
ffmpeg.output.txt (23.5 KB) - added by jonb 6 years ago.
The output from some of the ffmpeg commands before I interrupted it.
ls-la.txt (1.5 KB) - added by jonb 6 years ago.
ls -la of the directory where the input and passlogfiles was stored.
c2e3b56-bad_result.txt (7.6 KB) - added by jonb 6 years ago.
6968a7d-good_results.txt (10.7 KB) - added by jonb 6 years ago.

Download all attachments as: .zip

Change History (38)

Changed 6 years ago by jonb

The command lines for ffmpeg I was running

Changed 6 years ago by jonb

The output from some of the ffmpeg commands before I interrupted it.

Changed 6 years ago by jonb

ls -la of the directory where the input and passlogfiles was stored.

comment:1 Changed 6 years ago by cehoyos

  • Component changed from undetermined to documentation
  • Keywords dataloss ffmpeg manpage man removed
  • Priority changed from critical to minor

Could you confirm that the problem is not reproducible with -vcodec mpeg4 ?

comment:2 Changed 6 years ago by jonb

Excuse me, but dataloss is NEVER EVER minor. Dataloss is a serious problem and the documentation and/or code should be changed quickly.

I can confirm that I did not overwrite the input source file with these commands:

ffmpeg -i 00004.MTS -vcodec mpeg4 -sameq -acodec copy -deinterlace -r 25 -s 1440x1080 -t 1537 -pass 1 -an -passlogfile 00004.MTS -f rawvideo -y /dev/null
ffmpeg -i 00004.MTS -vcodec mpeg4 -sameq -acodec copy -deinterlace -r 25 -s 1440x1080 -t 1537 -pass 2 -an -passlogfile 00004.MTS -f rawvideo -y /dev/null
ffmpeg -i 00004.MTS -vcodec mpeg4 -sameq -acodec copy -deinterlace -r 25 -s 1440x1080 -t 1537 -pass 3 -passlogfile 00004.MTS 00004.MTS.t1537.mp4

comment:3 Changed 6 years ago by cehoyos

  • Version changed from unspecified to 0.7.13

This may have been fixed several months ago.
Could you test current git head?

comment:4 follow-up: Changed 6 years ago by jonb

If it has been fixed, and it is a dataloss bug, how come the online documentation was not updated?

Trying to follow https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide running ./configure --enable-static

Found yasm 0.8.0.2194
Minimum version is yasm-1.0.0
If you really want to compile without asm, configure with --disable-asm.

So no I can not just test against current git head.

comment:5 in reply to: ↑ 4 Changed 6 years ago by cehoyos

Replying to jonb:

If it has been fixed, and it is a dataloss bug, how come the online documentation was not updated?

The code was fixed to match the actual documentation.

Trying to follow https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide running ./configure --enable-static
Found yasm 0.8.0.2194
Minimum version is yasm-1.0.0
If you really want to compile without asm, configure with --disable-asm.

(It took me quite some time to find out you failed building x264 and not FFmpeg, did you test using the system-installed x264 instead? Version 118 from September 2011 is needed at least.)
yasm is a standalone binary that takes a few minutes to compile, you can either put it into /usr/local/bin or overwrite your current yasm executable.

comment:6 follow-up: Changed 6 years ago by jonb

What? No warning in the actual documentation that versions prior to $fixed_version has a dataloss bug.

I am sorry for not writting that I was trying to compile x264. Which I am trying to compile because this issue is only with -vcodec libx264 so naturally I need x264, right?

comment:7 in reply to: ↑ 6 Changed 6 years ago by cehoyos

Replying to jonb:

I am sorry for not writting that I was trying to compile x264. Which I am trying to compile because this issue is only with -vcodec libx264 so naturally I need x264, right?

Yes, but you have already a version of x264 on your system, and if it is >= 118 (FFmpeg configure will tell you), you do not have to reinstall.

comment:8 follow-up: Changed 6 years ago by jonb

I am sorry, but the compilation turns out to be complicated, I need lots of libs and utilities. How come you guys can not test with any video file and turn it into a libx264?

comment:9 in reply to: ↑ 8 Changed 6 years ago by cehoyos

Replying to jonb:

I am sorry, but the compilation turns out to be complicated, I need lots of libs

Could you elaborate? FFmpeg is known for being mostly self-contained, except for libx264 if you want to encode h264 no other external libraries should be needed.

comment:10 Changed 6 years ago by jonb

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3
ERROR: libfdk_aac not found

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.
jonbendtsen@media:/jon_compiles_ffmpeg/fdk-aac$ autoreconf -fiv
-bash: autoreconf: command not found

okay, so I install the Debian package autoconf2.13

jonbendtsen@media:/jon_compiles_ffmpeg/fdk-aac$ autoreconf -fiv
autoreconf2.50: Entering directory `.'
autoreconf2.50: configure.ac: not using Gettext
autoreconf2.50: running: aclocal --force -I m4
autoreconf2.50: configure.ac: tracing
autoreconf2.50: configure.ac: not using Libtool
autoreconf2.50: running: /usr/bin/autoconf --force
autoreconf2.50: configure.ac: not using Autoheader
autoreconf2.50: running: automake --add-missing --copy --force-missing
configure.ac:7: installing `./install-sh'
configure.ac:7: installing `./missing'
Makefile.am:30: Libtool library used but `LIBTOOL' is undefined
Makefile.am:30: The usual way to define LIBTOOL' is to add AC_PROG_LIBTOOL'
Makefile.am:30: to configure.ac' and run aclocal' and `autoconf' again.
Makefile.am:30: If AC_PROG_LIBTOOL' is in configure.ac', make sure
Makefile.am:30: its definition is in aclocal's search path.
Makefile.am: installing `./depcomp'
autoreconf2.50: automake failed with exit status: 1

but it is still complaining about missing stuff.

comment:11 Changed 6 years ago by jonb

oh and before ffmpeg configure complained about libfdk_aac it complained about

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-libfaac --enable-libfdk-aac --enable-libmp3lame \

--enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora \
--enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3

ERROR: libfaac not found

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.

which I also installed from debian package manager.

the machine is a server and naturally it does not have lots of stuff for compiling

comment:12 Changed 6 years ago by reimar

You don't need those to test this issue.
./configure --enable-gpl --enable-x264
should be enough.
And even though we can/could test it, it is always most reliable for the person who reported an issue to verify it is fixed.

comment:13 Changed 6 years ago by jonb

onbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-x264
Unknown option "--enable-x264".
See ./configure --help for available options.

comment:14 Changed 6 years ago by reimar

I obviously meant the --enable-libx264 you had in there originally.

comment:15 Changed 6 years ago by jonb

I just did cut'n'move

comment:16 follow-up: Changed 6 years ago by jonb

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg$ ./configure --enable-gpl --enable-libx264
ERROR: libx264 not found

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.

installing the debian package now.

okay, so it compiled, but the same cmdlines doesnt work.

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg/overwritetest$ cat ffmpeg_cmdlines.sh
ls -la
../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 1 -an -passlogfile 00000.MTS -f rawvideo -y /dev/null
ls -la
../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 2 -an -passlogfile 00000.MTS -f rawvideo -y /dev/null
ls -la
../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 3 -passlogfile 00000.MTS 00000.MTS.ss438.h264

that does not overwrite any files, but now there is another problem: No sound?

jonbendtsen@media:/jon_compiles_ffmpeg/ffmpeg/overwritetest$ ffmpeg -i *.h264
ffmpeg version 0.7.13, Copyright (c) 2000-2011 the FFmpeg developers

built on Jun 13 2012 14:14:09 with gcc 4.4.5
configuration: --enable-libdc1394 --prefix=/usr --extra-cflags='-Wall -g ' --cc='ccache cc' --enable-shared --enable-libmp3lame --enable-gpl --enable-libvorbis --enable-pthreads --enable-libfaac --enable-libxvid --enable-postproc --enable-x11grab --enable-libgsm --enable-libtheora --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libx264 --enable-libspeex --enable-nonfree --disable-stripping --enable-avfilter --enable-libdirac --disable-decoder=libdirac --enable-libfreetype --enable-libschroedinger --disable-encoder=libschroedinger --enable-version3 --enable-libopenjpeg --enable-libvpx --enable-librtmp --extra-libs=-lgcrypt --disable-altivec --disable-armv5te --disable-armv6 --disable-vis
libavutil 50. 43. 0 / 50. 43. 0
libavcodec 52.123. 0 / 52.123. 0
libavformat 52.111. 0 / 52.111. 0
libavdevice 52. 5. 0 / 52. 5. 0
libavfilter 1. 80. 0 / 1. 80. 0
libswscale 0. 14. 1 / 0. 14. 1
libpostproc 51. 2. 0 / 51. 2. 0

[h264 @ 0x24ed2e0] max_analyze_duration 5000000 reached at 5000000
[h264 @ 0x24ed2e0] Estimating duration from bitrate, this may be inaccurate

Seems stream 0 codec frame rate differs from container frame rate: 50.00 (50/1) -> 25.00 (50/2)
Input #0, h264, from '00000.MTS.ss438.h264':

Duration: N/A, bitrate: N/A

Stream #0.0: Video: h264 (High), yuv420p, 1440x1080 [PAR 4:3 DAR 16:9], 25 fps, 25 tbr, 1200k tbn, 50 tbc

At least one output file must be specified

comment:17 in reply to: ↑ 16 ; follow-up: Changed 6 years ago by cehoyos

Replying to jonb:

../ffmpeg -i 00000.MTS -vcodec libx264 -acodec copy -filter:v yadif -b:v 1000000 -r 25 -s 1440x1080 -pass 3 -passlogfile 00000.MTS 00000.MTS.ss438.h264

that does not overwrite any files, but now there is another problem: No sound?

(Complete, uncut console output missing.)

A h264 file (that is what your last line produces) cannot contain sound, the format only contains raw H264 video.

Carl Eugen

comment:18 in reply to: ↑ 17 ; follow-up: Changed 6 years ago by jonb

Replying to cehoyos:

A h264 file (that is what your last line produces) cannot contain sound, the format only contains raw H264 video.

okay, then I want to complain that FFmpeg continues regardless of conflicting options.

-acodec copy tells FFmpeg I want it to copy the sound. That means I want sound output. And if the output choice 00000.MTS.ss438.h264 indicates that I dont want sound, then there is a conflict which means FFmpeg should not continue. There is no reason to waste CPU cycles (and time) encoding something which is useless.

comment:19 in reply to: ↑ 18 ; follow-up: Changed 6 years ago by cehoyos

Replying to jonb:

There is no reason to waste CPU cycles (and time) encoding something which is useless.

No CPU cycles are wasted in the command line you use.

comment:20 in reply to: ↑ 19 Changed 6 years ago by cehoyos

Replying to cehoyos:

Replying to jonb:

There is no reason to waste CPU cycles (and time) encoding something which is useless.

No CPU cycles are wasted in the command line you use.

(Maybe except for the fact that some sources claim 3-pass encoding makes no sense.)

comment:21 follow-up: Changed 6 years ago by jonb

Yeah I noticed that when I reconnected my screen as well. The 3 pass must be a left over from when I experimented with mencoder. And the no sound must be an output from one of those previous runs when I did specify no sound.

comment:22 Changed 6 years ago by jonb

Anyway, back to the topic. Why havent http://ffmpeg.org/ffmpeg.html been updated to warn people that prior to version $insert_correct_version_here$ the passlogfile when choosing libx264 has a different name. Surely avoiding a dataloss should be high priority.

comment:23 follow-up: Changed 6 years ago by cehoyos

Do you know the value for $insert_correct_version_here$ ?

comment:24 in reply to: ↑ 23 Changed 6 years ago by jonb

Replying to cehoyos:

Do you know the value for $insert_correct_version_here$ ?

18 hours ago you did and wrote:

Version changed from unspecified to 0.7.13

This may have been fixed several months ago.
Could you test current git head?

And we found out it looks like it has been fixed. So did you guys publish any new versions between then and now ? If so that should be the value. Else you might pick the next version number.

comment:25 follow-up: Changed 6 years ago by cehoyos

Thanks to your work, we know now that the original bug was fixed between the time 0.7 was branched (June 2011 iirc) and yesterday's git head.
Unfortunately, several thousand changes were in-between, so we don't know (yet) if this is reproducible with 0.8, 0.9, 0.10 or 0.11...
If you want to help, please consider using git bisect to find the change fixing the original problem.

comment:26 in reply to: ↑ 25 ; follow-up: Changed 6 years ago by jonb

Replying to cehoyos:

Thanks to your work, we know now that the original bug was fixed between the time 0.7 was branched (June 2011 iirc) and yesterday's git head.
Unfortunately, several thousand changes were in-between, so we don't know (yet) if this is reproducible with 0.8, 0.9, 0.10 or 0.11...
If you want to help, please consider using git bisect to find the change fixing the original problem.

Does that mean I have to read code. I'm not so good at that. Couldnt we just start to write in the documentation that it has been discovered that there in some versions is a possibility for a dataloss when using libx264 and passlogfile?

Then we can look for the versions that overwrite or not.

I might script something that checks out every version from git between 0.7.13 and head and finds out when it was fixed.

comment:27 in reply to: ↑ 26 ; follow-up: Changed 6 years ago by cehoyos

Replying to jonb:

I might script something that checks out every version from git between 0.7.13 and head and finds out when it was fixed.

It will probably be enough to test 6968a7d and c2e3b56 (git checkout 6968a7d)

(Even if that surprisingly turns out to be not enough, git bisect would only require you to test ~12 revisions.)

comment:28 in reply to: ↑ 27 Changed 6 years ago by jonb

Replying to cehoyos:

Replying to jonb:

I might script something that checks out every version from git between 0.7.13 and head and finds out when it was fixed.

It will probably be enough to test 6968a7d and c2e3b56 (git checkout 6968a7d)

trying

comment:29 follow-up: Changed 6 years ago by jonb

jonbendtsen@media:/jon_compiles_ffmpeg$ mkdir c2e3b56
jonbendtsen@media:/jon_compiles_ffmpeg$ cd c2e3b56/
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ git clone --depth 1 git://source.ffmpeg.org/ffmpeg
Cloning into ffmpeg...
remote: Counting objects: 13044, done.
remote: Compressing objects: 100% (8272/8272), done.
remote: Total 13044 (delta 9107), reused 7128 (delta 4547)
Receiving objects: 100% (13044/13044), 13.49 MiB | 2.71 MiB/s, done.
Resolving deltas: 100% (9107/9107), done.
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ git checkout c2e3b56
fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ ls
ffmpeg
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ cd ffmpeg/
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56/ffmpeg$ git checkout c2e3b56
error: pathspec 'c2e3b56' did not match any file(s) known to git.
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56/ffmpeg$ git-checkout c2e3b56
-bash: git-checkout: command not found
jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56/ffmpeg$ git git-checkout c2e3b56
git: 'git-checkout' is not a git command. See 'git --help'.

comment:30 in reply to: ↑ 29 Changed 6 years ago by cehoyos

Replying to jonb:

jonbendtsen@media:/jon_compiles_ffmpeg/c2e3b56$ git clone --depth 1 git://source.ffmpeg.org/ffmpeg

Remove --depth 1

comment:31 Changed 6 years ago by jonb

The results are in: c2e3b56 is bad and it overwrites files. 6968a7d is good, it does not overwrite files.

Changed 6 years ago by jonb

Changed 6 years ago by jonb

comment:32 in reply to: ↑ 21 Changed 6 years ago by jonb

Replying to jonb:

Yeah I noticed that when I reconnected my screen as well. The 3 pass must be a left over from when I experimented with mencoder. And the no sound must be an output from one of those previous runs when I did specify no sound.

No, I found the source for my experiment with 3 pass encoding and ffmpeg, http://sonnati.wordpress.com/2011/08/19/ffmpeg-%E2%80%93-the-swiss-army-knife-of-internet-streaming-%E2%80%93-part-iii/

comment:33 Changed 6 years ago by cehoyos

  • Resolution set to fixed
  • Status changed from new to closed

I committed a documentation update for the next 0.7, 0.8, 0.9, 0.10 and 0.11 releases to clarify that there are two -passlogfile options with different syntax (depending on the used encoder).

I don't think we normally mention old behaviour in the current documentation, if you think a change is needed, change your local doc/ffmpeg.texi file and send the changes as unified diff to ffmpeg-devel.
See http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/150405 for the discussion so far.

Note: See TracTickets for help on using tickets.