Opened 11 years ago
Last modified 11 years ago
#2801 new defect
Concat protocol with too many files
| Reported by: | Bernardo | Owned by: | |
|---|---|---|---|
| Priority: | normal | Component: | avformat |
| Version: | git-master | Keywords: | concat |
| Cc: | Blocked By: | ||
| Blocking: | Reproduced by developer: | no | |
| Analyzed by developer: | no |
Description
Summary of the bug:
When using the concat protocol with a certain number of files, it just crashes at some point when parsing the input. The open files limit in my server was 1024, so it crashes when trying to concatenate that amount of files.
I see two problems here:
1 - FFmpeg doesn't tell me whats wrong. It just ends the process with error code 1 and no useful information whatsoever. Even gdb can't show any information.
I had to dig in the "libavformat/concat.c" file with some prints to see the system error.
2 - Why are all the files being opened at once? There should be a way to call "ffurl_open" only when the file is going to be used.
The command input line is quite huge to put here so I just wrote the first two files.
FFmpeg will truncate the text when printing the names.
How to reproduce:
$ ffmpeg -v 99 -y -i "concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|..." -c:v copy foo.mp4 ffmpeg version N-38232-g1f7acf3 Copyright (c) 2000-2013 the FFmpeg developers built on Jul 21 2013 01:55:20 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5) configuration: --enable-gpl --enable-libass --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libspeex --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-x11grab --enable-libx264 --enable-nonfree --enable-version3 libavutil 52. 40.100 / 52. 40.100 libavcodec 55. 18.102 / 55. 18.102 libavformat 55. 12.102 / 55. 12.102 libavdevice 55. 3.100 / 55. 3.100 libavfilter 3. 81.102 / 3. 81.102 libswscale 2. 4.100 / 2. 4.100 libswresample 0. 17.102 / 0. 17.102 libpostproc 52. 3.100 / 52. 3.100 Splitting the commandline. Reading option '-v' ... matched as option 'v' (set logging level) with argument '99'. Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'. Reading option '-i' ... matched as input file with argument 'concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.hReading option '-c:v' ... matched as option 'c' (codec name) with argument 'copy'. Reading option 'foo.mp4' ... matched as output file. Finished splitting the commandline. Parsing a group of options: global . Applying option v (set logging level) with argument 99. Applying option y (overwrite output files) with argument 1. Successfully parsed a group of options. Parsing a group of options: input file concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.Successfully parsed a group of options. Opening an input file: concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.h264|/video/videconcat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.h264|/video/videox/CCD_orig_32.h264|/vi
The problem cannot be on x264 (or any other external library) because it happens while still parsing the text.
Change History (2)
follow-up: 2 comment:1 by , 11 years ago
comment:2 by , 11 years ago
Replying to Cigaes:
- An option to work in non-seekable mode, with only one file open at once.
This should be enough for my application (and probably easier to implement). I don't know if others might want the other option.
How would it be done? with a variation of concat protocol like "concat2:foo.ts|bar.ts|baz.ts" or a flag like -no-seek-input ?
Note:
See TracTickets
for help on using tickets.
Replying to jbvsmo:
That is not exactly true: ffmpeg tries to tell you what is going wrong, but the line it prints is huge and truncated by the colored log system, so you do not get the error message itself. Patch sent on the mailing list.
To implement seeking and report useful information, concat needs at least the size of all component, and lavf does not allow to get that without opening. Two useful options could be added to the concat protocol to mitigate this problem:
Both options make the protocol slightly less reliable, but that is up to the user to choose to use them. Both should be pretty easy to implement (but not as easy as the first problem).
Note that as an alternative to the concat protocol you can use the concat demuxer, it does not suffer from this problem. But it will only work if your streams are cleanly split before keyframes.