Opened 5 years ago

Last modified 5 years ago

#2801 new defect

Concat protocol with too many files

Reported by: jbvsmo Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: concat
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
When using the concat protocol with a certain number of files, it just crashes at some point when parsing the input. The open files limit in my server was 1024, so it crashes when trying to concatenate that amount of files.

I see two problems here:

1 - FFmpeg doesn't tell me whats wrong. It just ends the process with error code 1 and no useful information whatsoever. Even gdb can't show any information.

I had to dig in the "libavformat/concat.c" file with some prints to see the system error.

2 - Why are all the files being opened at once? There should be a way to call "ffurl_open" only when the file is going to be used.

The command input line is quite huge to put here so I just wrote the first two files.
FFmpeg will truncate the text when printing the names.

How to reproduce:

$ ffmpeg -v 99 -y -i "concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|..." -c:v copy foo.mp4

ffmpeg version N-38232-g1f7acf3 Copyright (c) 2000-2013 the FFmpeg developers
  built on Jul 21 2013 01:55:20 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
  configuration: --enable-gpl --enable-libass --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libspeex --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-x11grab --enable-libx264 --enable-nonfree --enable-version3
  libavutil      52. 40.100 / 52. 40.100
  libavcodec     55. 18.102 / 55. 18.102
  libavformat    55. 12.102 / 55. 12.102
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 81.102 /  3. 81.102
  libswscale      2.  4.100 /  2.  4.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '99'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-i' ... matched as input file with argument 'concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.hReading option '-c:v' ... matched as option 'c' (codec name) with argument 'copy'.
Reading option 'foo.mp4' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 99.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input file concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.Successfully parsed a group of options.
Opening an input file: concat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.h264|/video/videconcat:/video/videox/CCD_orig_0.h264|/video/videox/CCD_orig_1.h264|/video/videox/CCD_orig_2.h264|/video/videox/CCD_orig_3.h264|/video/videox/CCD_orig_4.h264|/video/videox/CCD_orig_5.h264|/video/videox/CCD_orig_6.h264|/video/videox/CCD_orig_7.h264|/video/videox/CCD_orig_8.h264|/video/videox/CCD_orig_9.h264|/video/videox/CCD_orig_10.h264|/video/videox/CCD_orig_11.h264|/video/videox/CCD_orig_12.h264|/video/videox/CCD_orig_13.h264|/video/videox/CCD_orig_14.h264|/video/videox/CCD_orig_15.h264|/video/videox/CCD_orig_16.h264|/video/videox/CCD_orig_17.h264|/video/videox/CCD_orig_18.h264|/video/videox/CCD_orig_19.h264|/video/videox/CCD_orig_20.h264|/video/videox/CCD_orig_21.h264|/video/videox/CCD_orig_22.h264|/video/videox/CCD_orig_23.h264|/video/videox/CCD_orig_24.h264|/video/videox/CCD_orig_25.h264|/video/videox/CCD_orig_26.h264|/video/videox/CCD_orig_27.h264|/video/videox/CCD_orig_28.h264|/video/videox/CCD_orig_29.h264|/video/videox/CCD_orig_30.h264|/video/videox/CCD_orig_31.h264|/video/videox/CCD_orig_32.h264|/vi

The problem cannot be on x264 (or any other external library) because it happens while still parsing the text.

Change History (2)

comment:1 in reply to: ↑ description ; follow-up: Changed 5 years ago by Cigaes

Replying to jbvsmo:

1 - FFmpeg doesn't tell me whats wrong. It just ends the process with error code 1 and no useful information whatsoever. Even gdb can't show any information.

That is not exactly true: ffmpeg tries to tell you what is going wrong, but the line it prints is huge and truncated by the colored log system, so you do not get the error message itself. Patch sent on the mailing list.

2 - Why are all the files being opened at once? There should be a way to call "ffurl_open" only when the file is going to be used.

To implement seeking and report useful information, concat needs at least the size of all component, and lavf does not allow to get that without opening. Two useful options could be added to the concat protocol to mitigate this problem:

  • An option to open only one file at once, and close it when no longer necessary. Even better, an option to limit the number of opened files to any value, using a LRU logic.
  • An option to work in non-seekable mode, with only one file open at once.

Both options make the protocol slightly less reliable, but that is up to the user to choose to use them. Both should be pretty easy to implement (but not as easy as the first problem).

Note that as an alternative to the concat protocol you can use the concat demuxer, it does not suffer from this problem. But it will only work if your streams are cleanly split before keyframes.

comment:2 in reply to: ↑ 1 Changed 5 years ago by jbvsmo

Replying to Cigaes:

  • An option to work in non-seekable mode, with only one file open at once.

This should be enough for my application (and probably easier to implement). I don't know if others might want the other option.

How would it be done? with a variation of concat protocol like "concat2:foo.ts|bar.ts|baz.ts" or a flag like -no-seek-input ?

Note: See TracTickets for help on using tickets.