Opened 6 years ago
Last modified 6 years ago
#7257 new defect
concatenating MP4 files reads all of them twice
Reported by: | AyrA | Owned by: | |
---|---|---|---|
Priority: | normal | Component: | undetermined |
Version: | git-master | Keywords: | concat mov |
Cc: | Blocked By: | ||
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
Summary of the bug
When concatenating multiple mp4 files using -f concat
each input file is read twice but only on the second pass the file is actually processed. This problem does not seem to be present for other formats like AVI or MKV.
The generated output file looks as expected and works.
How to reproduce
Command: ffmpeg -f concat -safe 0 -i list.txt -c copy -strict experimental output.mp4
list.txt: File with at least one mp4 file in it
Other Info
- My device runs Microsoft Windows 7 x64 (6.1.7601)
- The first seemingly useless pass is slower than the second actual pass but CPU utilization stays almost at 0 during the entire time of the first pass (Unbuffered IO or seeking weirdly?)
- First pass is performed after 51 frames have been written to the output according to the console status line
- This behavior is only present when using
-f concat
and not when processing a single file - Each mp4 input file is processed twice
- The concat text file can contain different container (mixed mkv and mp4 for example) and only the MP4 files are processed twice
- The issue is present regardless of file location (Tested: local disk and SMB share)
- This issue almost doubles encoding time
Version Info
ffmpeg version N-91266-g8c20ea8ee0 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 7.3.0 (GCC) configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth libavutil 56. 18.102 / 56. 18.102 libavcodec 58. 20.101 / 58. 20.101 libavformat 58. 17.100 / 58. 17.100 libavdevice 58. 4.100 / 58. 4.100 libavfilter 7. 25.100 / 7. 25.100 libswscale 5. 2.100 / 5. 2.100 libswresample 3. 2.100 / 3. 2.100 libpostproc 55. 2.100 / 55. 2.100
Attachments (1)
Change History (10)
by , 6 years ago
Attachment: | ffmpeg-20180613-123713.log.xz added |
---|
comment:1 by , 6 years ago
Where should we put the information that compressing the log is (extremely) counter-productive?
comment:2 by , 6 years ago
Where should I put the info that the log you request me to add (https://ffmpeg.org/bugreports.html) does not fit the size constraints of this bug tracker? Increase the limit and I upload the uncompressed log. It's 39'031'691 bytes.
comment:3 by , 6 years ago
Hard to believe that the issue is not reproducible with a shorter encode.
How did you create the input file?
comment:4 by , 6 years ago
The files come from a GoPro camera. The camera splits them into 4GB chunks during creation and there is no way of avoiding it.
comment:5 by , 6 years ago
Is the issue reproducible if you convert the input files with qt-faststart
before feeding them to ffmpeg
?
comment:6 by , 6 years ago
Nope. It still stalls for each media file.
What did help though was doing ffmpeg.exe -i gopro_file.mp4 -c copy output_file.mp4
for each individual file and then concatenating these files instead. Not sure why but it looks like the GoPro writes media files in a way that messes up -f concat
but works fine for individual files.
comment:9 by , 6 years ago
Keywords: | concat mov added |
---|
The issue can now be shown with loglevel verbose
, I don't know if this is a bug that can be fixed or an unavoidable limitation of the concat demuxer.
$ ffmpeg -loglevel verbose -f concat -i concat.txt -t 1 -f null - ffmpeg version N-91317-gb41b6b3 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 6.4.0 (GCC) configuration: --enable-gpl libavutil 56. 18.102 / 56. 18.102 libavcodec 58. 20.102 / 58. 20.102 libavformat 58. 17.100 / 58. 17.100 libavdevice 58. 4.101 / 58. 4.101 libavfilter 7. 25.100 / 7. 25.100 libswscale 5. 2.100 / 5. 2.100 libswresample 3. 2.100 / 3. 2.100 libpostproc 55. 2.100 / 55. 2.100 [h264 @ 0x3ee14c0] Reinit context to 1920x1088, pix_fmt: yuvj420p [mov,mp4,m4a,3gp,3g2,mj2 @ 0x3edf880] Auto-inserting h264_mp4toannexb bitstream filter [h264 @ 0x3ee52c0] Reinit context to 1920x1088, pix_fmt: yuvj420p [concat @ 0x3ed6500] Could not find codec parameters for stream 2 (Unknown: none): unknown codec Consider increasing the value for the 'analyzeduration' and 'probesize' options [concat @ 0x3ed6500] Could not find codec parameters for stream 3 (Unknown: none): unknown codec Consider increasing the value for the 'analyzeduration' and 'probesize' options [concat @ 0x3ed6500] Could not find codec parameters for stream 4 (Unknown: none): unknown codec Consider increasing the value for the 'analyzeduration' and 'probesize' options Input #0, concat, from 'concat.txt': Duration: N/A, start: 0.000000, bitrate: 47190 kb/s Stream #0:0(eng): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuvj420p(pc, bt709, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 47001 kb/s, 50 fps, 50 tbr, 90k tbn, 100 tbc Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AVC encoder : GoPro AVC encoder timecode : 18:00:58:12 Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AAC timecode : 18:00:58:12 Stream #0:2: Unknown: none Stream #0:3: Unknown: none Stream #0:4: Unknown: none Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native)) Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native)) Press [q] to stop, [?] for help [h264 @ 0x4b55840] Reinit context to 1920x1088, pix_fmt: yuvj420p [graph_1_in_0_1 @ 0x3f18ec0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3 [format_out_0_1 @ 0x3f19540] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_1' [auto_resampler_0 @ 0x3f1a580] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz [graph 0 input from stream 0:0 @ 0x3f2fd80] w:1920 h:1080 pixfmt:yuvj420p tb:1/90000 fr:50/1 sar:1/1 sws_param:flags=2 Output #0, null, to 'pipe:': Metadata: encoder : Lavf58.17.100 Stream #0:0(eng): Video: wrapped_avframe, 1 reference frame, yuvj420p(left), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AVC timecode : 18:00:58:12 encoder : Lavc58.20.102 wrapped_avframe Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AAC timecode : 18:00:58:12 encoder : Lavc58.20.102 pcm_s16le No more output streams to write to, finishing.:01.00 bitrate=N/A speed=0.0226x frame= 50 fps=1.1 q=-0.0 Lsize=N/A time=00:00:01.00 bitrate=N/A speed=0.0224x video:26kB audio:188kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Input file #0 (concat.txt): Input stream #0:0 (video): 59 packets read (7258062 bytes); 52 frames decoded; Input stream #0:1 (audio): 55 packets read (27760 bytes); 55 frames decoded (56320 samples); Input stream #0:2 ((null)): 1 packets read (4 bytes); Input stream #0:3 ((null)): 1 packets read (4428 bytes); Input stream #0:4 ((null)): 66362 packets read (1062472 bytes); Total: 66478 packets (8352726 bytes) demuxed Output file #0 (pipe:): Output stream #0:0 (video): 50 frames encoded; 50 packets muxed (26800 bytes); Output stream #0:1 (audio): 47 frames encoded (48000 samples); 47 packets muxed (192000 bytes); Total: 97 packets (218800 bytes) muxed [AVIOContext @ 0x3ee8100] Statistics: 1117648442 bytes read, 33854 seeks [AVIOContext @ 0x3ed6c80] Statistics: 19 bytes read, 0 seeks
$ ffmpeg -loglevel verbose -i GH010051.MP4 -t 1 -f null - ffmpeg version N-91317-gb41b6b3 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 6.4.0 (GCC) configuration: --enable-gpl libavutil 56. 18.102 / 56. 18.102 libavcodec 58. 20.102 / 58. 20.102 libavformat 58. 17.100 / 58. 17.100 libavdevice 58. 4.101 / 58. 4.101 libavfilter 7. 25.100 / 7. 25.100 libswscale 5. 2.100 / 5. 2.100 libswresample 3. 2.100 / 3. 2.100 libpostproc 55. 2.100 / 55. 2.100 [h264 @ 0x3c00700] Reinit context to 1920x1088, pix_fmt: yuvj420p Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'GH010051.MP4': Metadata: major_brand : mp41 minor_version : 538120216 compatible_brands: mp41 creation_time : 2018-06-13T18:00:58.000000Z firmware : HD6.01.02.01.00 Duration: 00:11:18.02, start: 0.000000, bitrate: 47250 kb/s Stream #0:0(eng): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuvj420p(pc, bt709, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 47001 kb/s, 50 fps, 50 tbr, 90k tbn, 100 tbc (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AVC encoder : GoPro AVC encoder timecode : 18:00:58:12 Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AAC timecode : 18:00:58:12 Stream #0:2(eng): Data: none (tmcd / 0x64636D74) (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro TCD timecode : 18:00:58:12 Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 35 kb/s (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro MET Stream #0:4(eng): Data: none (fdsc / 0x63736466), 12 kb/s (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro SOS Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native)) Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native)) Press [q] to stop, [?] for help [h264 @ 0x3c273c0] Reinit context to 1920x1088, pix_fmt: yuvj420p [graph_1_in_0_1 @ 0x3eb04c0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3 [format_out_0_1 @ 0x3eb0e40] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_1' [auto_resampler_0 @ 0x3eb2340] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz [graph 0 input from stream 0:0 @ 0x4665580] w:1920 h:1080 pixfmt:yuvj420p tb:1/90000 fr:50/1 sar:1/1 sws_param:flags=2 Output #0, null, to 'pipe:': Metadata: major_brand : mp41 minor_version : 538120216 compatible_brands: mp41 firmware : HD6.01.02.01.00 encoder : Lavf58.17.100 Stream #0:0(eng): Video: wrapped_avframe, 1 reference frame, yuvj420p(left), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AVC timecode : 18:00:58:12 encoder : Lavc58.20.102 wrapped_avframe Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default) Metadata: creation_time : 2018-06-13T18:00:58.000000Z handler_name : GoPro AAC timecode : 18:00:58:12 encoder : Lavc58.20.102 pcm_s16le No more output streams to write to, finishing. frame= 50 fps=0.0 q=-0.0 Lsize=N/A time=00:00:01.00 bitrate=N/A speed=4.76x video:26kB audio:188kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Input file #0 (GH010051.MP4): Input stream #0:0 (video): 59 packets read (7257987 bytes); 52 frames decoded; Input stream #0:1 (audio): 55 packets read (27760 bytes); 55 frames decoded (56320 samples); Input stream #0:2 (data): 1 packets read (4 bytes); Input stream #0:3 (data): 0 packets read (0 bytes); Input stream #0:4 (data): 4 packets read (744 bytes); Total: 119 packets (7286495 bytes) demuxed Output file #0 (pipe:): Output stream #0:0 (video): 50 frames encoded; 50 packets muxed (26800 bytes); Output stream #0:1 (audio): 47 frames encoded (48000 samples); 47 packets muxed (192000 bytes); Total: 97 packets (218800 bytes) muxed [AVIOContext @ 0x3c07180] Statistics: 8451658 bytes read, 4 seeks
The seeks/reads take considerable time, concat.txt contains only the line file GH010051.MP4
, remuxing the input file with FFmpeg fixes the issue.
compressed ffmpeg log using -report