Regression: latest version drops frames when concatenating M2TS (to FFV1)

[Francois Visagie]

Concatenating from ffmpegclipfiles.txt:

ffconcat version 1.0
file 00000.MTS
file '00046 - Copy.MTS'

produces:

C:\Users\user\Documents\Working\vdcat>ffmpeg.new -y -f concat -safe 0 -i ffmpegclipfiles.txt -c:v ffv1 -c:a pcm_f32le ffmpegclips.new.avi
ffmpeg version N-75275-gd13a2df Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-l
ibilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --en
able-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc
--enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --ena
ble-decklink --enable-zlib
  libavutil      55.  2.100 / 55.  2.100
  libavcodec     57.  1.100 / 57.  1.100
  libavformat    57.  0.100 / 57.  0.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6.  3.100 /  6.  3.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, concat, from 'ffmpegclipfiles.txt':
  Duration: N/A, start: 0.520000, bitrate: 256 kb/s
    Stream #0:0: Video: h264 (High) (HDPR / 0x52504448), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
    Stream #0:1: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 256 kb/s
Output #0, avi, to 'ffmpegclips.new.avi':
  Metadata:
    ISFT            : Lavf57.0.100
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc
    Metadata:
      encoder         : Lavc57.1.100 ffv1
    Stream #0:1: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, stereo, flt, 3072 kb/s
    Metadata:
      encoder         : Lavc57.1.100 pcm_f32le
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> ffv1 (native))
  Stream #0:1 -> #0:1 (ac3 (native) -> pcm_f32le (native))
Press [q] to stop, [?] for help
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875520, current: 850759; changing to 875521. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875521, current: 852295; changing to 875522. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875522, current: 853831; changing to 875523. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875523, current: 855367; changing to 875524. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875524, current: 856903; changing to 875525. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875525, current: 858439; changing to 875526. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875526, current: 859975; changing to 875527. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875527, current: 861511; changing to 875528. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875528, current: 863047; changing to 875529. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875529, current: 864583; changing to 875530. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875530, current: 866119; changing to 875531. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875531, current: 867655; changing to 875532. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875532, current: 869191; changing to 875533. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875533, current: 870727; changing to 875534. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875534, current: 872263; changing to 875535. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875535, current: 873799; changing to 875536. This may result in incorrect timestamps in the
output file.
[avi @ 051364c0] Non-monotonous DTS in output stream 0:1; previous: 875536, current: 875335; changing to 875537. This may result in incorrect timestamps in the
output file.
Past duration 0.792763 too large
    Last message repeated 5 times
Past duration 0.792763 too large  578795kB time=00:00:18.55 bitrate=255521.2kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  586483kB time=00:00:18.74 bitrate=256263.8kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  594645kB time=00:00:18.94 bitrate=257195.9kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  601568kB time=00:00:19.16 bitrate=257149.3kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  609193kB time=00:00:19.38 bitrate=257400.0kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  616660kB time=00:00:19.58 bitrate=258000.1kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  623783kB time=00:00:19.80 bitrate=258028.3kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  631837kB time=00:00:19.99 bitrate=258850.2kbits/s dup=0 drop=25
    Last message repeated 8 times
Past duration 0.792763 too large  638183kB time=00:00:20.25 bitrate=258145.2kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  646583kB time=00:00:20.41 bitrate=259493.0kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  654655kB time=00:00:20.66 bitrate=259478.2kbits/s dup=0 drop=25
    Last message repeated 8 times
Past duration 0.792763 too large  661039kB time=00:00:20.86 bitrate=259597.1kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  669710kB time=00:00:21.05 bitrate=260603.4kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  676963kB time=00:00:21.27 bitrate=260652.4kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  684744kB time=00:00:21.46 bitrate=261290.7kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  692197kB time=00:00:21.72 bitrate=261021.9kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  700577kB time=00:00:21.91 bitrate=261867.4kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  708703kB time=00:00:22.17 bitrate=261846.2kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  715688kB time=00:00:22.36 bitrate=262156.9kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  723751kB time=00:00:22.62 bitrate=262110.0kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  732231kB time=00:00:22.81 bitrate=262949.2kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  739574kB time=00:00:23.03 bitrate=263003.8kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  748095kB time=00:00:23.22 bitrate=263834.9kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  756037kB time=00:00:23.45 bitrate=264089.0kbits/s dup=0 drop=25
    Last message repeated 8 times
Past duration 0.792763 too large  762628kB time=00:00:23.67 bitrate=263871.0kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  769615kB time=00:00:23.86 bitrate=264146.3kbits/s dup=0 drop=25
    Last message repeated 11 times
Past duration 0.792763 too large  778514kB time=00:00:24.09 bitrate=264716.6kbits/s dup=0 drop=25
    Last message repeated 10 times
Past duration 0.792763 too large  786104kB time=00:00:24.28 bitrate=265183.8kbits/s dup=0 drop=25
    Last message repeated 9 times
Past duration 0.792763 too large  793497kB time=00:00:24.47 bitrate=265578.1kbits/s dup=0 drop=25
    Last message repeated 2 times
frame= 1223 fps= 21 q=-0.0 Lsize=  795778kB time=00:00:24.47 bitrate=266341.5kbits/s dup=0 drop=25
video:786337kB audio:9384kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.007214%

The errors start appearing as soon as the boundary between clips is reached.
Clip ordering does not change the behaviour; it recurs at the new clip boundary.
Individually the clips encode fine.
Concatenating Lagarith-encoded AVI versions of the input clips works fine.

An older version of ffmpeg concatenates the same M2TS files as above to FFV1 without problems:

C:\Users\user\Documents\Working\vdcat>ffmpeg.old -y -f concat -safe 0 -i ffmpegclipfiles.txt -c:v ffv1 -c:a pcm_f32le ffmpegclips.old.avi
ffmpeg version N-54277-gff130d7 Copyright (c) 2000-2013 the FFmpeg developers
  built on Jun 30 2013 01:20:12 with gcc 4.7.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --
enable-libxvid --enable-zlib
  libavutil      52. 37.101 / 52. 37.101
  libavcodec     55. 17.100 / 55. 17.100
  libavformat    55. 10.100 / 55. 10.100
  libavdevice    55.  2.100 / 55.  2.100
  libavfilter     3. 77.101 /  3. 77.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
[concat @ 027a7a40] Estimating duration from bitrate, this may be inaccurate
Input #0, concat, from 'ffmpegclipfiles.txt':
  Duration: 00:00:00.04, start: 0.019967, bitrate: 12 kb/s
    Stream #0:0: Video: h264 (High) (HDPR / 0x52504448), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
    Stream #0:1: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 256 kb/s
Output #0, avi, to 'ffmpegclips.old.avi':
  Metadata:
    ISFT            : Lavf55.10.100
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 tbn, 50 tbc
    Stream #0:1: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, stereo, flt, 3072 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> ffv1)
  Stream #0:1 -> #0:1 (ac3 -> pcm_f32le)
Press [q] to stop, [?] for help
frame= 1248 fps= 12 q=-1.0 Lsize=  813483kB time=00:00:25.01 bitrate=266434.6kbits/s
video:804041kB audio:9384kB subtitle:0 global headers:0kB muxing overhead 0.007128%

Apart from absence of error messages, note the higher (and correct) frame count.

I’d be happy to upload the two M2TS input clips if need be.

[Carl Eugen Hoyos]

Please reopen this ticket if you can provide (short) input files.

[Francois Visagie]

I've uploaded

"Ticket #4853 concat from M2TS to FFV1 drops frames.txt" (205 bytes) and
"Ticket #4853 concat from M2TS to FFV1 drops frames.MTS" (8.08 MB)

to upload.ffmpeg.org/incoming.

The concat failure condition can be reproduced by concatenating multiple copies of "Ticket #4853 concat from M2TS to FFV1 drops frames.MTS", or the file with itself.

[Carl Eugen Hoyos]

If there is an issue, it is a regression since 1e5271a9fd6ddcceb083f2185a4bbd8d44c9a813 related to ticket #2143.