Duplication of time stamps when muxing files in a matroska container

[lpaone]

Summary of the bug:
I am running into some issues when splitting and then concatenating matroska files using ffmpeg. I am splitting files and then merging the files right after.

My issue is that after the merge, the duration of the video is shorter than the original file by the number of merges that occurred, in frames. In one test, the merged video was shorter than the original by 7 frames, which is the number of merges that happened between the 8 files that came out after the split. I used mkvinfo to look at the details of the .webm file and found that the problem is that ffmpeg is giving the first frame of the second part of a merged video the same time stamp as the last frame of the first part. Therefore I have duplicate time stamps at the point of every merge and I end up loosing a frame because of it.

I have tried this with several different mkv/webm files and it happens with all of them. I also tried it with an Apple ProRes file and it does not occur. Therefore I believe that this is a specific issue with muxing Matroska files.

How to reproduce:

The demuxing output is here:

%ffmpeg -i D:\Videos\SwP\sp.webm -codec copy -f segment -segment_time 22 -reset_timestamps 1 D:\Videos\SplitTest\swp_%04d.webm
ffmpeg version N-69156-gff5b9a1c Copyright (c) 2000-2015 the FFmpeg developers
  built on Jan 20 2015 00:38:00 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      54. 17.100 / 54. 17.100
  libavcodec     56. 20.100 / 56. 20.100
  libavformat    56. 19.100 / 56. 19.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5.  8.100 /  5.  8.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, matroska,webm, from 'D:\Videos\SwP\sp.webm':
  Metadata:
    encoder         : libwebm-0.2.1.0
  Duration: 00:02:47.21, start: 0.000000, bitrate: 5728 kb/s
    Stream #0:0(eng): Video: vp9, yuv420p, 1920x1080, SAR 1:1 DAR 16:9, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Output #0, segment, to 'D:\Videos\SplitTest\swp3_%04d.webm':
  Metadata:
    encoder         : Lavf56.19.100
    Stream #0:0(eng): Video: vp9, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame= 1648 fps=0.0 q=-1.0 size=N/A time=00:01:08.62 bitrate=N/A    
frame= 3602 fps=3602 q=-1.0 size=N/A time=00:02:30.04 bitrate=N/A    
frame= 4014 fps=3748 q=-1.0 Lsize=N/A time=00:02:47.20 bitrate=N/A    
video:116892kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

This works fine and the time stamps are correct on the separated files. I then take the separated files and try to merge them using the "-concat" command. This should merge the files together into a single continuous video without dropping any frames. I use the following command line to do this:

% ffmpeg -f concat -i D:\Videos\SplitTest\spList.txt -c copy D:\Videos\SplitTest\swp_merged.webm
ffmpeg version N-69156-gff5b9a1c Copyright (c) 2000-2015 the FFmpeg developers
  built on Jan 20 2015 00:38:00 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      54. 17.100 / 54. 17.100
  libavcodec     56. 20.100 / 56. 20.100
  libavformat    56. 19.100 / 56. 19.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5.  8.100 /  5.  8.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, concat, from 'D:\Videos\SplitTest\spList.txt':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: vp9, yuv420p, 1920x1080, SAR 1:1 DAR 16:9, 1k fps, 24 tbr, 1k tbn, 1k tbc
Output #0, webm, to 'D:\Videos\SplitTest\swp_merged.webm':
  Metadata:
    encoder         : Lavf56.19.100
    Stream #0:0: Video: vp9, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 1k fps, 24 tbr, 1k tbn, 1k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame= 2864 fps=0.0 q=-1.0 size=   86738kB time=00:01:59.08 bitrate=5966.9kbits/s    
frame= 4014 fps=0.0 q=-1.0 Lsize=  116925kB time=00:02:46.91 bitrate=5738.5kbits/s    
video:116892kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.027859%

Here is the FFPROBE output for the last frame of the 1st file in the merge and the 1st frame on the 2nd file of the merge after the merge:

...
[FRAME]
media_type=video
key_frame=0
pkt_pts=24958
pkt_pts_time=24.958000
pkt_dts=24958
pkt_dts_time=24.958000
best_effort_timestamp=24958
best_effort_timestamp_time=24.958000
pkt_duration=N/A
pkt_duration_time=N/A
pkt_pos=18554505
pkt_size=25065
width=1920
height=1080
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=P
coded_picture_number=0
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
[/FRAME]
[FRAME]
media_type=video
key_frame=1
pkt_pts=24958
pkt_pts_time=24.958000a
pkt_dts=24958
pkt_dts_time=24.958000
best_effort_timestamp=24958
best_effort_timestamp_time=24.958000
pkt_duration=N/A
pkt_duration_time=N/A
pkt_pos=18579594
pkt_size=98654
width=1920
height=1080
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=I
coded_picture_number=0
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
[/FRAME]
...

As you can see, the frames have the same time stamps. This occurs in the exact same way at the point of every merge.

[Ronald S. Bultje]

This is because the vp9 parser (which should really be a bitstream filter) splits superframes into frames (to split the altref/invisible reference frame from the following/visible frame). But when muxing the file back to matroska, we don't merge altref frames together into superframes with their subsequent visible frame, which causes the discrepancy.

The correct fix is to write a bitstream filter that undoes vp9_parser's superframe splitting and use that before muxing vp9 into any output file type, be that ivf, webm or something else.

[Ronald S. Bultje]

This patch fixes the issue: ​http://ffmpeg.org/pipermail/ffmpeg-devel/2016-February/189710.html

[Ronald S. Bultje]

Fixed in 2e6636aa87303d37b112e79f093ca39500f92364.