Make Matroska cluster size configurable

[dave rice]

When using '-vcodec ffv1 outfile.mkv', I use mkvalidator to check the resulting file which lists warnings for every cluster.

Making a ffv1 in mkv: ​http://pastebin.com/dhAHLpvc

Warnings from mkvalidator that first frame is cluster is not keyframe: ​http://pastebin.com/TJHtaAvY (aren't all ffv1 frames keyframes)

Same file in mkvtree: ​http://pastebin.com/cZ3bjF7M

[dave rice]

mkvalidator 0011TS.mkv 
WRN0C0: First Block for video track #1 in Cluster at 5663321 is not a keyframe
WRN0C0: First Block for video track #1 in Cluster at 14063747 is not a keyframe
WRN0C0: First Block for video track #1 in Cluster at 22504112 is not a keyframe
WRN0C0: First Block for video track #1 in Cluster at 31003740 is not a keyframe
WRN0C0: First Block for video track #1 in Cluster at 39367787 is not a keyframe
WRN0C0: First Block for video track #1 in Cluster at 48174775 is not a keyframe
mkvalidator 0.3.4: the file appears to be valid
	file created with Lavf53.2.0 / Lavf53.2.0

[dave rice]

ffmpeg -i 0011TS.mov -vcodec ffv1 -an 0011TS.mkv
ffmpeg version git-N-30446-g06a9da7, Copyright (c) 2000-2011 the FFmpeg developers
  built on Jul 14 2011 17:46:08 with gcc 4.2.1 (Apple Inc. build 5666) (dot 3)
  configuration: --prefix=/packages/ffmpeg-head/runtime --enable-static --disable-shared --enable-postproc --enable-nonfree --enable-libx264 --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libtheora --enable-libxvid --enable-libvorbis --enable-libgsm --enable-libvpx --enable-avfilter --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-version3 --enable-libfreetype --disable-ffplay --disable-ffserver --disable-ffprobe --disable-network --disable-devices --disable-doc --extra-cflags=--static --extra-ldflags= --bindir=/packages/ffmpeg-head/dist/bin --incdir=/packages/ffmpeg-head/dist/include --libdir=/packages/ffmpeg-head/dist/lib --prefix=/usr/local --enable-runtime-cpudetect
  libavutil    51.  3. 0 / 51.  3. 0
  libavcodec   53.  6. 1 / 53.  6. 1
  libavformat  53.  2. 0 / 53.  2. 0
  libavdevice  53.  1. 0 / 53.  1. 0
  libavfilter   2. 11. 0 /  2. 11. 0
  libswscale    0. 14. 0 /  0. 14. 0
  libpostproc  51.  2. 0 / 51.  2. 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] Unimplemented container channel layout.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] Unimplemented container channel layout.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] Unimplemented container channel layout.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] Unimplemented container channel layout.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x101807c00] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.

Seems stream 0 codec frame rate differs from container frame rate: 29.97 (30000/1001) -> 29.97 (2997/100)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '0011TS.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2009-06-10 22:51:08
  Duration: 00:00:02.60, start: 0.000000, bitrate: 118341 kb/s
    Stream #0.0(eng): Video: dvvideo, yuv422p, 1280x1080 [PAR 3:2 DAR 16:9], 115084 kb/s, 29.97 fps, 29.97 tbr, 2997 tbn, 29.97 tbc
    Metadata:
      creation_time   : 2009-06-10 22:51:08
    Stream #0.1(eng): Audio: pcm_s16le, 48000 Hz, 1 channels, s16, 768 kb/s
    Metadata:
      creation_time   : 2009-06-10 22:51:08
    Stream #0.2(eng): Audio: pcm_s16le, 48000 Hz, 1 channels, s16, 768 kb/s
    Metadata:
      creation_time   : 2009-06-10 22:51:08
    Stream #0.3(eng): Audio: pcm_s16le, 48000 Hz, 1 channels, s16, 768 kb/s
    Metadata:
      creation_time   : 2009-06-10 22:51:08
    Stream #0.4(eng): Audio: pcm_s16le, 48000 Hz, 1 channels, s16, 768 kb/s
    Metadata:
      creation_time   : 2009-06-10 22:51:08
    Stream #0.5(eng): Data: tmcd / 0x64636D74, 0 kb/s
    Metadata:
      creation_time   : 2009-06-10 22:51:08
[buffer @ 0x101505a80] w:1280 h:1080 pixfmt:yuv422p tb:1/1000000 sar:3/2 sws_param:
Output #0, matroska, to '0011TS.mkv':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2009-06-10 22:51:08
    encoder         : Lavf53.2.0
    Stream #0.0(eng): Video: ffv1, yuv422p, 1280x1080 [PAR 3:2 DAR 16:9], q=2-31, 200 kb/s, 1k tbn, 29.97 tbc
    Metadata:
      creation_time   : 2009-06-10 22:51:08
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop, [?] for help
frame=   78 fps=  5 q=0.0 Lsize=   54037kB time=2.60 bitrate=170061.7kbits/s    
video:54035kB audio:0kB global headers:0kB muxing overhead 0.003204%

[Carl Eugen Hoyos]

Could you make 0011TS.mov available?

[dave rice]

I probably shouldn't (not my file), but I think this occurs on any file. Perhaps I can produce one with the new testsrc filter.

[reimar]

aren't all ffv1 frames keyframes

No, they depend on the arithmetic coding state of previous frames unless you use -g 1.
The matroska encoder opens a new cluster at the latest after 5 MB of data. Since your ffv1 frames are very large it can't do that without creating some clusters without keyframes.
This could probably be improved/made configurable in the muxer, but those are only warnings after all.

[dave rice]

Replying to reimar:

aren't all ffv1 frames keyframes

No, they depend on the arithmetic coding state of previous frames unless you use -g 1.
The matroska encoder opens a new cluster at the latest after 5 MB of data. Since your ffv1 frames are very large it can't do that without creating some clusters without keyframes.
This could probably be improved/made configurable in the muxer, but those are only warnings after all.

Thanks for the explanation of the 5MB limit. For now, I'll look into using '-g 1' or 'mkclean --remux' on the result.

[Carl Eugen Hoyos]

I believe this was implemented by Luca in 2013 and merged in ac957bc6