Impossible to play incomplete mkv videos

[thagoodboy]

Summary of the bug:

While encoding H.264 videos with ffmpeg in the past, i was able to play the output video in the middle of the encoding, with the condition of using the "mkv" or "ts" containers for that output, but with the last ffmpeg versions this is not possible, so i have to wait until finishing the encoding (this may take hours or days in my case) to play the resulting video.

How to reproduce:
ffmpeg -i anyvideo -c:v libx264 -c:a copy output.mkv

Note:
Please try to play the output video with Potplayer or VLC.

ffmpeg version N-82597-gd316b21
built on 2016/11/22

Thanks

[Carl Eugen Hoyos]

For future tickets: Please remember to always provide a minimal command line that allows to reproduce the issue together with the complete, uncut console output.
Regression since ee888cfbe777cd2916a3548c750e433ab8f8e6a5

$ ffmpeg -f lavfi -i testsrc out.mkv
ffmpeg version N-82629-g9955123 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.107 / 57. 66.107
  libavformat    57. 58.100 / 57. 58.100
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Output #0, matroska, to 'out.mkv':
  Metadata:
    encoder         : Lavf57.58.100
    Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 1k tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.66.107 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
Press [q] to stop, [?] for help
frame= 6069 fps=1286 q=9.9 Lsize=    6265kB time=00:04:02.72 bitrate= 211.4kbits/s speed=51.4x

$ ffmpeg -i out.mkv
ffmpeg version N-82629-g9955123 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.107 / 57. 66.107
  libavformat    57. 58.100 / 57. 58.100
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
[matroska,webm @ 0x204b280] Duplicate element
Input #0, matroska,webm, from 'out.mkv':
  Duration: N/A, bitrate: N/A
At least one output file must be specified

It is unlikely that such an issue is reproducible with mpegts output.

[thagoodboy]

You are right, i'm sorry "ts" seems to be working fine unlike "mkv"
By the way, the last ffmpeg version which doesn't show this issue is: N-81894-g54220ce built on 2016-10-05

[softworkz]

Developed a patch. Will submit soon.

[softworkz]

Patch details:

The following three commits created a regression by writing initially invalid mkv headers:

650e17d88b63b5aca6e0a43483e89e64b0f7d2dd avformat/matroskaenc: write a CRC32 element on Tags
3bcadf822711720ff0f8d14db71ae47cdf97e652 avformat/matroskaenc: write a CRC32 element on Info
ee888cfbe777cd2916a3548c750e433ab8f8e6a5 avformat/matroskaenc: postpone writing the Tracks master

Symptoms:

• You can no longer playback a file that is still processed by ffmpeg, e.g. VLC fails playback
• You can no longer stream a file to a client while if is still being processed
• Various diagnosing tools show header errors or incomplete headers (e.g. ffprobe, mediainfo, mkvalidator)

Note: The symptoms do not apply to completed files or ffmpeg runs that were interrupted with 'q'

Cause:

The mentioned commits made changes in a way that some header elements are only partially written in
mkv_write_header, leaving the header in an invalid state. Only in mkv_write_trailer, these elements
are finished correctly, but that does only occur at the end of the process.

Regression:

Before these commits were applied, mkv headers have always been valid, even before completion of ffmpeg.
This has worked reliably over many versions of ffmpeg, to it was an obvious regression.

Bugtracker:

This issue has been recorded as #5977 which is resolved by this patch

Patch:

The patch adds a new function 'end_ebml_master_crc32_preliminary' that preliminarily finishes the ebl
element without destroying the buffer. The buffer can be used to update the ebml element later during
mkv_write_trailer. But most important: mkv_write_header finishes with a valid mkv header again.

[James]

Fixed in 20e8be0c20c7b51964fa4d317073bd36b983eb55 by softworkz.