Muxing using av_interleaved_write_frame() does not interleave

[DeHackEd]

Summary of the bug:

When using av_interleaved_write_frame (or just ffmpeg straight up, since ffmpeg uses av_interleaved_write_frame) output packets are not sorted properly into the output. The dts in the output file skips around. This can be viewed with ffprobe or wireshark when using MPEG-TS output.

How to reproduce:

% ./ffmpeg -y -i input.mp4 -t 20 -c copy -f mpegts output-copy.ts
ffmpeg version N-86315-g0dea011 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-libx264 --extra-cflags='-Iarch=westmere' --enable-gpl --enable-nonfree --enable-libfreetype --cc=/opt/gcc-6.3.0/bin/gcc
  libavutil      55. 63.100 / 55. 63.100
  libavcodec     57. 96.101 / 57. 96.101
  libavformat    57. 72.101 / 57. 72.101
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 90.100 /  6. 90.100
  libswscale      4.  7.101 /  4.  7.101
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.72.101
  Duration: 00:23:12.22, start: 0.000000, bitrate: 5313 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], Closed Captions, 4924 kb/s, 29.97 fps, 29.97 tbr, 10000k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 384 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
    Side data:
      audio service type: main
Output #0, mpegts, to 'output-copy.ts':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.72.101
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 4924 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 10000k tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 384 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
    Side data:
      audio service type: main
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  600 fps=0.0 q=-1.0 Lsize=    7796kB time=00:00:19.98 bitrate=3195.4kbits/s speed= 166x    
video:6165kB audio:938kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.759675%

% ffprobe -show_packets output-copy.ts | egrep 'dts_time=|codec_type='
...
dts_time=19.918467
codec_type=audio
dts_time=19.960367
codec_type=video
dts_time=19.951833
codec_type=audio
dts_time=19.992367
codec_type=video
dts_time=19.985200
codec_type=audio
dts_time=20.024367
codec_type=video
dts_time=20.018567
codec_type=audio
dts_time=20.056367
codec_type=video
dts_time=20.051933
codec_type=audio
dts_time=20.088367
codec_type=video
dts_time=20.085300
codec_type=audio
dts_time=20.120367
codec_type=audio
dts_time=20.152367
codec_type=video
dts_time=20.118667
codec_type=audio
dts_time=20.184367
codec_type=video
dts_time=20.152033
codec_type=audio
dts_time=20.216367
codec_type=video
dts_time=20.185400
codec_type=audio
dts_time=20.248367
codec_type=video
dts_time=20.218767
...

dts_time should be monotonically increasing even with the intermixed codec_types. Instead each stream is independently increasing but globally the dts is out of order.

[Hendrik]

Are you sure its not just a demuxing artifact? Depending on the codec and if it needs parsing, demuxing could delay packets and throw them out of order.

[DeHackEd]

The issue was first noticed when trying to make a multi-bitrate HLS stream collection. When shifting between bitrates the audio might pop or video might glitch, but it's inconsistent. I traced it down to this problem. By replacing a call to av_interleaved_write_frame() with my own function that does the sorting and calls av_write_frame() itself, everything is okay now.

As an alternative to ffprobe you could open an mpegts file in Wireshark and view the transport stream chunks with it.