MPEG-2 muxer overshoots and corrects with undershoot in CBR mode (-muxrate)

[Warren Young]

If you use the -muxrate option with -c copy to turn a VBR file into a CBR output, ffmpeg's MPEG-TS muxer can approximately achieve your goals if you set the mux rate high enough to cover all of the peaks, but it has a bug that causes it to occasionally overshoot — sometimes massively — followed by an immediate undershoot correction.

This is a problem because when you need CBR, you really do need CBR, not CBR-ish. My local need is to allow Live555's media server (​http://live555.com/mediaServer/) to smoothly stream files, and it requires CBR in order to schedule a smooth output frame rate based on the input bit rate. Smooth streaming with VBR requires clever algorithms that simply aren't in Live555.

I've prepared some files and charts to show the problem:

​https://drive.google.com/drive/folders/1TKvC1_Mn8OvifsLPcG7tEhwWnq94iPE6?usp=sharing

The file input.mp4 is a highly-variable bitrate output from PowerPoint, since it's mostly still frames with occasional transitions. The input.png file shows the bit rate of this stream. Each bar represents 2 seconds of the video.

(These charts were created by etr-bv, an MIT-licensed tool I created based on ffprobe -show_frames: ​https://github.com/wyoung/etr-bv)

The file output.ts was created by a straightforward remux command. Something like:

$ ffmpeg -f mpegts -i input.mp4 -c copy -muxrate 13.5M output.ts

The -muxrate value was chosen to be +10% or so over the highest peak found in input.mp4.

This results in output.ts and output.png in the above-linked Google Drive folder. As you can see, it's mostly CBR, but it overshot the limit I gave it 4 times, with 4 immediate compensating corrections.

When this is bad enough, it causes buffer overruns and such in memory-constrained decoders, causing bad decodes.

If you modify the windowSize value in the etr-bv.R script from its default of 2 seconds, you can blur out these faults, but it isn't until you get to 10 seconds that it looks like it's truly CBR. That means you need at least 10 seconds worth of video buffering in the decoder to compensate for these overshoots! That's asking way too much for many embedded hardware decoders, particularly at such high bit rates.

---

How to reproduce: Take the files in the Google Drive and use my command above along with etr-bv to replicate the symptom. See the instructions on the GitHub home page for this project if you've never used R or RStudio.

This is an ongoing problem, probably first seen in the early 3.x series, and I just replicated it with tip-of-master. I've been hoping someone would notice and fix it before now, but since that apparently isn't going to happen, I've gathered the data to show the problem clearly here.

[Carl Eugen Hoyos]

This has currently no similarities with a valid ticket.

Please provide a working command line together with the complete, uncut console output to make this a valid ticket and explain how you tested the actual muxing rate.

[Warren Young]

Please provide a working command line

The above command line was constructed from memory, so yes, it fails, but all you have to do to fix is is swap the order of the -i and -f options. The fixed version is below.

---

complete, uncut console output

$ ffmpeg -i input.mp4 -f mpegts -c copy -muxrate 13.5M -y ~/Desktop/output.ts
ffmpeg version 4.1.4 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.4_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-12.0.1.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-12.0.1.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp41isom
    creation_time   : 2019-07-28T20:14:36.000000Z
  Duration: 00:01:26.63, start: 0.000000, bitrate: 2383 kb/s
    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 2382 kb/s, 30.30 fps, 30.30 tbr, 30k tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2019-07-28T20:14:36.000000Z
      handler_name    : VideoHandler
      encoder         : AVC Coding
Output #0, mpegts, to '/Users/tangent/Desktop/output.ts':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp41isom
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 2382 kb/s, 30.30 fps, 30.30 tbr, 90k tbn, 30k tbc (default)
    Metadata:
      creation_time   : 2019-07-28T20:14:36.000000Z
      handler_name    : VideoHandler
      encoder         : AVC Coding
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mpegts @ 0x7ff4d6800000] dts < pcr, TS is invalid:00:21.94 bitrate=13474.5kbits/s speed=40.5x    
    Last message repeated 31 times
[mpegts @ 0x7ff4d6800000] dts < pcr, TS is invalid:00:34.88 bitrate=13587.8kbits/s speed=  32x    
    Last message repeated 34 times
[mpegts @ 0x7ff4d6800000] dts < pcr, TS is invalid:00:43.23 bitrate=13486.2kbits/s speed=26.7x    
    Last message repeated 16 times
frame= 2625 fps=826 q=-1.0 Lsize=  142700kB time=00:01:26.59 bitrate=13500.0kbits/s speed=27.3x    
video:25198kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 466.314453%

This machine doesn't have the git-master build on it, so I'm using the latest stable release version instead, but as I wrote above, it's been happening for probably years now, so I don't believe we need to be too picky about which version we test with.

---

explain how you tested the actual muxing rate.

That's explained above: with etr-bv, an open source tool I wrote for the purpose and put on GitHub. It is a wrapper around ffprobe -show_frames output. Based on the reported pkt_pos values, I can get bits per frame, from which I can produce the bar charts I linked you to.

etr-bv runs using widely-available open source tools, so you can replicate the symptom there.

If you have a preferred bit rate graphing tool, I expect I can show the same symptom using it instead.

[Carl Eugen Hoyos]

Please test current FFmpeg git head, the only version supported here.

[Warren Young]

Quoting my original posting, "I just replicated it with tip-of-master." The output.ts file and the bit rate graphs in the Google Drive were produced with that version.

[Warren Young]

I've gotten back to the machine with a git-master build on it, updated it, and re-tested, and it's giving the exact same symptom on the input file. Here's the output:

$ ffmpeg -i input.mp4 -f mpegts -c copy -muxrate 13.5M -y output.ts
ffmpeg version N-94423-ga0c1970 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-36)
  configuration: --extra-cflags='-march=native' --enable-gpl --enable-libx264 --enable-libvpx
  libavutil      56. 32.100 / 56. 32.100
  libavcodec     58. 55.100 / 58. 55.100
  libavformat    58. 30.100 / 58. 30.100
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 58.100 /  7. 58.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp41isom
    creation_time   : 2019-07-28T20:14:36.000000Z
  Duration: 00:01:26.63, start: 0.000000, bitrate: 2383 kb/s
    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 2382 kb/s, 30.30 fps, 30.30 tbr, 30k tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2019-07-28T20:14:36.000000Z
      handler_name    : VideoHandler
      encoder         : AVC Coding
Output #0, mpegts, to 'output.ts':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp41isom
    encoder         : Lavf58.30.100
    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 2382 kb/s, 30.30 fps, 30.30 tbr, 90k tbn, 30k tbc (default)
    Metadata:
      creation_time   : 2019-07-28T20:14:36.000000Z
      handler_name    : VideoHandler
      encoder         : AVC Coding
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mpegts @ 0x449e380] dts < pcr, TS is invalid
    Last message repeated 83 times
frame= 2625 fps=0.0 q=-1.0 Lsize=  142700kB time=00:01:26.59 bitrate=13500.0kbits/s speed= 341x    
video:25198kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 466.314453%