Incorrect video start time when muxing with opus audio

[jstrot]

Problem description

In its simplest form, my goal is to create an mkv by muxing a video (vp9) track and an audio (opus) track. Should be a simple task but I'm facing issues with the resulting video start time.

Due to the inherent 6.5 ms (312 * 48K) delay of the opus codec, the audio start time is expected to be -0.007ms but the video should still start at 0ms yet ffmpeg muxes with the video starting at +0.007ms:

$ ffmpeg -i track0.vp9.ivf -i track1.opus.ogg -codec copy output.mkv
ffmpeg version 4.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8 (Debian 8.2.0-10)
  configuration: --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-stripping --enable-libaom --enable-avfilter --enable-avresample --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, ivf, from 'track0.vp9.ivf':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 720x480, 23.98 tbr, 23.98 tbn, 23.98 tbc
[ogg @ 0x56377efc2500] 654 bytes of comment header remain
Input #1, ogg, from 'track1.opus.ogg':
  Duration: 00:10:00.49, start: 0.000000, bitrate: 409 kb/s
    Stream #1:0: Audio: opus, 48000 Hz, 5.1, fltp
    Metadata:
      ENCODER         : opusenc from opus-tools 0.1.10
      ENCODER_OPTIONS : --vbr --bitrate 448
Output #0, matroska, to 'output.mkv':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 720x480, q=2-31, 23.98 tbr, 1k tbn, 23.98 tbc
    Stream #0:1: Audio: opus ([255][255][255][255] / 0xFFFFFFFF), 48000 Hz, 5.1, fltp
    Metadata:
      ENCODER         : opusenc from opus-tools 0.1.10
      ENCODER_OPTIONS : --vbr --bitrate 448
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=14397 fps=0.0 q=-1.0 Lsize=   95680kB time=00:10:00.47 bitrate=1305.3kbits/s speed=1.48e+03x    
video:65466kB audio:29900kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.329659%

$ ffprobe -i output.mkv -show_streams 2>/dev/null | grep start_time
start_time=0.007000
start_time=-0.007000

$ ffprobe -i output.mkv -show_packets -of compact 2>/dev/null | head -2
packet|codec_type=audio|stream_index=1|pts=-7|pts_time=-0.007000|dts=-7|dts_time=-0.007000|duration=20|duration_time=0.020000|convergence_duration=N/A|convergence_duration_time=N/A|size=1737|pos=924|flags=K_
packet|codec_type=video|stream_index=0|pts=7|pts_time=0.007000|dts=7|dts_time=0.007000|duration=N/A|duration_time=N/A|convergence_duration=N/A|convergence_duration_time=N/A|size=10995|pos=2668|flags=K_

If instead I use an ac3 audio track (w/o delay) then all tracks are muxed withno delay, as expected:

$ ffmpeg -i track0.vp9.ivf -i track1.ac3 -codec copy output.mkv
...
Input #0, ivf, from 'track0.vp9.ivf':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: vp9 (Profile 0) (VP90 / 0x30395056), yuv420p(tv), 720x480, 23.98 tbr, 23.98 tbn, 23.98 tbc
[ac3 @ 0x55fdae1a1500] Estimating duration from bitrate, this may be inaccurate
Input #1, ac3, from 'track1.ac3':
  Duration: 00:10:00.48, start: 0.000000, bitrate: 448 kb/s
    Stream #1:0: Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s
...

$ ffprobe -i output.mkv -show_streams 2>/dev/null | grep start_time
start_time=0.000000
start_time=0.000000

$ ffprobe -i output.mkv -show_packets -of compact 2>/dev/null | head -2
packet|codec_type=video|stream_index=0|pts=0|pts_time=0.000000|dts=0|dts_time=0.000000|duration=N/A|duration_time=N/A|convergence_duration=N/A|convergence_duration_time=N/A|size=10995|pos=753|flags=K_
packet|codec_type=audio|stream_index=1|pts=0|pts_time=0.000000|dts=0|dts_time=0.000000|duration=32|duration_time=0.032000|convergence_duration=N/A|convergence_duration_time=N/A|size=1792|pos=11755|flags=K_

Workaround

I tried all sorts of combinations of -itsoffset, -copyts, -start_at_zero, -avoid_negative_ts, ... the only workaround I found (by luck) was to provide a large (-10) negative itsoffset for the audio track and suddenly ffmpeg stops applying a delay to the video track.

$ ffmpeg -i track0.vp9.ivf -itsoffset -10 -i track1.opus.ogg -codec copy output.mkv
...
$ ffprobe -i output.mkv -show_streams 2>/dev/null | grep start_time
start_time=0.000000
start_time=-0.007000

$ ffprobe -i output.mkv -show_packets -of compact 2>/dev/null | head -2
packet|codec_type=video|stream_index=0|pts=0|pts_time=0.000000|dts=0|dts_time=0.000000|duration=N/A|duration_time=N/A|convergence_duration=N/A|convergence_duration_time=N/A|size=10995|pos=924|flags=K_
packet|codec_type=audio|stream_index=1|pts=-7|pts_time=-0.007000|dts=-7|dts_time=-0.007000|duration=20|duration_time=0.020000|convergence_duration=N/A|convergence_duration_time=N/A|size=1737|pos=11926|flags=K_

Other observations

• With multiple audio tracks the -itsoffset has to be specified for each audio track.

• Compared number of video and audio packets in ffprobe -show_packets output and all are accounted for.

• With other itsoffset values there are weird behaviors too! With 0 to -5 it just adds a delay to the video track, with -6 to -9 it aligns both to -7ms:

with -itsoffset 0:
start_time=0.007000
start_time=-0.007000

with -itsoffset -1:
start_time=1.007000
start_time=-0.007000

with -itsoffset -2:
start_time=2.007000
start_time=-0.007000

...
with -itsoffset -5:
start_time=5.007000
start_time=-0.007000

with -itsoffset -6 to -9:
start_time=-0.007000
start_time=-0.007000

with -itsoffset -10 to -99:59:59.99999999:
start_time=0.000000
start_time=-0.007000

[Carl Eugen Hoyos]

For future bug reports: Please remember that only current FFmpeg git head is supported here.

Looks like a duplicate of ticket #7182.