Gaps in input timestamps cause invalid Ogg Opus output

[MarkZV]

When transcoding to Ogg Opus from a format with per-packet timestamps, such as mp4 audio, if the input contains a gap in packet timestamps (e.g. due to a lost/corrupted packet), FFmpeg writes invalid Ogg Opus output. Specifically, the Ogg granule position written by FFmpeg increments by more than the number of samples contained in packets that complete on the page.

This violates ​section 4 of the specification and may cause synchronization problems.

Because the Ogg container does not have explicit per-frame or per-packet timestamps, Ogg Opus does not allow gaps in the audio. Instead, an equal duration of silence or predicted samples may be substituted for the missing samples when encoding. If pre-encoded Opus is being copied to Ogg without encoding, any missing Opus packets may be replaced with Opus PLC (packet loss concealment) packets as described in ​section 4.1 of the specification.

$ ffmpeg -v 9 -loglevel 99 -i lostpacket.m4a out.opus
ffmpeg version N-60696-g38a08e0 Copyright (c) 2000-2014 the FFmpeg developers
  built on Feb 17 2014 01:24:12 with gcc 4.8.2 (MacPots gcc48 4.8.2_0)
  configuration: --enable-debug --assert-level=2 --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libspeex --enable-libass --enable-libbluray --enable-gnutls --enable-fontconfig --enable-libfreetype --mandir=/opt/local/share/man --enable-pthreads --cc=/opt/local/bin/gcc-mp-4.8 --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-nonfree --enable-libfdk-aac --enable-libfaac
  libavutil      52. 64.100 / 52. 64.100
  libavcodec     55. 52.102 / 55. 52.102
  libavformat    55. 33.100 / 55. 33.100
  libavdevice    55. 10.100 / 55. 10.100
  libavfilter     4.  1.102 /  4.  1.102
  libavresample   1.  1.  0 /  1.  1.  0
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'lostpacket.m4a'.
Reading option 'out.opus' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input file lostpacket.m4a.
Successfully parsed a group of options.
Opening an input file: lostpacket.m4a.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x103009800] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x103009800] ISO: File Type Major Brand: M4A 
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x103009800] Before avformat_find_stream_info() pos: 28861 bytes read:28861 seeks:0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x103009800] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x103009800] After avformat_find_stream_info() pos: 48 bytes read:28861 seeks:0 frames:1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'lostpacket.m4a':
  Metadata:
    major_brand     : M4A 
    minor_version   : 512
    compatible_brands: isomiso2
    encoder         : Lavf55.32.101
  Duration: 00:00:03.54, start: 0.042667, bitrate: 65 kb/s
    Stream #0:0(und), 1, 1/48000: Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 61 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Successfully opened the file.
Parsing a group of options: output file out.opus.
Successfully parsed a group of options.
Opening an output file: out.opus.
Successfully opened the file.
detected 4 logical cores
[graph 0 input from stream 0:0 @ 0x102a12fc0] Setting 'time_base' to value '1/48000'
[graph 0 input from stream 0:0 @ 0x102a12fc0] Setting 'sample_rate' to value '48000'
[graph 0 input from stream 0:0 @ 0x102a12fc0] Setting 'sample_fmt' to value 'fltp'
[graph 0 input from stream 0:0 @ 0x102a12fc0] Setting 'channel_layout' to value '0x3'
[graph 0 input from stream 0:0 @ 0x102a12fc0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
[audio format for output stream 0:0 @ 0x102a138a0] Setting 'sample_fmts' to value 's16|flt'
[audio format for output stream 0:0 @ 0x102a138a0] Setting 'sample_rates' to value '48000|24000|16000|12000|8000'
[audio format for output stream 0:0 @ 0x102a138a0] Setting 'channel_layouts' to value '0x4|0x3|0x7|0x33|0x37|0x3f|0x70f|0x63f'
[audio format for output stream 0:0 @ 0x102a138a0] auto-inserting filter 'auto-inserted resampler 0' between the filter 'Parsed_anull_0' and the filter 'audio format for output stream 0:0'
[AVFilterGraph @ 0x102a11be0] query_formats: 4 queried, 6 merged, 3 already done, 0 delayed
[auto-inserted resampler 0 @ 0x102a14080] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:flt r:48000Hz
[libopus @ 0x10300f800] No bit rate set. Defaulting to 96000 bps.
Output #0, opus, to 'out.opus':
  Metadata:
    major_brand     : M4A 
    minor_version   : 512
    compatible_brands: isomiso2
    encoder         : Lavf55.33.100
    Stream #0:0(und), 0, 1/48000: Audio: opus (libopus), 48000 Hz, stereo, flt, 96 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      major_brand     : M4A 
      minor_version   : 512
      compatible_brands: isomiso2
      encoder         : Lavf55.33.100
Stream mapping:
  Stream #0:0 -> #0:0 (aac -> libopus)
Press [q] to stop, [?] for help
[libopus @ 0x10300f800] Trying to remove 584 more samples than there are in the queue
[output stream 0:0 @ 0x102a13300] EOF on sink link output stream 0:0:default.
No more output streams to write to, finishing.
size=      49kB time=00:00:03.57 bitrate= 112.8kbits/s    
video:0kB audio:49kB subtitle:0 data:0 global headers:0kB muxing overhead 1.290400%
166 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0x102a12c80] Statistics: 0 seeks, 6 writeouts
[AVIOContext @ 0x102a11dc0] Statistics: 28861 bytes read, 0 seeks
$ 

The incorrect Ogg Opus file produces warnings from the opusinfo tool (part of ​opus-tools); "Sample count behind granule":

$ opusinfo out.opus
Processing file "out.opus"...

New logical stream (#1, serial: f039efde): type opus
Encoded with Lavf55.33.100
User comments section follows...
	language=und
	handler_name=SoundHandler
	major_brand=M4A 
	minor_version=512
	compatible_brands=isomiso2
	encoder=Lavf55.33.100
WARNING: Sample count behind granule (95040>96064) in stream 1
WARNING: Sample count behind granule (143040>144064) in stream 1
WARNING: Sample count behind granule (170880>171320) in stream 1
Opus stream 1:
	Pre-skip: 312
	Playback gain: 0 dB
	Channels: 2
	Original sample rate: 48000Hz
	Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
	Page duration:   1000.0ms (max),  890.0ms (avg),  580.0ms (min)
	Total data length: 50394 bytes (overhead: 1.31%)
	Playback length: 0m:03.562s
	Average bitrate: 113.2 kb/s, w/o overhead: 111.7 kb/s
Logical stream 1 ended
$ 

It is expected that FFmpeg would produce a valid Ogg Opus file, in this case by inserting silence or predicted samples in place of the missing samples before encoding.

[MarkZV]

mp4 audio with a 1-packet gap

[Carl Eugen Hoyos]

Is this only reproducible with opus output or also with the native vorbis encoder?

[MarkZV]

Using FFmpeg to produce Ogg Vorbis from this input file, using either the FFmpeg native vorbis encoder or the libvorbis encoder, has the same issue. I'm not aware of a utility that checks the granule position consistency for Ogg Vorbis files but I just checked it manually. Also different utilities displayed different timestamps for the packets in the resulting file depending on whether it computed them going forward or backward. The timestamps should be the same either way.