Muxing libvorbis audio fails; no sound in the output file

[gyunaev]

There seem to be an issue related to muxing vorbis audio in ffmpeg, both 2.1.1 release and GIT head. The issue is fairly easy to reproduce with the muxing.c example in the FFmpeg doc/example directory by issuing the following command:

./muxing test.webm

The resulting test.webm will play video fine with ffplay, and will have the audio track, but will output no sound. It does not generate any errors either.

This issue is related to vorbis and not libvpx/webm container, which could be further proved by adding the following two lines to force muxing test.ogg into theora/vorbis:

--- muxing.c.orig       2013-12-18 01:25:24.126903125 -0800
+++ muxing.c    2013-12-18 01:29:12.643882209 -0800
@@ -487,6 +487,7 @@
     video_st = NULL;
     audio_st = NULL;
 
+       fmt->audio_codec = AV_CODEC_ID_VORBIS;
     if (fmt->video_codec != AV_CODEC_ID_NONE) {
         video_st = add_stream(oc, &video_codec, fmt->video_codec);
     }

(this patch is necessary because by default it attempts to mux test.ogg into theora/flac - which by the way fails right away). And running make and executing as following:

./muxing test.ogg

The resulted test.ogg will work exactly as test.webm above - ffplay will play only video and no sound.

Further investigating this issue it looks like vorbis audio requires setting the frame->pts. Applying the following patch to muxing.c fixes the audio output:

--- muxing.c.orig       2013-12-18 01:25:24.126903125 -0800
+++ muxing.c    2013-12-18 01:36:03.566844598 -0800
@@ -258,6 +258,9 @@
         dst_nb_samples = src_nb_samples;
     }
 
+    static int audio_pts = 0;
+    frame->pts = audio_pts;
+    audio_pts += dst_nb_samples;
     frame->nb_samples = dst_nb_samples;
     avcodec_fill_audio_frame(frame, c->channels, c->sample_fmt,
                              dst_samples_data[0], dst_samples_size, 0);

With this patch theora/vorbis audio mixes fine, and ./muxing test.ogg plays the sound. However this patch breaks muxing every other format - for example, .patched /muxing test.avi uses mpeg4/vorbis and loses sound after 4 seconds. It is also seem to be exclusively vorbis-specific. libtheora+flac in ogg muxes without problem, while mpeg4+vorbis in AVI experiences the audio loss. More important, it doesn't fix webm libvpx/libvorbis, so it may be a different issue.

[gyunaev]

Also my libvorbis version is 1.3.3 and my system is OpenSuSE Linux 13.1 on x86_64

[gyunaev]

Could you please provide more feedback? If this is purely the documentation issue, why the FFmpeg example does not work?

[Carl Eugen Hoyos]

muxing.c is part of the FFmpeg documentation, but I wonder if it is ok to first change the source code and then complain that it does not work (I didn't test myself). I leave it to the documentation maintainers to decide.

[gyunaev]

Thank you very much for following this up. You are correct, but please keep in mind that actual problem reported is that muxing WEBM (libvpx + libvorbis) does not work at all with unmodified muxing.c. As of right now I'm not sure whether the problem is in muxing.c or in libavcodec/libavformat since after spending two days of it I was never able to actually mux vpx and vorbis in the container which would play audio. No matter what I do it results exactly as what happens if you run unmodified ./muxing test.webm

The modifications below are there to describe why I believe the problem is in vorbis codec, and not in Theora coder or Webm container. They're not to be applied to the released muxing.c, otherwise I'd posted them into -devel.

[Carl Eugen Hoyos]

Replying to gyunaev:

Thank you very much for following this up. You are correct, but please keep in mind that actual problem reported is that muxing WEBM (libvpx + libvorbis) does not work at all with unmodified muxing.c.

So this is not a bug report but a documentation enhancement request?

As of right now I'm not sure whether the problem is in muxing.c or in libavcodec/libavformat since after spending two days of it I was never able to actually mux vpx and vorbis in the container which would play audio.

Does it work with ffmpeg?

[gyunaev]

Yes, either documentation or source code enhancement request. It is basically "please make it working in the default scenario" request.

I have no means to test it with ffmpeg since it operates differently from muxing.c - it uses input file(s) while muxing.c generates the data. So yes, ffmpeg would have the same behavior if it uses the same code as muxing.c.

[Carl Eugen Hoyos]

Replying to gyunaev:

I have no means to test it with ffmpeg since it operates differently from muxing.c - it uses input file(s) while muxing.c generates the data. So yes, ffmpeg would have the same behavior if it uses the same code as muxing.c.

Could you elaborate?
Which code could ffmpeg use that muxing.c cannot use? And why can ffmpeg not process the same data as muxing.c?

[gyunaev]

As far as I know, FFmpeg cannot generate video/audio as muxing.c does. It works with the external input audio and video streams, and in converting those it uses the DTS/PTS information from those streams.

muxing.c however does not use any external streams, and generates the video and audio streams internally. Thus it cannot rely on provided PTS/DTS and has to supply it. muxing.c does it for video, but not for audio. This works fine with the majority of formats, but it does not work with WEBM or theora/vorbis.

This could mean that FFmpeg coders are ok and muxing.c needs fixing (because changing in as above fixes the output for theora/vorbis, although not for WebM). This could also mean there's a bug in relevant codec/format parser in FFmpeg, and the change above just avoids this bug in some way. This theory could be further supported by the fact that any other format except Vorbis audio muxes fine with the unmodified example, while any format containing Vorbis audio fail to mux properly.

This is important issue because the muxing example is the only one which is relevant for applications such as my ​Karaoke Lyrics Editor (which is a free GPL app) which generates an AV interleaved file based on the user entered data.

[Stefano Sabatini]

Fixed in:

commit 094c500caf0c2a67792f2ee8238f065926aebfc4
Author: Stefano Sabatini <stefasab@gmail.com>
Date:   Wed Jan 8 15:43:25 2014 +0100

    examples/muxing: set timestamps in output audio packet
    
    In particular, fix trac ticket #3231.