AAC Scalable Sample Rate (SSR)

[Carl Eugen Hoyos]

I will attach a 60 second aac sample uploaded by a user. libavcodec claims five times "SSR not implemented" during decode, output sounds fine.

$ ffmpeg -i ssr.aac -f null -
ffmpeg version N-43938-gbe862c0 Copyright (c) 2000-2012 the FFmpeg developers
  built on Aug 27 2012 19:54:31 with gcc 4.5.3 (GCC)
  configuration: --cc=/usr/local/gcc-4.5.3/bin/gcc
  libavutil      51. 70.100 / 51. 70.100
  libavcodec     54. 54.100 / 54. 54.100
  libavformat    54. 25.104 / 54. 25.104
  libavdevice    54.  2.100 / 54.  2.100
  libavfilter     3. 13.101 /  3. 13.101
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 15.100 /  0. 15.100
[aac @ 0x1480240] max_analyze_duration 5000000 reached at 5013333
[aac @ 0x1480240] Estimating duration from bitrate, this may be inaccurate
Input #0, aac, from 'ssr.aac':
  Duration: 00:01:09.48, bitrate: 112 kb/s
    Stream #0:0: Audio: aac, 48000 Hz, stereo, s16, 112 kb/s
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf54.25.104
    Stream #0:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (aac -> pcm_s16le)
Press [q] to stop, [?] for help
[aac @ 0x14865a0] SSR not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x14865a0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
Error while decoding stream #0:0: Operation not permitted
[aac @ 0x14865a0] SSR not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x14865a0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
Error while decoding stream #0:0: Operation not permitted
[aac @ 0x14865a0] SSR not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x14865a0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
Error while decoding stream #0:0: Operation not permitted
[aac @ 0x14865a0] SSR not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x14865a0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
Error while decoding stream #0:0: Operation not permitted
[aac @ 0x14865a0] SSR not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x14865a0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/MPlayer/incoming/ and contact the ffmpeg-devel mailing list.
Error while decoding stream #0:0: Operation not permitted
size=       0kB time=00:01:00.13 bitrate=   0.0kbits/s
video:0kB audio:11256kB subtitle:0 global headers:0kB muxing overhead -100.000000%

[Carl Eugen Hoyos]

An additional sample is in ​http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket1693/

[Aman]

According to ​​https://lists.libav.org/pipermail/ffmpeg-devel/2008-July/049282.html, there was an implementation of SSR that never made it to master:

The code for LTP and SSR is still in the Summer of Code repository for anyone who is interested in working on either of those features in the future.

And according to ​​https://en.wikipedia.org/wiki/MPEG-4_Part_3#AAC-SSR:

MPEG-4 AAC-SSR is very similar to ATRAC and ATRAC-3.

So the existing atrac and atrac3 decoders might be useful as well.

The SoC repo mentioned is available via:

svn co ​​svn://svn.mplayerhq.hu/soc
cd aac

The SSR implementation is behind a -DAAC_SSR compile flag, and disabled by default with the following comment:

AAC SSR (Scalable Sample Rate) is currently not working, and therefore not compiled in. SSR files play without crashing but produce audible artifacts that seem to be related to EIGHT_SHORT_SEQUENCE windows.

It seems mostly complete, though it will definitely take some work to port the changes over to master.

[Carl Eugen Hoyos]

I suspect none of the samples here are actually SSR, samples are in ​http://standards.iso.org/ittf/PubliclyAvailableStandards/ISO_IEC_14496-4_2004_Conformance_Testing/audio_conformance/mpeg4audio-conformance/compressedMp4/ and show Audio object type 3 is not implemented as error message.

[Aman]

Here's another sample: ​https://s3.amazonaws.com/tmm1/VasHD.mpg

[Carl Eugen Hoyos]

Why do you think that the sample from ticket #4544 contains SSR audio?
Even more so as apparently no real-world sample exist.

[Carl Eugen Hoyos]

Even more so as you have already confirmed a year ago that the sample does not contain SSR audio...

[Aman]

Correct, the sample from #4544 does not contain SSR. I thought it was the same as this issue but I was mistaken. That bug has since been resolved.

The VasHD.mpg sample I provided above was captured in Singapore, where most of the OTA channels have aac tracks that ffmpeg reports as being SSR.

[Carl Eugen Hoyos]

Replying to tmm1:

The VasHD.mpg sample I provided above was captured in Singapore, where most of the OTA channels have aac tracks that ffmpeg reports as being SSR.

Command line and complete, uncut console output missing.

[Dale Curtis]

There's more samples and discussion here too, ​https://bugs.chromium.org/p/chromium/issues/detail?id=808064

This became in issue in Chrome because we stopped silently dropping these packets and instead emit an error now. Not decoding the SSR packet seems to lead to an accumulation of ~1 seconds of audio loss every 10 seconds.

Given the volume of complaints quite a few folks have been silently losing their audio over the years. The workaround for now is to re-encode with libfdk_aac which can parse the SSR packets:

./ffmpeg -acodec libfdk_aac -i <file> -acodec libfdk_aac -vcodec copy <out file>

Here's a recent log from a sample I have:

$ ffmpeg -i ssr_chrome.aac out.wav
ffmpeg version N-89956-gcaa4bd7a9f Copyright (c) 2000-2018 the FFmpeg developers

built with clang version 6.0.0 (trunk 321529)
configuration: --enable-shared --enable-nonfree --enable-gpl --cc=clang --ld=clang --enable-libfdk-aac
libavutil 56. 7.100 / 56. 7.100
libavcodec 58. 9.100 / 58. 9.100
libavformat 58. 7.100 / 58. 7.100
libavdevice 58. 0.101 / 58. 0.101
libavfilter 7. 11.101 / 7. 11.101
libswscale 5. 0.101 / 5. 0.101
libswresample 3. 0.101 / 3. 0.101
libpostproc 55. 0.100 / 55. 0.100

[aac @ 0x24ad480] Estimating duration from bitrate, this may be inaccurate
Input #0, aac, from 'ssr_chrome.aac':

Duration: 00:00:19.50, bitrate: 134 kb/s

Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 134 kb/s

File 'out.wav' already exists. Overwrite ? [y/N] y
Stream mapping:

Stream #0:0 -> #0:0 (aac (native) -> pcm_s16le (native))

Press [q] to stop, ? for help
Output #0, wav, to 'out.wav':

Metadata:

ISFT : Lavf58.7.100
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
Metadata:

encoder : Lavc58.9.100 pcm_s16le

[aac @ 0x24d0b00] SSR is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x24d0b00] If you want to help, upload a sample of this file to ​ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (​ffmpeg-devel@ffmpeg.org)
Error while decoding stream #0:0: Not yet implemented in FFmpeg, patches welcome
[aac @ 0x24d0b00] SSR is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[aac @ 0x24d0b00] If you want to help, upload a sample of this file to ​ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (​ffmpeg-devel@ffmpeg.org)

[Dale Curtis]

SSR AAC file.

[Carl Eugen Hoyos]

Replying to dalecurtis:

This became an issue in Chrome because we stopped silently dropping these packets and instead started emitting an error. Skipping the SSR packet (like we did in M63 and below) leads to an accumulation of ~1 seconds of audio loss every 10 seconds.

If I understand correctly, you are saying that the sample you uploaded that plays for approximately 19.5 seconds with current FFmpeg, is supposed to play longer.

Given the volume of complaints quite a few folks have been silently losing their audio over the years. The workaround for now is to re-encode with libfdk_aac which can parse the SSR packets:

./ffmpeg -acodec libfdk_aac -i <file> -acodec libfdk_aac -vcodec copy <out file>

I tested the following command line:
$ ffmpeg -acodec libfdk_aac -i ssr_chrome.aac -acodec libfdk_aac out.aac
The output file plays for approximately 17 seconds: What did I misunderstand?

[Dale Curtis]

Yes, it's 20.08 seconds of audio. The duration information isn't accurate when you write to .aac container, if you use out.mp4 instead you'll see the true accurate duration:

$ ffmpeg -i out.mp4
ffmpeg version N-89956-gcaa4bd7a9f Copyright (c) 2000-2018 the FFmpeg developers

built with clang version 6.0.0 (trunk 321529)
configuration: --enable-shared --enable-nonfree --enable-gpl --cc=clang --ld=clang --enable-libfdk-aac
libavutil 56. 7.100 / 56. 7.100
libavcodec 58. 9.100 / 58. 9.100
libavformat 58. 7.100 / 58. 7.100
libavdevice 58. 0.101 / 58. 0.101
libavfilter 7. 11.101 / 7. 11.101
libswscale 5. 0.101 / 5. 0.101
libswresample 3. 0.101 / 3. 0.101
libpostproc 55. 0.100 / 55. 0.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'out.mp4':

Metadata:

major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf58.7.100

Duration: 00:00:20.08, start: 0.000000, bitrate: 132 kb/s

Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 130 kb/s (default)
Metadata:

handler_name : SoundHandler

At least one output file must be specified

[Carl Eugen Hoyos]

Replying to dalecurtis:

Yes, it's 20.08 seconds of audio.

So how can I get 20 second output from the file you uploaded?

[Dale Curtis]

Ah, sorry I thought it was clear when I said use mp4. Here's the command line:

ffmpeg -acodec libfdk_aac -i ssr_chrome.aac -acodec libfdk_aac out.mp4

[Carl Eugen Hoyos]

Replying to dalecurtis:

ffmpeg -acodec libfdk_aac -i ssr_chrome.aac -acodec libfdk_aac out.mp4

This produces a stream here that plays for approximately 19.5 seconds, just as the original file you attached with the native decoder.

I also wonder why you believe that libfdk works so much better for this sample if re-encoding to aac with libfdk produces a shorter file.

[Dale Curtis]

How are you getting that 19.5 number? I'm not able to generate a shorter clip with libfdk. The SSR packets have duration, so by dropping them the ffmpeg decoded copy will by necessity be shorter.

[Hendrik]

The fdk decoder also emits a lot of errors on this clip and decodes with audible distortions (albeit different ones then the native decoder), so either the specified clip is broken or its an extremely unusual encoding.

[Dale Curtis]

The fdk errors are just gain control ones and don't result in duration loss AFAICT:

[libfdk_aac @ 0xa66940] aacDecoder_DecodeFrame() failed: 400a
Error while decoding stream #0:0: Unknown error occurred

400a is AAC_DEC_UNSUPPORTED_GAIN_CONTROL_DATA = 0x400A, /*!< Gain control data found but not supported. Most probably the bitstream is corrupt, or has a wrong format. */

[Dale Curtis]

Actually now that I mention that they may be causing duration loss too. Whether or not these are also invalid I'm not sure.

[Hendrik]

"just" Gain Control results in a worse audible result then the native decoder, fwiw.

The way I understand SSR, you can actually decode it with a LC decoder, you just get a worse result (ie. only the base layer). So perhaps the decoder can be tweaked to just ignore the SSR data (or warn about it) instead of discarding the entire audio frame.

[Dale Curtis]

I don't disagree :) It seems neither is capable of fully decoding the stream, so now I'm not sure what the duration of this clip should be. The original is available at ​http://demo.stockweight.com/mmm/dir/ok.m3u8 which seems to imply it should be ~20 seconds (matching video length), but even Safari cuts out at ~18 seconds; so it's possibly just bad encoding.

[peloverde]

just parsing the raw SSR gain control data is pretty trivial. It's just a handful of fixed length fields in a variable loop.

[Dale Curtis]

Thanks for the info peloverde. I'll look at implementing that.

[Dale Curtis]

Parsing gain control was indeed trivial. With the attached patch (assembled from the summer of code repo) the SSR samples in comment:3 (as*.mp4) decode without error.

However, it's not sufficient to decode the ssr_chrome.aac file from the m3u8 in comment:21. That file now causes "[aac @ 0x32a5df854f80] Input buffer exhausted before END element found" to be emitted from here:

​https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/aacdec_template.c#L3223

get_bits_left(gb) == -8, in this case after decode_spectrum_and_dequant().

​https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/aacdec_template.c#L1633

Possibly this is because the file is actually damaged and incorrect.

[Dale Curtis]

Decodes and drops gain control data.

[Carl Eugen Hoyos]

Thank you for looking into this!
I believe your patch is missing a copyright notice regarding the original author...

[Carl Eugen Hoyos]

Replying to dalecurtis:

Possibly this is because the file is actually damaged and incorrect.

I am not an audiophile but this seemed obvious to me...

Please change the commit message or if you prefer add some information that the patch is not primarily about SSR but about incorrect warnings shown that indicated SSR for broken AAC (non-SSR) samples.

The patch is very helpful, more reports like this one exist, thank you!

[Hendrik]

Technically, if you use the Gain Control coding, the profile should be SSR as well. At least according to the spec, but specifically for AAC there are a bunch of things that the spec seems to disagree with.

[Dale Curtis]

@cehoyos: Thanks for calling that out. I'll fix the attribution on subsequent patches, I just uploaded that one in case others here had ideas on why the file was still failing. I'm not sure it should be applied upstream without at least printing a warning.

I just received another non-SSR sample which is failing in the same way, so it may be that there's a secondary bug lurking in decode_spectrum_and_dequant() - I'll continue to dig some more today.
​https://bugs.chromium.org/p/chromium/issues/detail?id=808064#c29

[Dale Curtis]

Actually, that linked file seems to have something totally different going on, so not really relevant to this issue I think.

[Dale Curtis]

On closer examination one SSR conformance test fails with the SSR drop patch,
​http://standards.iso.org/ittf/PubliclyAvailableStandards/ISO_IEC_14496-4_2004_Conformance_Testing/audio_conformance/mpeg4audio-conformance/compressedMp4/as08_32.mp4

[aac @ 0x1a8fac0] Input buffer exhausted before END element found

as08_32 == core setup 8, sampling frequency 32

It's failing for similar reasons to all my other SSR test cases, decode_spectrum_and_dequant() runs out of bits, but no idea if the root cause is the same.

[Dale Curtis]

That test case has a gain control bit, but no gain control data and all spectra are encoded with escape sequences, so a bug here seems possible:
​https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/aacdec_template.c#L1803

The other failing samples I have don't use escape sequences.

[Dale Curtis]

Okay, the as08_32.mp4 file is corrupt. It's only 50kb but has sample indices into the 90kb+ range, so the ssr drop patch I attached works fine.

Also, the ssr_chrome.aac file, plus a couple more samples I received, were created by libfaac 1.26.1, which has never had support for writing gain control data: ​https://sourceforge.net/p/faac/faac/ci/648c55c5a0a049e557ac6d40ffe56ef49d4682a7/tree/libfaac/bitstream.c#l716 -- so that file is simply corrupt. The encoding string can be found in a hexdump of the file:

00000000 ff f1 4c 80 07 5f fc de 36 00 00 6c 69 62 66 61 |..L.._..6..libfa|
00000010 61 63 20 31 2e 32 36 2e 31 20 28 4e 6f 76 20 31 |ac 1.26.1 (Nov 1|
00000020 34 20 32 30 31 30 29 20 55 4e 53 54 41 42 4c 45 |4 2010) UNSTABLE|

As such I've attached the same ssr drop patch again with attribution to Robert Swain, who seems to be the author of the original patch set per the ffmpeg-devel e-mails. I'll send this to ffmpeg-devel and we can continue discussion there as to whether this should be submitted.

[Carl Eugen Hoyos]

Replying to dalecurtis:

As such I've attached the same ssr drop patch again with attribution to Robert Swain, who seems to be the author of the original patch set per the ffmpeg-devel e-mails.

Did you find the e-mails? Can you point me to them?

[Dale Curtis]

Hmm, I found this one:
https://ffmpeg.org/pipermail/ffmpeg-devel/2008-June/049495.html

But now I see this goes all the way back to:
https://ffmpeg.org/pipermail/ffmpeg-devel/2008-March/052231.html

And just says it's from the SoC repository. svn log/blame for that seems to imply the author is maxim@, so maybe it's actually Maxim Gavrilov?
​https://www.ffmpeg.org/doxygen/3.0/aactab_8h_source.html

[Dale Curtis]

Landed as a246701e9abe8ef7cb9b0dd9fb5fa877e78334ef; this also has the nice consequence of causing things that don't actually have gain control to print the more useful:

[aac @ 0x1a8fac0] Input buffer exhausted before END element found

So missing SSR is not incorrectly implicated anymore.

[Balling]

Okay, the as08_32.mp4 file is corrupt. It's only 50kb but has sample indices into the 90kb+ range, so the ssr drop patch I attached works fine.

It is not corrupt per se, it is just stream 0, offset 0xc629: partial file

since dts = 188416.