Front and end padding not trimmed from m4a (dash) causing gapless playback to fail

[Patrick Kan]

When using FFmpeg to concat the m4a samples from ​https://www2.iis.fraunhofer.de/AAC/gapless.html, the output is one that plays seamlessly. These samples are of ftyp->major_brand 'mp42' (according to ffprobe).

On the other hand, m4a streams from YouTube / YouTube Music are of ftyp->major brand 'dash'. When concatenating streams downloaded from YT, the samples corresponding to front and end padding are not discarded. The output is therefore one that contains gaps at points where streams meet.

On YouTube Music in particular, provided you have a Premium subscription, you can enjoy gapless playback of performances spanning multiple tracks. An example is Pink Floyd's The Wall album.

My limited understanding is that for MP4 containers, the edts/elst boxes provide information about the front padding (is it called encoder delay?) as well as the track duration, which can be used to determine the end padding. I have verified that these are contained in the YT streams, but FFmpeg doesn't seem to use them as it does with "regular" m4a files.

My use case is to actually pass the URLs of the YT / YT Music streams to MusicPlayerDaemon, which will then utilize FFmpeg to decode them. But I suppose as long as FFmpeg deals with edts/elst for these streams, and remove the front and end padding during decoding, then it shouldn't matter what player I use in the end.

The following illustrates the steps I took in diagnosing the issue (emphasis: my knowledge is limited):

1. Download streams with youtube-dl for closer inspection:

// Album: Pink Floyd - The Wall
// Track 2
# youtube-dl --fixup never -v -f m4a "https://music.youtube.com/watch?v=Ciai1aZ_odg"

// Track 3
# youtube-dl --fixup never -v -f m4a "https://music.youtube.com/watch?v=-cfJqYtmmqA"

// Above commands create the following files for track 2 and 3 respectively:
// 'The Thin Ice-Ciai1aZ_odg.m4a'
// 'Another Brick In The Wall (Part 1)--cfJqYtmmqA.m4a'

2. Verify that edts/elst exist in stream:

# mediainfo --DETAILS=1 The The\ Thin\ Ice-Ciai1aZ_odg.m4a

(Output in attached mediainfo.txt)

3. Concat the two m4a files created in step 1:

# ffmpeg -v 9 -loglevel 99 -f concat -safe 0 -report -i concat.txt concat.m4a

// concat.txt:
file 'The Thin Ice-Ciai1aZ_odg.m4a'
file 'Another Brick In The Wall (Part 1)--cfJqYtmmqA.m4a'

(Output in attached ffmpeg-20230516-212923.log)

The logs do not mention FFmpeg skipping or discarding any samples. This is different from concatenating "regular" aac files, which I did as follows for testing purpose:

# wget https://www2.iis.fraunhofer.de/AAC/gapless-sweep_part1_iis.m4a
# wget https://www2.iis.fraunhofer.de/AAC/gapless-sweep_part2_iis.m4a
# ffmpeg -v 9 -loglevel 99 -f concat -safe 0 -report -i concat-regular.txt concat-regular.m4a

// concat-regular.txt
file 'gapless-sweep_part1_iis.m4a'
file 'gapless-sweep_part2_iis.m4a'

(Output in attached ffmpeg-20230517-004305.log)

From logs, FFmpeg did skip some samples due to side data (which I presume refers to edts / elst?):

# cat ffmpeg-20230517-004305.log | grep skip
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558bdd922d00] skip 576 audio samples from curr_cts: 1024
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558bdd922d00] demuxer injecting skip 1600 / discard 0
[aac @ 0x558bdd923e00] skip 1600 / discard 0 samples due to side data
[aac @ 0x558bdd93a000] skip 1600 / discard 0 samples due to side data
[aac @ 0x558bdd929e80] skip 1600 / discard 0 samples due to side data
[aac @ 0x558bdd929e80] skip 576/1024 samples
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558bdd9242c0] skip 576 audio samples from curr_cts: 1024
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558bdd9242c0] demuxer injecting skip 1600 / discard 0
[aac @ 0x558bddb6d880] skip 1600 / discard 0 samples due to side data
[aac @ 0x558bdd929e80] skip 1600 / discard 0 samples due to side data
[aac @ 0x558bdd929e80] skip 576/1024 samples

(Opus streams on the other hand work fine - I read that they are inherently gapless, but YT Music provides "high-quality" streams, available with Premium subscriptions, in m4a only)

[Patrick Kan]

mediainfo output of downloaded stream

[Patrick Kan]

FFmpeg concat log of YT (aac - dash) streams (1)

[Patrick Kan]

FFmpeg concat log of YT (aac - dash) streams (2)

[Patrick Kan]

FFmpeg concat log of YT (aac - dash) streams (3)

[Patrick Kan]

Conat log of "regular" aac files