FFmpeg treats all audio samples as sync samples for fragmented MP4 files

[berndcz]

Summary of the bug:
It seems that currently all audio samples (AVMEDIA_TYPE_AUDIO) belonging to a fragmented MP4 file are treated as sync samples (keyframe = 1, mov.c:4937).
Since there are audio formats where samples depend on other samples, e.g. xHE-AAC , the sample metadata should not be overwritten.

How to reproduce:
Attached hereto, you can find a fragmented MP4 file containing one xHE-AAC audio track. For each fragment, only the first file format sample is signalled as sync sample. However, when seeking into the file, FFmpeg is forwarding non-sync samples to the decoder (please have a look at the libfdk_aac log message)..

% ./ffmpeg -acodec libfdk_aac -ss 10 -i ./fragmented-5s_frag_duration-2ch-xheaac-64kbps.mp4 skip_start.wav
ffmpeg version N-97830-gb707abf091 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --enable-libfdk-aac --disable-encoder=aac --disable-decoder=aac --disable-decoder=aac_latm
  libavutil      56. 46.100 / 56. 46.100
  libavcodec     58. 84.100 / 58. 84.100
  libavformat    58. 43.100 / 58. 43.100
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 81.100 /  7. 81.100
  libswscale      5.  6.101 /  5.  6.101
  libswresample   3.  6.100 /  3.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from './fragmented-5s_frag_duration-2ch-xheaac-64kbps.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42isomdash
    creation_time   : 2020-04-15T12:36:42.000000Z
  Duration: 00:04:00.00, start: 0.000000, bitrate: 66 kb/s
    Stream #0:0(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, s16, 1 kb/s (default)
    Metadata:
      creation_time   : 2020-04-15T12:36:42.000000Z
      handler_name    : soun
Stream mapping:
  Stream #0:0 -> #0:0 (aac (libfdk_aac) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[libfdk_aac @ 0x55e234844c80] aacDecoder_DecodeFrame() failed: 5
Error while decoding stream #0:0: Unknown error occurred
[libfdk_aac @ 0x55e234844c80] aacDecoder_DecodeFrame() failed: 5
Error while decoding stream #0:0: Unknown error occurred
[libfdk_aac @ 0x55e234844c80] aacDecoder_DecodeFrame() failed: 5
Error while decoding stream #0:0: Unknown error occurred
[libfdk_aac @ 0x55e234844c80] aacDecoder_DecodeFrame() failed: 5
Error while decoding stream #0:0: Unknown error occurred
Output #0, wav, to 'skip_start.wav':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42isomdash
    ISFT            : Lavf58.43.100
    Stream #0:0(und): Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2020-04-15T12:36:42.000000Z
      handler_name    : soun
      encoder         : Lavc58.84.100 pcm_s16le
size=   43096kB time=00:03:49.99 bitrate=1535.0kbits/s speed= 108x    
video:0kB audio:43096kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000177%

[Balling]

That is all cool and all, but libfdk_aac only recently added support to xHE-AAC and aac native decoder is still used to probe the stream.