Advanced Edit List support causes decode errors

[dbuitenh]

Summary of the bug:

The "advanced edit list" code in libavformat's MOV/MP4 demuxer causes decode errors on some files. I think this is related to the fact that these files are frames stored out of order with negative entries in the CTTS atom, and the edit list reodering/frame dropping/GOP code doesn't handle this properly, since it was written with only MP4 in mind, which only has unsigned value in the CTTS box (MOV can have signed values). As a result, some edit list entries cause decode errors and artifacting (see below). Disabling the advanced edit list code (aka not hanling it at all) with -advanced_editlist 0 or -ignored_editlist 1 prevents these decode errors.

Also of note is that this file as a CSLG atom, which FFmpeg seems to not parse at all, which could also be related.

How to reproduce:

% ffmpeg -i test2.mov -f null -
ffmpeg version N-99165-gfda5363c80 Copyright (c) 2000-2020 the FFmpeg developers
  built with clang version 9.0.1-14
  configuration: --cc=clang
  libavutil      56. 59.100 / 56. 59.100
  libavcodec     58.106.100 / 58.106.100
  libavformat    58. 56.100 / 58. 56.100
  libavdevice    58. 11.102 / 58. 11.102
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test2.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2020-08-13T20:58:58.000000Z
    com.apple.finalcutstudio.media.uuid: 41E16D85-BEBA-4ADE-8AB4-A0E359562D54
  Duration: 00:59:36.72, start: 0.000000, bitrate: 5 kb/s
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080, 1003 kb/s, 25 fps, 25 tbr, 25 tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2020-08-13T20:58:58.000000Z
      handler_name    : Apple Video Media Handler
      encoder         : AVC Coding
      timecode        : 00:00:00:00
    Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2020-08-13T20:58:58.000000Z
      handler_name    : Apple Sound Media Handler
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74) (default)
    Metadata:
      creation_time   : 2020-08-13T20:59:27.000000Z
      handler_name    : Time Code Media Handler
      timecode        : 00:00:00:00
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (pcm_s16le (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    com.apple.finalcutstudio.media.uuid: 41E16D85-BEBA-4ADE-8AB4-A0E359562D54
    encoder         : Lavf58.56.100
    Stream #0:0(eng): Video: wrapped_avframe, yuv420p, 1920x1080, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2020-08-13T20:58:58.000000Z
      handler_name    : Apple Video Media Handler
      timecode        : 00:00:00:00
      encoder         : Lavc58.106.100 wrapped_avframe
    Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2020-08-13T20:58:58.000000Z
      handler_name    : Apple Sound Media Handler
      encoder         : Lavc58.106.100 pcm_s16le
[h264 @ 0x2d7ad80] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x2d7ad80] co located POCs unavailable
[h264 @ 0x2d7ad80] reference count overflow
[h264 @ 0x2d7ad80] decode_slice_header error

[errors continue for a long time from here]

I have attached a trimmed 2.5mb version of one of these files that should be sufficient for testing and reproducing, but I also have the creator's permission to share the whole ~1gb file if necessary or prudent.

[Balling]

Well, support for cslg atom was nuked in e74527219668b720aefb7ecb79a44a8137aafa60 (it #419 is not about THIS bug tracker, it uses old one on the mailing list).

Also see ​https://github.com/gpac/mp4box.js/issues/226