mp3 probe regression in 4.2.2

[Mattias Wadman]

Summary of the bug:

mp3 file probe without file extension hint work fine with 4.2.1 but fails with 4.2.2

How to reproduce:

Download mp3 file from https://anondino.squarespace.com/osc/2020/1/28/osc-53-super-smash-bros-brawl and pipe file to 4.2.1 and 4.2.2 to skip file extension hint. Increased formatprobesize fixes issue.

My guess is that commit https://github.com/FFmpeg/FFmpeg/commit/e9a335150a62bb377a26ce096187b4476145d02b#diff-c66b10cb5a4804b2196f81772f8cd68c caused the regression. Looking at the mp3 file it seems like the id3v2 tag is quite big (nearly 1MB) which causes the first mp3 frame to be partially read and fails the new frame size fits into probe buffer check.

Not sure what could be a sane fix for this. Keep increasing the default probe buffer size might not be feasible? allow partial mp3 frames again would fix only this narrow case i guess?

4.2.1:
$ cat OSC053.mp3 | ffmpeg -v trace -i pipe:0
ffprobe version 4.2.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 8.3.0 (Alpine 8.3.0)
  configuration: --pkg-config-flags=--static --extra-cflags=-fopenmp --extra-ldflags='-static -fopenmp' --toolchain=hardened --disable-debug --disable-shared --disable-ffplay --enable-static --enable-gpl --enable-nonfree --enable-openssl --enable-iconv --enable-libxml2 --enable-libmp3lame --enable-libfdk-aac --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvpx --enable-libx264 --enable-libx265 --enable-libwebp --enable-libwavpack --enable-libspeex --enable-libaom --enable-libvidstab --enable-libkvazaar --enable-libfreetype --enable-fontconfig --enable-libfribidi --enable-libass --enable-libzimg --enable-libsoxr --enable-libopenjpeg --enable-libdav1d
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[NULL @ 0x5555559993c0] Opening 'pipe:0' for reading
[pipe @ 0x555555999c80] Setting default whitelist 'crypto'
Probing mp3 score:12 size:2048
Probing mp3 score:12 size:4096
Probing mp3 score:12 size:8192
Probing mp3 score:12 size:16384
Probing mp3 score:12 size:32768
Probing mp3 score:12 size:65536
Probing mp3 score:12 size:131072
Probing mp3 score:12 size:262144
Probing mp3 score:12 size:524288
Probing mp3 score:1 size:82
[mp3 @ 0x5555559993c0] Format mp3 detected only with low score of 1, misdetection possible!
id3v2 ver:3 flags:00 len:1048484
[mp3 @ 0x5555559993c0] Skipping 0 bytes of junk at 1048494.
[mp3 @ 0x5555559993c0] Before avformat_find_stream_info() pos: 1048494 bytes read:1081344 seeks:0 nb_streams:2
[mjpeg @ 0x55555599c040] marker=d8 avail_size_in_buf=883980
[mjpeg @ 0x55555599c040] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x55555599c040] marker=e1 avail_size_in_buf=883978
[mjpeg @ 0x55555599c040] marker parser used 8597 bytes (68776 bits)
[mjpeg @ 0x55555599c040] marker=ed avail_size_in_buf=875378
[mjpeg @ 0x55555599c040] marker parser used 10641 bytes (85128 bits)
[mjpeg @ 0x55555599c040] marker=e1 avail_size_in_buf=864734
[mjpeg @ 0x55555599c040] marker parser used 4543 bytes (36344 bits)
[mjpeg @ 0x55555599c040] marker=e2 avail_size_in_buf=860188
[mjpeg @ 0x55555599c040] marker parser used 3160 bytes (25280 bits)
[mjpeg @ 0x55555599c040] marker=ee avail_size_in_buf=857026
[mjpeg @ 0x55555599c040] marker parser used 33 bytes (264 bits)
[mjpeg @ 0x55555599c040] marker=db avail_size_in_buf=856991
[mjpeg @ 0x55555599c040] index=0
[mjpeg @ 0x55555599c040] qscale[0]: 0
[mjpeg @ 0x55555599c040] index=1
[mjpeg @ 0x55555599c040] qscale[1]: 0
[mjpeg @ 0x55555599c040] marker parser used 132 bytes (1056 bits)
[mjpeg @ 0x55555599c040] marker=c2 avail_size_in_buf=856857
[mjpeg @ 0x55555599c040] Changing bps from 0 to 8
[mjpeg @ 0x55555599c040] sof0: picture: 3100x3100
[mjpeg @ 0x55555599c040] component 0 1:1 id: 0 quant:0
[mjpeg @ 0x55555599c040] component 1 1:1 id: 1 quant:1
[mjpeg @ 0x55555599c040] component 2 1:1 id: 2 quant:1
[mjpeg @ 0x55555599c040] pix fmt id 11111100
[mjpeg @ 0x55555599c040] Format yuvj444p chosen by get_format().
[mjpeg @ 0x55555599c040] marker parser used 17 bytes (136 bits)
[mjpeg @ 0x55555599c040] marker=c4 avail_size_in_buf=856838
[mjpeg @ 0x55555599c040] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x55555599c040] escaping removed 750224 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=856479
[mjpeg @ 0x55555599c040] marker parser used 106255 bytes (850040 bits)
[mjpeg @ 0x55555599c040] escaping removed 728692 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=750100
[mjpeg @ 0x55555599c040] marker parser used 21408 bytes (171264 bits)
[mjpeg @ 0x55555599c040] escaping removed 702901 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=728519
[mjpeg @ 0x55555599c040] marker parser used 25618 bytes (204944 bits)
[mjpeg @ 0x55555599c040] escaping removed 585158 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=702808
[mjpeg @ 0x55555599c040] marker parser used 117650 bytes (941200 bits)
[mjpeg @ 0x55555599c040] escaping removed 528692 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=584856
[mjpeg @ 0x55555599c040] marker parser used 56164 bytes (449312 bits)
[mjpeg @ 0x55555599c040] escaping removed 441063 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=528460
[mjpeg @ 0x55555599c040] marker parser used 87397 bytes (699176 bits)
[mjpeg @ 0x55555599c040] escaping removed 1978 bytes
[mjpeg @ 0x55555599c040] marker=da avail_size_in_buf=440548
[mjpeg @ 0x55555599c040] marker parser used 438570 bytes (3508560 bits)
[mjpeg @ 0x55555599c040] marker=d9 avail_size_in_buf=0
[mjpeg @ 0x55555599c040] decode frame unused 0 bytes
[mp3 @ 0x5555559993c0] All info found
[mp3 @ 0x5555559993c0] stream 0: start_time: 0.000 duration: -653583619391.637
[mp3 @ 0x5555559993c0] stream 1: start_time: -102481911520608.625 duration: -102481911520608.625
[mp3 @ 0x5555559993c0] format: start_time: 0.000 duration: -9223372036854.775 bitrate=128 kb/s
[mp3 @ 0x5555559993c0] After avformat_find_stream_info() pos: 1069998 bytes read:1081344 seeks:0 frames:51
Input #0, mp3, from 'pipe:0':
  Metadata:
    genre           : Podcast
    track           : 53
    album           : Original Soundchat
    title           : SUPER SMASH BROS. BRAWL - Episode 53 (JAN 28, 2020)
    artist          : Anonymous Dinosaur
    album_artist    : Joe DeVader & Peter Spezia
    date            : 2020-01-27 20:20
    id3v2_priv.XMP  : <?xpacket begin="\xef\xbb\xbf" id="W5M0MpCehiHzreSzNTczkc9d"?>\x0a<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.6-c148 79.164036, 2019/08/13-01:06:57        ">\x0a <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">\x0a  <rdf
  Duration: N/A, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0, 50, 1/14112000: Audio: mp3, 44100 Hz, stereo, fltp, 128 kb/s
    Stream #0:1, 1, 1/90000: Video: mjpeg (Progressive), 1 reference frame, yuvj444p(pc, bt470bg/unknown/unknown, center), 3100x3100, 0/1, 90k tbr, 90k tbn, 90k tbc (attached pic)
    Metadata:
      comment         : Other
[AVIOContext @ 0x5555559a1e80] Statistics: 1081344 bytes read, 0 seeks

4.2.2:
$ cat OSC053.mp3 | ffmpeg -v trace -i pipe:0
ffprobe version 4.2.2 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 9.2.0 (Alpine 9.2.0)
  configuration: --pkg-config-flags=--static --extra-cflags=-fopenmp --extra-ldflags='-static -fopenmp' --toolchain=hardened --disable-debug --disable-shared --disable-ffplay --enable-static --enable-gpl --enable-nonfree --enable-openssl --enable-iconv --enable-libxml2 --enable-libmp3lame --enable-libfdk-aac --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvpx --enable-libx264 --enable-libx265 --enable-libwebp --enable-libwavpack --enable-libspeex --enable-libaom --enable-libvidstab --enable-libkvazaar --enable-libfreetype --enable-fontconfig --enable-libfribidi --enable-libass --enable-libzimg --enable-libsoxr --enable-libopenjpeg --enable-libdav1d
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[NULL @ 0x55555641e4c0] Opening 'pipe:0' for reading
[pipe @ 0x55555641ed80] Setting default whitelist 'crypto'
Probing mp3 score:12 size:2048
Probing mp3 score:12 size:4096
Probing mp3 score:12 size:8192
Probing mp3 score:12 size:16384
Probing mp3 score:12 size:32768
Probing mp3 score:12 size:65536
Probing mp3 score:12 size:131072
Probing mp3 score:12 size:262144
Probing mp3 score:12 size:524288
[AVIOContext @ 0x555556426f80] Statistics: 1073152 bytes read, 0 seeks
pipe:0: Invalid data found when processing input

4.2.2 with increased formatprobesize:
$ cat OSC053.mp3 | ffmpeg -formatprobesize 2000000 -v trace -i pipe:0
ffprobe version 4.2.2 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 9.2.0 (Alpine 9.2.0)
  configuration: --pkg-config-flags=--static --extra-cflags=-fopenmp --extra-ldflags='-static -fopenmp' --toolchain=hardened --disable-debug --disable-shared --disable-ffplay --enable-static --enable-gpl --enable-nonfree --enable-openssl --enable-iconv --enable-libxml2 --enable-libmp3lame --enable-libfdk-aac --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvpx --enable-libx264 --enable-libx265 --enable-libwebp --enable-libwavpack --enable-libspeex --enable-libaom --enable-libvidstab --enable-libkvazaar --enable-libfreetype --enable-fontconfig --enable-libfribidi --enable-libass --enable-libzimg --enable-libsoxr --enable-libopenjpeg --enable-libdav1d
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[NULL @ 0x5555557b5540] Opening 'pipe:0' for reading
[pipe @ 0x5555557b5e80] Setting default whitelist 'crypto'
Probing mp3 score:12 size:2048
Probing mp3 score:12 size:4096
Probing mp3 score:12 size:8192
Probing mp3 score:12 size:16384
Probing mp3 score:12 size:32768
Probing mp3 score:12 size:65536
Probing mp3 score:12 size:131072
Probing mp3 score:12 size:262144
Probing mp3 score:12 size:524288
Probing mp3 score:51 size:1048658
[mp3 @ 0x5555557b5540] Format mp3 probed with size=2097152 and score=51
id3v2 ver:3 flags:00 len:1048484
[mp3 @ 0x5555557b5540] Skipping 0 bytes of junk at 1048494.
[mp3 @ 0x5555557b5540] Before avformat_find_stream_info() pos: 1048494 bytes read:2105344 seeks:0 nb_streams:2
[mjpeg @ 0x5555557b8180] marker=d8 avail_size_in_buf=883980
[mjpeg @ 0x5555557b8180] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x5555557b8180] marker=e1 avail_size_in_buf=883978
[mjpeg @ 0x5555557b8180] marker parser used 8597 bytes (68776 bits)
[mjpeg @ 0x5555557b8180] marker=ed avail_size_in_buf=875378
[mjpeg @ 0x5555557b8180] marker parser used 10641 bytes (85128 bits)
[mjpeg @ 0x5555557b8180] marker=e1 avail_size_in_buf=864734
[mjpeg @ 0x5555557b8180] marker parser used 4543 bytes (36344 bits)
[mjpeg @ 0x5555557b8180] marker=e2 avail_size_in_buf=860188
[mjpeg @ 0x5555557b8180] marker parser used 3160 bytes (25280 bits)
[mjpeg @ 0x5555557b8180] marker=ee avail_size_in_buf=857026
[mjpeg @ 0x5555557b8180] marker parser used 33 bytes (264 bits)
[mjpeg @ 0x5555557b8180] marker=db avail_size_in_buf=856991
[mjpeg @ 0x5555557b8180] index=0
[mjpeg @ 0x5555557b8180] qscale[0]: 0
[mjpeg @ 0x5555557b8180] index=1
[mjpeg @ 0x5555557b8180] qscale[1]: 0
[mjpeg @ 0x5555557b8180] marker parser used 132 bytes (1056 bits)
[mjpeg @ 0x5555557b8180] marker=c2 avail_size_in_buf=856857
[mjpeg @ 0x5555557b8180] Changing bps from 0 to 8
[mjpeg @ 0x5555557b8180] sof0: picture: 3100x3100
[mjpeg @ 0x5555557b8180] component 0 1:1 id: 0 quant:0
[mjpeg @ 0x5555557b8180] component 1 1:1 id: 1 quant:1
[mjpeg @ 0x5555557b8180] component 2 1:1 id: 2 quant:1
[mjpeg @ 0x5555557b8180] pix fmt id 11111100
[mjpeg @ 0x5555557b8180] Format yuvj444p chosen by get_format().
[mjpeg @ 0x5555557b8180] marker parser used 17 bytes (136 bits)
[mjpeg @ 0x5555557b8180] marker=c4 avail_size_in_buf=856838
[mjpeg @ 0x5555557b8180] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x5555557b8180] escaping removed 750224 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=856479
[mjpeg @ 0x5555557b8180] marker parser used 106255 bytes (850040 bits)
[mjpeg @ 0x5555557b8180] escaping removed 728692 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=750100
[mjpeg @ 0x5555557b8180] marker parser used 21408 bytes (171264 bits)
[mjpeg @ 0x5555557b8180] escaping removed 702901 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=728519
[mjpeg @ 0x5555557b8180] marker parser used 25618 bytes (204944 bits)
[mjpeg @ 0x5555557b8180] escaping removed 585158 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=702808
[mjpeg @ 0x5555557b8180] marker parser used 117650 bytes (941200 bits)
[mjpeg @ 0x5555557b8180] escaping removed 528692 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=584856
[mjpeg @ 0x5555557b8180] marker parser used 56164 bytes (449312 bits)
[mjpeg @ 0x5555557b8180] escaping removed 441063 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=528460
[mjpeg @ 0x5555557b8180] marker parser used 87397 bytes (699176 bits)
[mjpeg @ 0x5555557b8180] escaping removed 1978 bytes
[mjpeg @ 0x5555557b8180] marker=da avail_size_in_buf=440548
[mjpeg @ 0x5555557b8180] marker parser used 438570 bytes (3508560 bits)
[mjpeg @ 0x5555557b8180] marker=d9 avail_size_in_buf=0
[mjpeg @ 0x5555557b8180] decode frame unused 0 bytes
[mp3 @ 0x5555557b5540] All info found
[mp3 @ 0x5555557b5540] stream 0: start_time: 0.000 duration: -653583619391.637
[mp3 @ 0x5555557b5540] stream 1: start_time: -102481911520608.625 duration: -102481911520608.625
[mp3 @ 0x5555557b5540] format: start_time: 0.000 duration: -9223372036854.775 bitrate=128 kb/s
[mp3 @ 0x5555557b5540] After avformat_find_stream_info() pos: 1069998 bytes read:2105344 seeks:0 frames:51
Input #0, mp3, from 'pipe:0':
  Metadata:
    genre           : Podcast
    track           : 53
    album           : Original Soundchat
    title           : SUPER SMASH BROS. BRAWL - Episode 53 (JAN 28, 2020)
    artist          : Anonymous Dinosaur
    album_artist    : Joe DeVader & Peter Spezia
    date            : 2020-01-27 20:20
    id3v2_priv.XMP  : <?xpacket begin="\xef\xbb\xbf" id="W5M0MpCehiHzreSzNTczkc9d"?>\x0a<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.6-c148 79.164036, 2019/08/13-01:06:57        ">\x0a <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">\x0a  <rdf
  Duration: N/A, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0, 50, 1/14112000: Audio: mp3, 44100 Hz, stereo, fltp, 128 kb/s
    Stream #0:1, 1, 1/90000: Video: mjpeg (Progressive), 1 reference frame, yuvj444p(pc, bt470bg/unknown/unknown, center), 3100x3100, 0/1, 90k tbr, 90k tbn, 90k tbc (attached pic)
    Metadata:
      comment         : Other
[AVIOContext @ 0x5555557be080] Statistics: 2105344 bytes read, 0 seeks

[Carl Eugen Hoyos]

Indeed a regression since e9a335150a62bb377a26ce096187b4476145d02b

[Carl Eugen Hoyos]

Fixed by Michael in 23645f95ec0ae3f4ccc8ad2b23d1fbb16575ae16