Massive memory usage caused by a small file leads to crashes in some applications

[Athari]

A lot of applications playing this small WebM+VP9 file either crash or allocate gigabytes of memory. Applications included: Chrome, MPC-HC, Telegram on Windows, Linux, Android and iOS. This file is crafted as a Telegram sticker and is used by malicious users to make Telegram unusable or even crash OS (I've heard it crashes Apple's OSes).

When FFMPEG executable works with this file, it allocates 3 GB of memory on my machine and decoding takes much more time than normal.

> ffmpeg -i CrashSticker.webm CrashSticker.ivf
ffmpeg version 2022-05-08-git-f77ac5131c-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.3.0 (Rev1, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      57. 24.101 / 57. 24.101
  libavcodec     59. 27.100 / 59. 27.100
  libavformat    59. 23.100 / 59. 23.100
  libavdevice    59.  6.100 / 59.  6.100
  libavfilter     8. 38.100 /  8. 38.100
  libswscale      6.  6.100 /  6.  6.100
  libswresample   4.  6.100 /  4.  6.100
  libpostproc    56.  5.100 / 56.  5.100
Input #0, matroska,webm, from 'CrashSticker.webm':
  Metadata:
    title           : https://t.me/totallynormalchannel
    ENCODER         : Lavf58.29.100
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv), 512x512, SAR 1:1 DAR 1:1, 25 fps, 25 tbr, 1k tbn (default)
    Metadata:
      ALPHA_MODE      : 1
      ENCODER         : Lavc58.54.100 libvpx-vp9
      DURATION        : 00:00:04.920000000
Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (native) -> vp8 (libvpx))
Press [q] to stop, [?] for help
[libvpx @ 0000023b9c822ec0] v1.11.0-194-g8ac72859e
[libvpx @ 0000023b9c822ec0] Neither bitrate nor constrained quality specified, using default CRF of 32 and bitrate of 256kbit/sec
Output #0, ivf, to 'CrashSticker.ivf':
  Metadata:
    title           : https://t.me/totallynormalchannel
    encoder         : Lavf59.23.100
  Stream #0:0: Video: vp8 (VP80 / 0x30385056), yuv420p(tv, progressive), 512x512 [SAR 1:1 DAR 1:1], q=2-31, 256 kb/s, 25 fps, 25 tbn (default)
    Metadata:
      ALPHA_MODE      : 1
      DURATION        : 00:00:04.920000000
      encoder         : Lavc59.27.100 libvpx
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  123 fps=5.5 q=32.0 Lsize=      43kB time=00:00:04.92 bitrate=  71.3kbits/s speed=0.222x
video:41kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.560803%

I initially reported the issue to WebM/VPX bug tracker, here's the analysis from its dev:

libwebm fails to load the file (in vpxdec, mkvparser_sample, webm_info), though the newer webm_parser handles it all right with minimal memory usage. There are different demuxers and decoders in the applications listed, though. In some of them the demuxer is from ffmpeg, like Chrome. Apple uses their own parser for the platform.

In ffmpeg using both the ffvp9 and libvpx decoders I see a similar max rss:
libvpx: bench: maxrss=7470548kB
ffvp9: bench: maxrss=7466176kB

Converting the file to an ivf file for use in vpxdec shows ~1.4GiB of usage. I'll take a closer look, but in general the bitstream and file appear to be valid so far. If you have a crash stack trace that points to libvpx we can take a look. I haven't hit an allocation error on any test devices, but can try something with lower memory.

The initial 2 frames are 512x512, but after that they are 15000x15000 which would be the cause of the large allocation in vpxdec. I'm not sure about the difference between the source webm and the transmuxed ivf yet.

Decoding the transmuxed ivf file in ffmpeg shows about the same 7GiB memory usage, so there may be more frame buffering or other allocations. If I set the number of threads in ffmpeg to 1 the allocation profile will roughly match libvpx.

There's nothing actionable here from the libvpx side. The library can be configured to reject large frames sizes using --size-limit [1].

I don't know what arguments they used exactly and couldn't reduce memory usage by setting threads to 1, but FFMPEG does show unreasonable memory allocations with the command line I used above.

[mkver]

This is basically a duplicate of #9738.