UBSan: int overflow in avidec.c

[andreafioraldi]

Build ffmpeg 4.2.2 using clang and ubsan (-fsanitize=undefined).

Command line: ./ffmpeg.ubsan -y -i ./input -c:v mpeg4 -c:a out.mp4
Output:

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with clang version 8.0.0-3~ubuntu18.04.2 (tags/RELEASE_800/final)
  configuration: --cc=clang-8 --cxx=clang++-8 --ld=clang-8
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
Trailing options were found on the commandline.
libavformat/avidec.c:846:44: runtime error: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
[avi @ 0x902f600] Something went wrong during header parsing, tag [0][0][0][0] has size 1314212352, I will ignore it and try to continue anyway.
[NULL @ 0x9031140] [IMGUTILS @ 0x7fffffffcfc8] Picture size 1426063360x2147483648 is invalid
[avi @ 0x902f600] Failed to open codec in avformat_find_stream_info
[rawvideo @ 0x9031140] Invalid pixel format.
[avi @ 0x902f600] Failed to open codec in avformat_find_stream_info
[avi @ 0x902f600] Could not find codec parameters for stream 0 (Video: rawvideo, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, avi, from './ffmpeg_ubsan_out/f1/crashes/id:000072,sig:04,src:002134,time:5783265,op:havoc,rep:2':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo, none, 100 fps, 100 tbr, 100 tbn, 100 tbc
At least one output file must be specified

The problem is here:
st->codecpar->height = FFABS(st->codecpar->height);

The FFABS result in 0x80000000 that is a negative number.

As you can see in the warning after the UBSan violation this results in a large size in images, not sure if this can be used to attack the application when st->codecpar->height is used in relation to controlled buffers.

I attach a minimal file to reproduce the issue in base64:

UklGRi4WAP9BVkkgTElTVOwRAABoZHJsYXZpaDgAAAAQJwAAqGEAAP////IQCQAAAAAAAAAAAAABAAAAAAAQAAEAAAABAAAAAAAAAAAAAAAAAAAAAAAAAExJU1SUEAAAc3RybHN0cmg4AAAAdmlkc0ZNUDQAAAAAAAAAAAAAAAABAAAAZAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAEAAQBzdHJmKAAAACgAAAAAAABVAAAAgAAAAAAAAAAAAE5LGBAAAAQAAQAAAAEAGABGTVA0AwAAAAAAAAAAAAAAAAAAAAAAAABKVU5LGBAAAAQAgAAAAAAAMDBkYwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgXAIV3d3d3d3d3Zm9mgGZmZmaCZmYAABQAAAAiAQAAAA9VBecAF7UAd3d3d3d3d3d3d3d3d3dsaFf/AAAAd3d3EAAAAA4AAAAABv///4BXkHU2//b5Df/wAAD//vYAABIAfwB/HwMA5GQAAID/APo=

[mkver]

Fixed in ec8ff659f57786c4cb089b07dfeab7e5cbab8d52.