libavfilter results in a flaw during veracode scan

[Bhawna Khosla]

Veracode Scan of the code where libavfilter is used resulted in a very high flaw.

Flaw : CWE-190 - Integer Overflow or Wraparound

Module: libavfilter.so.7.40.101

Filename : transform.c: 167

Function Name: avfilter_transform

Description: This arithmetic operation results in an integer overflow error. Because the result of this calculation is larger than the maximum possible value for this data type, the result may wrap to become a very small, or negative number, therefore providing an unintended value. Integer overflows can often trigger buffer overflows, which can be exploited to execute arbitrary code.

[mkver]

This function was only ever exported by mistake and was meant to be internal; since the latest major bump, it is internal and has been renamed to ff_affine_transform(). So you need to actually provide a way to trigger this overflow when using this function as intended (which currently is only in the deshake filter); simply passing in a matrix of your choosing is not enough. Can you do so?

Also notice that your version is not even close to git master (the only actually supported version here).

[Bhawna Khosla]

Replying to Bhawna Khosla:
We are not using the function. But it comes up in veracode scan.

[Bhawna Khosla]

Replying to Bhawna Khosla:
We are not using the function. But it comes up in veracode scan.

[mkver]

Reopen this is if you can provide a way to actually trigger this overflow.