wmav2 decoding fails for some files

[Andrew Eikum]

libavcodec is failing to properly decode some XWMA files. These files can be decoded properly with Microsoft's xWMAEncode.exe decoder.

The failure can be reproduced with the ffmpeg CLI:

$ ffmpeg -i b8edfb1e970ed7892f35b34a1ef36fee_wma.wav -c:a pcm_s16le out.wav
...
[wmav2 @ 0x55f612b2e700] overflow (2891 > 1864) in spectral RLE, ignoring
[wmav2 @ 0x55f612b2e700] overflow (3421 > 1864) in spectral RLE, ignoring
[wmav2 @ 0x55f612b2e700] overflow (128 > 116) in spectral RLE, ignoring
[wmav2 @ 0x55f612b2e700] next_block_len_bits 4 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x55f612b2e700] frame_len overflow
Error while decoding stream #0:0: Operation not permitted

Aside from the error messages, the audible output from ffmpeg has some glitches. I will attach here a handful of sample files, all of which fail similarly.

[Andrew Eikum]

xwma sample file

[Andrew Eikum]

Unfortunately most of the sample files are too large to upload. I've uploaded them to our server:

​https://www.codeweavers.com/xfer/aeikum/misc/
Access key: ffmpeg

Most are XWMA sample files that show the problem.

b8edfb1e970ed7892f35b34a1ef36fee-wma.wav demonstrates audible corruption in ffmpeg's output.

b8edfb1e970ed7892f35b34a1ef36fee-ms-pcm.wav is the output of that file from Microsoft's decoder, for comparison.

cf198afb3f44977bc029db12d4a5861a-wma.wav can be nice to work with because the errors occur early in the file.

[Olivier Crête]

This should have been fixed by commit f9fbe2f9a9c7d2db98a6e49504a7154f743cd804 in the master branch.

Any chance we can have it backported to 4.4 ?