flac md5 error with LPC precision 15

[Carl Eugen Hoyos]

​http://thread.gmane.org/gmane.comp.video.ffmpeg.user/56347
A user uploaded a wav file that can be converted to flac but the flac cli reports a md5 error if the default lpc precision 15 was chosen and the output file is not bit-identical to the input file. FFmpeg allows to decode the flac file losslessly.

$ ffmpeg -i testwav_cut.wav out.flac
ffmpeg version N-71222-gd858c3a Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      54. 22.100 / 54. 22.100
  libavcodec     56. 33.100 / 56. 33.100
  libavformat    56. 28.100 / 56. 28.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 13.101 /  5. 13.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Guessed Channel Layout for  Input Stream #0.0 : stereo
Input #0, wav, from 'testwav_cut.wav':
  Metadata:
    encoder         : Lavf56.28.100
  Duration: 00:00:10.00, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, 2 channels, s16, 1411 kb/s
Output #0, flac, to 'out.flac':
  Metadata:
    encoder         : Lavf56.28.100
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s16, 128 kb/s
    Metadata:
      encoder         : Lavc56.33.100 flac
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le (native) -> flac (native))
Press [q] to stop, [?] for help
size=    1114kB time=00:00:10.00 bitrate= 913.0kbits/s
video:0kB audio:1106kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.731545%

$ flac -t out.flac

flac 1.3.1, Copyright (C) 2000-2009  Josh Coalson, 2011-2014  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

out.flac: ERROR, MD5 signature mismatch

Works fine with -lpc_coeff_precision 14.

[lvqcl]

There are two functions in libavcodec/flacdsp.c: flac_lpc_encode_c_16() and flac_lpc_encode_c_32(). The former uses variables of type int32_t to store the sum of coefs[j] * smp[j]; the latter uses variables of type int64_t for this.

Similar functions exist in libFLAC: FLAC__lpc_compute_residual_from_qlp_coefficients() and FLAC__lpc_compute_residual_from_qlp_coefficients_wide(), respectively.

Here's how libFLAC chooses which function to use:

if(subframe_bps + qlp_coeff_precision + FLAC__bitmath_ilog2(order) <= 32)
	use_32bit_sum();
else
	use_64bit_sum();

while FFMPEG uses the following condition:

    if (bps > 16) {
        use_64bit_sum();
    } else {
        use_32bit_sum();
    }

It seems that libFLAC doesn't allow the sum to overflow while FFMPEG allows it, and it's a bug in FFMPEG.

---

The same logic applies to a choice between decoding routines, that's why FFMPEG is able to decode the flac file it created: a bug in the decoder compensates a bug in the encoder.

[Carl Eugen Hoyos]

Thank you for the analysis!
Initial patch sent:
​http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/192906

[Carl Eugen Hoyos]

Applied in e609cfd6