af_loudnorm: Limiter does not handle new peaks in ATTACK/RELEASE states

[Sebastian Dröge]

Currently the limiter would cause output with too high peaks if a new, higher peak occurs while inside the ATTACK/RELEASE states. These states don't check for any new peaks but simply reduce/increase the gain over the given window based on the previous peak / no peak being found before. This potentially causes clipping and did so when I tested my implementation.

I've found this while porting the filter to Rust for a GStreamer plugin, the code of which can be found here: ​https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/master/audio/audiofx/src/audioloudnorm.rs . This also contains lots of code comments for all the steps, which might be helpful to take over to the ffmpeg code too.

My solutions to these issues might not be the best or most simple approach, but during my testing they worked so feel free to adapt them for the ffmpeg code.

ATTACK happens over a 10ms window, RELEASE over a 100ms window and the peak detection uses a 10ms lookahead and returns the first true peak it found.

ATTACK state:
===================

This happens over a window of 10ms and the gain reduction is slowly increased until the peak that was detected (and is at the end of the window) would have exactly the configured maximum peak.

Now what can happen is that during processing this window there is a new (higher!) peak after the end of the window (remember: 10ms window and 10ms lookahead for peak detection) that would require a steeper slope. We wouldn't detect this as no peak detection at all happens in ATTACK mode, and would let the peak through.

So peak detection has to run, and once we're exactly 10ms before that higher peak the ATTACK state needs to be restarted with the steeper slope.

A possible implementation for this can be found here: ​https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/1730de6cea10cb73ff1ab882a8235d540bfa5061/audio/audiofx/src/audioloudnorm.rs#L924

See also the comments, there are a few tricky corner-cases to consider that also affect the SUSTAIN state.

RELEASE state:
===================

This happens over a window of 100ms, the gain reduction is slowly decreased until it is at 0 again. No search for a new peak inside this window or 10ms after it is performed, that means if a new peak is found during this window there are two possible cases

a) New peak requires lower/equal gain reduction to the *current* gain reduction: in this case we have to switch back to SUSTAIN with the current gain reduction to ensure that this peak is reduced enough to stay below the configured maximum peak.

A possible implementation can be found here: ​https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/1730de6cea10cb73ff1ab882a8235d540bfa5061/audio/audiofx/src/audioloudnorm.rs#L1321-1331

b) New peak requires higher gain reduction: We need to go back to ATTACK mode, but we can SUSTAIN the current gain reduction until we're 10ms before the peak. ATTACK mode can start at the current gain reduction and go to the target necessary reduction as usual.

A possible implementation can be found here: ​https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/1730de6cea10cb73ff1ab882a8235d540bfa5061/audio/audiofx/src/audioloudnorm.rs#L1280-1319

[Carl Eugen Hoyos]

Please provide a sample that allows to reproduce the issue and post the ffmpeg command line you tested together with the complete, uncut console output here.

[Sebastian Dröge]

I provided an analysis and solution above, you can follow it by reading the code. I worked on this a few months ago and don't have the sample anymore that caused the clipping and also didn't try reproducing it with ffmpeg but only noticed it while working on my own code, but it was very easy to reproduce with music and selecting a higher-than-default target loudness.

I reported this here in case you want to fix it in your version of the code too. If there's no interest in that feel free to just close this issue.

[Carl Eugen Hoyos]

Please do not provide an analysis: If you are able to provide an analysis, send a patch to the FFmpeg development mailing list instead. Or - to make this a valid ticket - provide the command line you tested together with the complete, uncut console output and an input sample.

[Sebastian Dröge]

That's fine, let's keep it broken then :) This kind of attitude in the ffmpeg community is exactly why I'm always very hesitant to contribute anything at all, thanks for confirming this again.

Just sad for the time I wasted writing the above and refreshing my memory about the code.