FLAC in Ogg decoding is missing WAVEFORMATEXTENSIBLE_CHANNEL_MASK support for non-standard channel layouts

[John Regan]

When decoding: ffmpeg honors the WAVEFORMATEXTENSIBLE_CHANNEL_MASK Vorbis comment in a FLAC file, but ignores it for a FLAC-in-Ogg file.

How to reproduce:

Using the attached "2_point_1.wav" file (contains a 2.1 channel layout - Front Left, Front Right, LFE):

# verify wav file has a 2.1 channel layout:
% ffprobe -v error -show_entries stream=channel_layout 2_point_1.wav 
[STREAM]
channel_layout=2.1
[/STREAM]

# encode with FLAC in Ogg with reference implementation
flac 2_point_1.wav --ogg -o 2_point_1.ogg

flac 1.4.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2023  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2_point_1.wav: WARNING: skipping unknown chunk 'LIST' (use --keep-foreign-metadata to keep)
2_point_1.wav: wrote 157016 bytes, ratio=0.091



# encode to FLAC (native) with reference implementation
% flac 2_point_1.wav -o 2_point_1.flac

flac 1.4.3
Copyright (C) 2000-2009  Josh Coalson, 2011-2023  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

2_point_1.wav: WARNING: skipping unknown chunk 'LIST' (use --keep-foreign-metadata to keep)
2_point_1.wav: wrote 156987 bytes, ratio=0.091


# decode from FLAC in Ogg with ffmpeg

% ffmpeg -i 2_point_1.ogg 2_point_1_from_ogg.wav
ffmpeg version N-111429-g9a2335444b Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.1.1 (GCC) 20230429
  configuration: 
  libavutil      58. 14.100 / 58. 14.100
  libavcodec     60. 22.100 / 60. 22.100
  libavformat    60. 10.100 / 60. 10.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9.  8.102 /  9.  8.102
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
Input #0, ogg, from '2_point_1.ogg':
  Duration: 00:00:06.00, start: 0.000000, bitrate: 210 kb/s
  Stream #0:0: Audio: flac, 48000 Hz, 3.0, s16
    Metadata:
      WAVEFORMATEXTENSIBLE_CHANNEL_MASK: 0x000B
Stream mapping:
  Stream #0:0 -> #0:0 (flac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to '2_point_1_from_ogg.wav':
  Metadata:
    ISFT            : Lavf60.10.100
  Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, 3.0, s16, 2304 kb/s
    Metadata:
      WAVEFORMATEXTENSIBLE_CHANNEL_MASK: 0x000B
      encoder         : Lavc60.22.100 pcm_s16le
[out#0/wav @ 0x5579caa03d80] video:0kB audio:1688kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.005903%
size=    1688kB time=00:00:05.97 bitrate=2314.4kbits/s speed=1.5e+03x 

# decode from FLAC (native) with ffmpeg
% ffmpeg -i 2_point_1.flac 2_point_1_from_flac.wav
ffmpeg version N-111429-g9a2335444b Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.1.1 (GCC) 20230429
  configuration: 
  libavutil      58. 14.100 / 58. 14.100
  libavcodec     60. 22.100 / 60. 22.100
  libavformat    60. 10.100 / 60. 10.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9.  8.102 /  9.  8.102
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
Input #0, flac, from '2_point_1.flac':
  Duration: 00:00:06.00, start: 0.000000, bitrate: 209 kb/s
  Stream #0:0: Audio: flac, 48000 Hz, 2.1, s16
Stream mapping:
  Stream #0:0 -> #0:0 (flac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to '2_point_1_from_flac.wav':
  Metadata:
    ISFT            : Lavf60.10.100
  Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, 2.1, s16, 2304 kb/s
    Metadata:
      encoder         : Lavc60.22.100 pcm_s16le
[out#0/wav @ 0x564b6b34fdc0] video:0kB audio:1688kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.005903%
size=    1688kB time=00:00:05.97 bitrate=2314.4kbits/s speed=1.61e+03x

# verify channel layouts of wave files:
% ffprobe -v error -show_entries stream=channel_layout 2_point_1_from_ogg.wav 
[STREAM]
channel_layout=3.0
[/STREAM]
% ffprobe -v error -show_entries stream=channel_layout 2_point_1_from_flac.wav 
[STREAM]
channel_layout=2.1
[/STREAM]

[John Regan]

A 3-channel, 16-bit little endian WAV file with a 2.1 channel layout (front left, front right, lfe)

[John Regan]

A FLAC in Ogg file produced with the reference encoder, contains a WAVEFORMATEXTENSIBLE_CHANNEL_MASK comment signaling the channel layout.

[John Regan]

A FLAC file produced with the reference encoder, contains a WAVEFORMATEXTENSIBLE_CHANNEL_MASK comment signaling the channel layout.