Add .avif support (AV1 Image File Format

[Ewout]

Netflix released ​samples of the new AV1 Image File Format (​AVIF), it would be nice if support could be added for this format.

[Ewout]

Some more test files: ​https://github.com/AOMediaCodec/av1-avif/tree/master/testFiles/Microsoft

[Ewout]

Current debug commandline if .avif is used as input:

Opening an input file: C:\Users\Ewout\AVIF\Chimera-AV1-8bit-1920x1080-6736kbps-6.avif.
[NULL @ 000001c464c4a780] Opening 'C:\Users\Ewout\AVIF\Chimera-AV1-8bit-1920x1080-6736kbps-6.avif' for reading
[file @ 000001c464c4b900] Setting default whitelist 'file,crypto'
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001c464c4a780] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001c464c4a780] ISO: File Type Major Brand: mif1
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001c464c4a780] moov atom not found
[AVIOContext @ 000001c464c53bc0] Statistics: 66192 bytes read, 3 seeks
C:\Users\Ewout\AVIF\Chimera-AV1-8bit-1920x1080-6736kbps-6.avif: Invalid data found when processing input

Current debug commandline if .avif is used as output:

Opening an output file: png-to-avif.avif.
[NULL @ 0000021f3f4ecc40] Unable to find a suitable output format for 'png-to-avif.avif'
png-to-avif.avif: Invalid argument
[AVIOContext @ 0000021f3f4f3b00] Statistics: 115667 bytes read, 0 seeks

[Ewout]

Depends somewhat on #7600 for lossless encoding.

Microsoft ​is going to support AVIF in Windows 10.

[Carl Eugen Hoyos]

Definitely independent of ticket #7600.

[Ewout]

Aside from using libaom directly for encoding and muxing into an .avif file, ​libavif could also be considered.

[Artem S. Tashkinov]

AV1 decoding support is already there, the container format is probably already supported as well, why doesn't ffmpeg understand AVIF files? ;-)

[av1masterrace]

Replying to birdie:

AV1 decoding support is already there, the container format is probably already supported as well, why doesn't ffmpeg understand AVIF files? ;-)

It doesn't seem like it. HEIC images aren't supported yet either. See #6521
Would be nice if it got support because it supports animations but doesn't look as trashy as WebP and GIF.

[Balling]

Ticket #7600 is fixed in 288ca1c66740640d5742f2921efbf502d83f8b6a and thus we now have no blocking stuff, AVIF container is implemented here: ​https://github.com/Swaraj1998/FFmpeg/commit/9a885cddb3550ab863a60d02c5fb78e4ae206cf1 and here: ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20170819072456.11276-1-rodger.combs@gmail.com/

Microsoft ​is going to support AVIF in Windows 10.

That will not help, cause AVIF extensions use MFF and we do not.

[bat_guano]

Hi

These avif files are showing up a lot now on the web.
I can see them in Chrome Developer tools.
This is an example website ---> ​https://besteltaart.nl/

As Ewout said 2 years ago, there is a decoder available at ---> ​https://github.com/AOMediaCodec/libavif

But it would be useful if FFmpeg could decode them too.

~ $ avifdec --version
Version: 0.9.1 (dav1d [dec]:185194b

~ $ avifdec --help
Syntax: avifdec [options] input.avif output.[jpg|jpeg|png|y4m]
        avifdec --info    input.avif

[Balling]

Finally ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220217211838.3404301-1-vigneshv@google.com/

[Balling]

[flac @ 000002129c2059c0] Unknown attached picture mimetype: image/avif.

For this sample ​https://github.com/ktmf01/flac-test-files/blob/main/subset/59%20-%20AVIF%20PICTURE.flac should be also fixed.

[Balling]

It now mostly works except for decoding some samples with error

iloc: item_count > 1 not supported

but that should be fixed in ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220424183514.986374-1-vigneshv@google.com/

Sample file with this error is here:
​https://bugzilla.mozilla.org/attachment.cgi?id=9241562

Ecoding should be good, lossless does work with same md5 as original frame!

[Vignesh Venkatasubramanian]

@Balling, thanks for verifying the AVIF support!

Support for still and animated AVIF image muxing/parsing has been merged. There are some patches under review to support more AVIF files.

I think this ticket can be marked as fixed for now since basic support has been implemented. Please feel free to re-open if there are any other issues with AVIF muxing/parsing.

[Balling]

@Vignesh

We need

a) the chroma siting support for top left of 420 right here: ​https://github.com/AOMediaCodec/av1-avif/issues/88
b) avif MIME
c) the patch for for item_count merged
d) alpha patch support merged
e) HEIC container patches reworked and applied (see HEIF ticket) and HEIF integrated too, since it is so close
f) maybe cpu-used set to 0 and of course defaulting for lossless (crf 0).

Thank you and sorry for reopening, but I found out that AVIF is more effective than JPEG XL since JPEG XL requires R'G'B' convertion which is lossy, dither is hard to compress, but with it turned off rounding of YCbCr is even harder to compress apparently, does not support subsampling like in old JPEG, no support for superblack/superwhite in limited range and out-of-gamut like sYCC, no support for BT.709 matrix like in JPEG 2000, which is just 💩. JPEG XL is useless for me.

[Balling]

Assigning to you, see previous comment.

[Vignesh Venkatasubramanian]

Hi Balling,

a) the chroma siting support for top left of 420 right here: ​​https://github.com/AOMediaCodec/av1-avif/issues/88

What exactly do you need ffmpeg to do for this issue? I am not sure i understand the requirement.

b) avif MIME

Same here. What mime type support do you want in ffmpeg?

c) the patch for for item_count merged
d) alpha patch support merged

These two are now merged. I am working on some follow-up patches for still image alpha and grid image support.

e) HEIC container patches reworked and applied (see HEIF ticket) and HEIF integrated too, since it is so close

I am not very familiar with the HEIC features. I will try to look into it.

f) maybe cpu-used set to 0 and of course defaulting for lossless (crf 0).

You can pass these options now to libaom (or other av1 encoders availabvle on ffmpeg) using the command line options.

[Balling]

What exactly do you need ffmpeg to do for this issue? I am not sure i understand the requirement.

All HDR movies use top-left siting for chroma. When you extract a plane 420 samples should be tagged as top-left in AVIF. Use any HDR10 movie. You already pass transfer, matrix and primaries.

What mime type support do you want in ffmpeg?

image/avif, see comment 14.

[Vignesh Venkatasubramanian]

image/avif, see comment 14.

I see. It seems like the flac picture parser does not support images with container formats and only works with images that can be directly parsed (like png, jpeg, etc). Since AVIF always lives in a container + codec set up, it might be non-trivial to add support for that in the flac picture parser.

I will leave this as an open item.

[Vignesh Venkatasubramanian]

All HDR movies use top-left siting for chroma. When you extract a plane 420 samples should be tagged as top-left in AVIF. Use any HDR10 movie. You already pass transfer, matrix and primaries.

If you are using libaom-av1 encoder for creating AVIF images, you can pass -aom-params "chroma-sample-location=xx" for whatever value you need.

[Balling]

If you are using libaom-av1 encoder for creating AVIF images, you can pass -aom-params "chroma-sample-location=xx" for whatever value you need.

It should work automatically.

[Vignesh Venkatasubramanian]

It should work automatically.

When encoding, parameters like transfer, matrix, primaries and chroma_sample_positiona are usually set by the user (according to libavcodec documentation).

[Balling]

When encoding, parameters like transfer, matrix, primaries and chroma_sample_positiona are usually set by the user (according to libavcodec documentation).

That is false ever since the auto insertion got implemented in 7369595c556516b90bf3acdf85043dc11add7d89 (in practice not everyone migrated to it, like libx264 encoder wrapper and there is a bug when you go from RGB to some of 3 types of YCbCr). The point is it does work though for AVIF. You just forgot to implement chroma_sample_position syntax element (encoded using two bits). See this, looks like center will be implemented using CSP_UNKNOWN. ​https://github.com/AOMediaCodec/av1-avif/issues/88

[diltuepromil]

Replying to Balling:

JPEG XL requires R'G'B' convertion which is lossy, dither is hard to compress, but with it turned off rounding of YCbCr is even harder to compress apparently, does not support subsampling like in old JPEG, no support for superblack/superwhite in limited range and out-of-gamut like sYCC, no support for BT.709 matrix like in JPEG 2000

On ​JXL Discord, BlueSwordM ​said:

1. JXL requires RGB conversion

Nope, it converts stuff to XYB in lossy mode at very high bit depth internally, which is basically the best thing psycho-visually speaking, especially since no modern lossy video codecs use RGB(for a good reason).

2. Dither is hard to compress.

With its current featureset, libjxl with its encoder cjxl is the absolute best encoder I've found to encode any kind of dither efficiently.

3. Does not support subsampling.

Not a bad thing honestly. We have to move with the times.

4. No support for BT709 matrix.

Is that even really needed these days since you can just attach the relevant metadata or use something like BT1886?

[Balling]

Nope, it converts stuff to XYB in lossy mode at very high

I was only testing lossless mode that does not support XYB.

libjxl with its encoder cjxl is the absolute best encoder I've found to encode any kind of dither efficiently.

No dither needed in AVIF. Dither is applied on decode as it should be.

Not a bad thing honestly. We have to move with the times.

Will not be lossless.

just attach the relevant metadata or use something like BT1886?

Has nothing to do with BT.709 matrix. That is EOTF, transfer.

[Balling]

So, can you implement chroma_sample_position syntax element?

[Balling]

Another ping. chroma_sample_position syntax element must be implemented.

[hydra3333]

Just a thought ...
​https://trac.ffmpeg.org/ticket/6521#comment:21

which asks about any possibility of ffmpeg supporting libheif
​​https://github.com/strukturag/libheif

[Vignesh Venkatasubramanian]

Replying to hydra3333:

Just a thought ...
​https://trac.ffmpeg.org/ticket/6521#comment:21

which asks about any possibility of ffmpeg supporting libheif
​​https://github.com/strukturag/libheif

The ffmpeg MOV parser (and the muxer) has the capability to handle HEIF with minimal changes. I have a patch locally for parsing HEIF. i will try to clean it up and upload it.

I don't think we need to add a seperate dependency for something that ffmpeg almost already natively supports. :)

[hydra3333]

Thank you.
I'll keep an eye out for a libheif patch (if it's not already implemented).

[Balling]

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231004164019.1428483-1-vigneshv@google.com/