avcodec_find_best_pix_fmt_of_list depends on the order?

[Matt]

avcodec_find_best_pix_fmt_of_list seems to be returning different values depending on the order of the input.

Following is a example using python/cython, but it should be easy to understand:

AVPixelFormat *fmts2 = [AV_PIX_FMT_RGB24, AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE]
AVPixelFormat *fmts3 = [AV_PIX_FMT_GRAY8, AV_PIX_FMT_RGB24, AV_PIX_FMT_NONE]
print <int>avcodec_find_best_pix_fmt_of_list(fmts2, AV_PIX_FMT_YUV420P, 0, &loss)
print <int>avcodec_find_best_pix_fmt_of_list(fmts3, AV_PIX_FMT_YUV420P, 0, &loss)

Which prints:

2
8

I.e. if rgb24 is first that is returned, otherwise gray is preferred. If this is somewhat intentional, then it should be documented as such as as well as mentioned that the highest quality pixel formats should be listed first.

$ ffmpeg
ffmpeg version N-76278-gd897d4c Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55.  4.100 / 55.  4.100
  libavcodec     57. 10.100 / 57. 10.100
  libavformat    57. 11.100 / 57. 11.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 14.100 /  6. 14.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100

[Frederic Devernay]

Confirmed.
The "considered" loss mask is not the same for all pixel formats (the loss of first format is used a a mask for the second, etc) so that if the first pixel format loses resolution, resolution loss is not considered for other formats.

A workaround is to always pass NULL as the last parameter. This really returns the best pixel format.

A real fix is the following code, which uses the same "considered loss" mask for all formats in the list:

enum AVPixelFormat avcodec_find_best_pix_fmt_of_list(const enum AVPixelFormat *pix_fmt_list,
                                            enum AVPixelFormat src_pix_fmt,
                                            int has_alpha, int *loss_ptr){
    int i;

    enum AVPixelFormat best = AV_PIX_FMT_NONE;

    for(i=0; pix_fmt_list[i] != AV_PIX_FMT_NONE; i++) {
        int loss = loss_ptr ? *loss_ptr : 0;
        best = avcodec_find_best_pix_fmt_of_2(best, pix_fmt_list[i], src_pix_fmt, has_alpha, &loss);
    }
    if (loss_ptr) {
        *loss_ptr = av_get_pix_fmt_loss(best,  src_pix_fmt, has_alpha);
    }
    return best;
}

[Carl Eugen Hoyos]

Replying to devernay:

A real fix is the following code, which uses the same "considered loss" mask for all formats in the list:

Please send your patch made with git format-patch to the development mailing list, patches on this bug tracker are ignored.

[Frederic Devernay]

patch submitted to the mailing list

[Carl Eugen Hoyos]

No, unfortunately not;-(

[Frederic Devernay]

it's being moderated (I'm not an ffmpeg-devel list member)