Output pixel format should be chosen based on H.264 profile, not the other way around

[MarkZV]

Users are more likely to know what they need in terms of H.264 profiles, rather than lower level details like pixel formats, cabac, 8x8 transforms, etc. The H.264 profile that is supported by a player or device is usually specified in the documentation or technical specifications, and its purpose is to make knowledge of the other technical codec parameters unnecessary for most users. It is expected that the other codec parameters can be chosen automatically based on the profile.

Right now that works for some things like cabac but is broken for pixel format. Even when a profile is specified explicitly, FFmpeg will choose a pixel format that is not compatible with the profile that is specified. It makes the user specify the pixel format even when only 4:2:0 will work. There is no indication in the documentation that this is necessary or what pixel format should be specified for H.264 baseline, main, or high profile. There is a message within the FFmpeg output but the way it is worded sounds like the option is only needed for greater compatibility and not for the command to work, and it does not make clear why the user should have to worry about pixel formats especially if the profile only supports 4:2:0.

$ ffmpeg -f lavfi -i testsrc -t 5 -profile:v baseline out.mp4
ffmpeg version 2.0-c96b3ae Copyright (c) 2000-2013 the FFmpeg developers
  built on Jul 29 2013 15:12:25 with gcc 4.2.1 (GCC) (Apple Inc. build 5666) (dot 3)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libspeex --enable-libass --enable-libbluray --enable-gnutls --enable-libfreetype --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/gcc-4.2 --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-nonfree --enable-libfdk-aac --enable-libfaac
  libavutil      52. 40.100 / 52. 40.100
  libavcodec     55. 19.100 / 55. 19.100
  libavformat    55. 12.102 / 55. 12.102
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 82.100 /  3. 82.100
  libswscale      2.  4.100 /  2.  4.100
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
No pixel format specified, yuv444p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
x264 [error]: baseline profile doesn't support 4:4:4
[libx264 @ 0x103801600] Error setting profile baseline.
[libx264 @ 0x103801600] Possible profiles: baseline main high high10 high422 high444
Output #0, mp4, to 'out.mp4':
    Stream #0:0: Video: h264, yuv444p, 320x240 [SAR 1:1 DAR 4:3], q=-1--1, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> libx264)
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

If the principle is to not lose any data unless the user specifies otherwise, then the default should be -crf 0. However I believe that the principle is actually to choose defaults that users will find the most useful. Therefore, it seems that by default the libx264 encoder should make choices for unspecified settings based on the profile, and when no profile or options requiring a higher profile are explicitly specified (e.g. -crf 0), it should choose defaults compatible with high profile. Not because high produces the best quality, just as -crf 23 does not produce the best quality, but because high profile (like -crf 23) will work well for most people that don't care enough to specify a profile.

In other words a 4:4:4 pixel format should not be automatically chosen unless the high444 profile was chosen (since that is the only profile compatible with 4:4:4), and the high444 profile should not be chosen unless the user explicitly specifies an option that requires it (e.g. -profile:v high444 or -crf 0). An error should be reported if the user specifies conflicting options, such as -profile:v baseline -crf 0, but no error should be reported if the specified parameters can be accommodated by choosing a different value for an automatically chosen unspecified parameter.