videotoolbox and vda fail to initialize for h264

[ponpon]

thank you for the excellent programs.
videotoolbox fails to initialize for not only mpeg2, mpeg4 but also h264.
vda fails to initialize for h264.
can't encode with videotoolbox.

How to reproduce:

ffmpeg -y -hwaccel videotoolbox -threads 1 -i  test.mp4  -f null - 
ffmpeg version N-80117-gdac030d Copyright (c) 2000-2016 the FFmpeg developers
  built with clang version 3.7.1 (tags/RELEASE_371/final)
  configuration: --prefix=/sw --cc=clang --cxx=clang++
  libavutil      55. 24.100 / 55. 24.100
  libavcodec     57. 43.100 / 57. 43.100
  libavformat    57. 37.101 / 57. 37.101
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 46.100 /  6. 46.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  0.101 /  2.  0.101
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.37.101
  Duration: 00:00:10.01, start: 0.000000, bitrate: 214 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 176x144 [SAR 128:117 DAR 1408:1053], 212 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : VideoHandler
[null @ 0x7fdb21018000] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.37.101
    Stream #0:0(und): Video: wrapped_avframe, yuv420p, 176x144 [SAR 128:117 DAR 1408:1053], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc57.43.100 wrapped_avframe
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fdb21017a00] decode_slice_header error
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fdb21017a00] decode_slice_header error
[h264 @ 0x7fdb21017a00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
...
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fdb21017a00] decode_slice_header error
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fdb21017a00] decode_slice_header error
[h264 @ 0x7fdb21017a00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fdb21017a00] decode_slice_header error
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fdb21017a00] decode_slice_header error
[h264 @ 0x7fdb21017a00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Conversion failed!

ffmpeg -y -hwaccel vda -threads 1 -i  test.mp4  -f null - 
ffmpeg version N-80117-gdac030d Copyright (c) 2000-2016 the FFmpeg developers
  built with clang version 3.7.1 (tags/RELEASE_371/final)
  configuration: --prefix=/sw --cc=clang --cxx=clang++
  libavutil      55. 24.100 / 55. 24.100
  libavcodec     57. 43.100 / 57. 43.100
  libavformat    57. 37.101 / 57. 37.101
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 46.100 /  6. 46.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  0.101 /  2.  0.101
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.37.101
  Duration: 00:00:10.01, start: 0.000000, bitrate: 214 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 176x144 [SAR 128:117 DAR 1408:1053], 212 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : VideoHandler
[null @ 0x7ffed381d600] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.37.101
    Stream #0:0(und): Video: wrapped_avframe, yuv420p, 176x144 [SAR 128:117 DAR 1408:1053], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc57.43.100 wrapped_avframe
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264 @ 0x7ffed381d000] Cannot initialize VDA -12473
Error creating VDA decoder.
vda hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7ffed381d000] decode_slice_header error
[h264 @ 0x7ffed381d000] Cannot initialize VDA -12473
Error creating VDA decoder.
vda hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7ffed381d000] decode_slice_header error
[h264 @ 0x7ffed381d000] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
[h264 @ 0x7ffed381d000] Cannot initialize VDA -12473
...
[h264 @ 0x7ffed381d000] Cannot initialize VDA -12473
Error creating VDA decoder.
vda hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7ffed381d000] decode_slice_header error
[h264 @ 0x7ffed381d000] Cannot initialize VDA -12473
Error creating VDA decoder.
vda hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7ffed381d000] decode_slice_header error
[h264 @ 0x7ffed381d000] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Conversion failed!

[ponpon]

please ignore 'can't encode with videotoolbox.'. it's my mistake.

i did some tests. from it, allso, this problem arises for 480p, 720p, 1080p since commit 772ad7142dff590c7bb30370acf5d3c9a8fb512b.

commit 772ad7142dff590c7bb30370acf5d3c9a8fb512b is merged from commit ca2f19b9cc37be509d85f05c8f902860475905f8, avconv. for 720p, 1080p, this problem doesn't arises in commit ca2f19b9cc37be509d85f05c8f902860475905f8. for 480p, another problem arises. probably, its cause is avconv.

for 480p, 720p, 1080p, this problem doesn't arises in commit 9cc1ab63ac0fa524ad9b6c8ef600dc4fc69e2b31.

[ponpon]

using a patch ​http://permalink.gmane.org/gmane.comp.video.ffmpeg.devel/214982, 480p.mp4, 720p.mp4, 1080p.mp4 can be playbacked with videotoolbox on osx1095.
test.mp4 may not be supported by videdotoolbox because it can be playbacked with commands on #5272.

playbacking 480p.mp4, 720p.mp4, 1080p.mp4 with -hwaccel auto and the patch
it seems hwaccel auto uses vda and falls back h264 decoder of ffmpeg. where videotoolbox hardware decoding isnt used.
i dont know if it is a right behavior.

[Rick Kern]

Videotoolbox has a lower limit on the resolution it can use with hardware. To use test.mp4 (which is 176x144) you would need to use software (don't use -hwaccel xxx) and enable software encoding in the videotoolbox encoder with -allow_sw 1.

[ponpon]

Replying to kernrj:
thank you for your suggestion.
as a matter of fact, test.mp4 was made with the videotoolbox encoder with -allow_sw 1.
i am enjoying your work!!! probably, many mac users, too.
thank you for your work.

[Rick Kern]

Fixed in 8e47a99f576d