h264_videotoolbox encoder gets stuck in an invalid state

[kthelgason]

Summary of the bug:
When encoding with videotoolbox, if an error occurs (for example in my case, an invalid option specified on the command line) or FFmpeg for some reason exits abnormally, the hardware encoder gets stuck in a state where it is unavailable, and requires a restart of the computer to put it back online.
All subsequent attempts to use the encoder fail with:

[h264_videotoolbox @ 0x7f8b8901e400] Error: cannot create compression session: -12915
[h264_videotoolbox @ 0x7f8b8901e400] Try -allow_sw 1. The hardware encoder may be busy, or not supported.

How to reproduce:

$  ffmpeg -v debug -i sample_lossless.mkv -c:v h264_videotoolbox -profile:v baseline -b:v 200k -bufsize 200k -vbsf h264_mp4toannexb -vf scale=320:240 baseline_200kbps_qvga.mkv

Full output:

ffmpeg version 3.1.2 Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 7.3.0 (clang-703.0.31)
  configuration: --prefix=/Users/kthelgason/homebrew/Cellar/ffmpeg/3.1.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libxvid --disable-lzma --enable-vda
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 47.100 /  6. 47.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'.
Reading option '-i' ... matched as input file with argument 'sample_lossless.mkv'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_videotoolbox'.
Reading option '-profile:v' ... matched as option 'profile' (set profile) with argument 'baseline'.
Reading option '-b:v' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '200k'.
Reading option '-bufsize' ... matched as AVOption 'bufsize' with argument '200k'.
Reading option '-vbsf' ... matched as option 'vbsf' (deprecated) with argument 'h264_mp4toannexb'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'scale=320:240'.
Reading option 'baseline_200kbps_qvga.mkv' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input file sample_lossless.mkv.
Successfully parsed a group of options.
Opening an input file: sample_lossless.mkv.
[file @ 0x7f8b88d00160] Setting default whitelist 'file,crypto'
[matroska,webm @ 0x7f8b89800000] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
[matroska,webm @ 0x7f8b89800000] Before avformat_find_stream_info() pos: 698 bytes read:32768 seeks:0 nb_streams:1
[h264 @ 0x7f8b89811e00] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 0x7f8b89811e00] nal_unit_type: 8, nal_ref_idc: 3
[h264 @ 0x7f8b89811e00] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 0x7f8b89811e00] nal_unit_type: 8, nal_ref_idc: 3
[h264 @ 0x7f8b89811e00] user data:"x264 - core 148 r2668 fd2c324 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0"
[h264 @ 0x7f8b89811e00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7f8b89811e00] nal_unit_type: 5, nal_ref_idc: 3
[h264 @ 0x7f8b89811e00] user data:"x264 - core 148 r2668 fd2c324 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0"
[h264 @ 0x7f8b89811e00] user data:"x264 - core 148 r2668 fd2c324 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0"
[h264 @ 0x7f8b89811e00] nal_unit_type: 1, nal_ref_idc: 2
    Last message repeated 5 times
[matroska,webm @ 0x7f8b89800000] All info found
[matroska,webm @ 0x7f8b89800000] After avformat_find_stream_info() pos: 1313186 bytes read:1313186 seeks:0 frames:8
Input #0, matroska,webm, from 'sample_lossless.mkv':
  Metadata:
    ENCODER         : Lavf57.41.100
  Duration: 00:00:30.03, start: 0.000000, bitrate: 21992 kb/s
    Stream #0:0, 8, 1/1000: Video: h264 (High 4:4:4 Predictive), 1 reference frame, yuv422p(left), 640x480, 0/1, SAR 1:1 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
    Metadata:
      ENCODER         : Lavc57.48.101 libx264
      DURATION        : 00:00:30.030000000
Successfully opened the file.
Parsing a group of options: output file baseline_200kbps_qvga.mkv.
Applying option c:v (codec name) with argument h264_videotoolbox.
Applying option profile:v (set profile) with argument baseline.
Applying option b:v (video bitrate (please use -b:v)) with argument 200k.
Applying option vbsf (deprecated) with argument h264_mp4toannexb.
Applying option vf (set video filters) with argument scale=320:240.
Successfully parsed a group of options.
Opening an output file: baseline_200kbps_qvga.mkv.
File 'baseline_200kbps_qvga.mkv' already exists. Overwrite ? [y/N] y
[file @ 0x7f8b88c184e0] Setting default whitelist 'file,crypto'
Successfully opened the file.
detected 8 logical cores
[Parsed_scale_0 @ 0x7f8b8a800000] Setting 'w' to value '320'
[Parsed_scale_0 @ 0x7f8b8a800000] Setting 'h' to value '240'
[Parsed_scale_0 @ 0x7f8b8a800000] Setting 'flags' to value 'bicubic'
[Parsed_scale_0 @ 0x7f8b8a800000] w:320 h:240 flags:'bicubic' interl:0
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] Setting 'video_size' to value '640x480'
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] Setting 'pix_fmt' to value '4'
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] Setting 'time_base' to value '1/1000'
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] Setting 'frame_rate' to value '30000/1001'
[graph 0 input from stream 0:0 @ 0x7f8b8a800420] w:640 h:480 pixfmt:yuv422p tb:1/1000 fr:30000/1001 sar:1/1 sws_param:flags=2
[format @ 0x7f8b8a800ae0] compat: called with args=[videotoolbox_vld|nv12|yuv420p]
[format @ 0x7f8b8a800ae0] Setting 'pix_fmts' to value 'videotoolbox_vld|nv12|yuv420p'
[AVFilterGraph @ 0x7f8b88c16e60] query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
[Parsed_scale_0 @ 0x7f8b8a800000] picking yuv420p out of 2 ref:yuv422p alpha:0
[Parsed_scale_0 @ 0x7f8b8a800000] w:640 h:480 fmt:yuv422p sar:1/1 -> w:320 h:240 fmt:yuv420p sar:1/1 flags:0x4
[h264 @ 0x7f8b89052e00] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 0x7f8b89052e00] nal_unit_type: 8, nal_ref_idc: 3
[h264_videotoolbox @ 0x7f8b8901e400] Error: cannot create compression session: -12915
[h264_videotoolbox @ 0x7f8b8901e400] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
Output #0, matroska, to 'baseline_200kbps_qvga.mkv':
  Metadata:
    ENCODER         : Lavf57.41.100
    Stream #0:0, 0, 0/0: Unknown: none, SAR 1:1 DAR 0:0 (default)
    Metadata:
      DURATION        : 00:00:30.030000000
      encoder         : Lavc57.48.101 h264_videotoolbox
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_videotoolbox))
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[AVIOContext @ 0x7f8b88c183c0] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 0x7f8b88d002a0] Statistics: 1313186 bytes read, 0 seeks

[kthelgason]

I jumped the gun on this one. Turns out the issue is with qvga resolution. Not sure if this is a bug or a feature, in any case I will report it separately if it turns out to be a bug.