VideoToolbox H264 RTSP stream: empty image buffer on iOS (error -12909)

[damiaan]

Summary of the bug:
When playing a H264 encoded RTSP stream on iOS 15.4.1 with AV_PIX_FMT_VIDEOTOOLBOX, the stream fails to play. The Videotoolbox code path results in a OSStatus -12909 passed into the videotoolbox_decoder_callback. The image_buffer argument is also NULL.

The issue occurs with both FFMPEG version 5.0.1 and a version built from the master branch, commit hash ​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/c6364b711bad1fe2fbd90e5b2798f87080ddf5ea.

I have also tested the same RTSP stream against ffmpeg running on MacOS 12.3.1 FFMPEG version 4.0 fails with the following error log, repeated for each frame:

cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 0x7fb517816e00] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517816e00] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517816e00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517816e00] nal_unit_type: 1, nal_ref_idc: 1
[h264 @ 0x7fb517816e00] ct_type:0 pic_struct:1
[h264 @ 0x7fb517816e00] VideoToolbox session not available.
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fb517816e00] decode_slice_header error
[h264 @ 0x7fb517816e00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 0x7fb517817400] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517817400] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517817400] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517817400] nal_unit_type: 1, nal_ref_idc: 1
[h264 @ 0x7fb517817400] ct_type:0 pic_struct:2
[h264 @ 0x7fb517817400] VideoToolbox session not available.
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fb517817400] decode_slice_header error
[h264 @ 0x7fb517817400] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 0x7fb517817a00] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517817a00] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517817a00] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517817a00] nal_unit_type: 1, nal_ref_idc: 0
[h264 @ 0x7fb517817a00] ct_type:0 pic_struct:1
[h264 @ 0x7fb517817a00] VideoToolbox session not available.
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fb517817a00] decode_slice_header error
[h264 @ 0x7fb517817a00] no frame!
Error while decoding stream #0:0: Invalid data found when processing input
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 0x7fb517818000] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x7fb517818000] nal_unit_type: 8, nal_ref_idc: 1
[h264 @ 0x7fb517818000] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x7fb517818000] nal_unit_type: 1, nal_ref_idc: 0
[h264 @ 0x7fb517818000] ct_type:0 pic_struct:2
[h264 @ 0x7fb517818000] VideoToolbox session not available.

FFMPEG 5.0.1 on MacOS succeeds with the following log (trimmed down to a couple of frames):

[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:1
[h264 @ 0x7fe09bf61600] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf6b580] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf6b580] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf6b580] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf6b580] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf75500] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf75500] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf75500] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf75500] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:1
[h264 @ 0x7fe09bf75500] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf7f480] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf7f480] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf7f480] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf7f480] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[NULL @ 0x7fe0bc904e40] ct_type:0 pic_struct:2
[h264 @ 0x7fe09bf89400] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7fe09bf89400] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x7fe09bf89400] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fe09bf89400] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1

The MacOS command I use to run FFMPEG is as follows:

ffmpeg -benchmark -hwaccel videotoolbox -i ~/Desktop/test.mp4 -f null -loglevel debug -

The iOS debug log is as follows:

[rtsp @ 0x159fdf0d0] SDP:
v=0

s=udp://239.192.3.33:5000

c=IN IP4 10.130.142.80

m=video 0 RTP/AVP 96

a=control:trackID=0

a=rtpmap:96 H264/90000

a=fmtp:96 packetization-mode=1


[rtsp @ 0x159fdf0d0] video codec set to: h264
[rtsp @ 0x159fdf0d0] RTP Packetization Mode: 1
[rtsp @ 0x159fdf0d0] setting jitter buffer size to 0
[rtsp @ 0x159fdf0d0] hello state=0
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15b6579b0] nal_unit_type: 7(SPS), nal_ref_idc: 1
[h264 @ 0x15b6579b0] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15b6579b0] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15b6579b0] nal_unit_type: 5(IDR), nal_ref_idc: 1
[h264 @ 0x15b6579b0] Format yuv420p chosen by get_format().
[h264 @ 0x15b6579b0] Reinit context to 720x480, pix_fmt: yuv420p
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[rtsp @ 0x159fdf0d0] first_dts 0 not matching first dts NOPTS (pts NOPTS, duration 1501) in the queue
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:2
[h264 @ 0x15b6579b0] ct_type:0 pic_struct:1
[rtsp @ 0x159fdf0d0] All info found
[rtsp @ 0x159fdf0d0] rfps: 59.000000 0.009195
[rtsp @ 0x159fdf0d0] rfps: 59.000000 0.009195
[rtsp @ 0x159fdf0d0] rfps: 60.000000 0.000000
[rtsp @ 0x159fdf0d0] rfps: 120.000000 0.000000
[rtsp @ 0x159fdf0d0] rfps: 240.000000 0.000001
[rtsp @ 0x159fdf0d0] rfps: 59.940060 0.000031
[rtsp @ 0x159fdf0d0] rfps: 59.940060 0.000031
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 7(SPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 5(IDR), nal_ref_idc: 1
[h264 @ 0x15c660c40] Format videotoolbox_vld chosen by get_format().
[h264 @ 0x15c660c40] Format videotoolbox_vld requires hwaccel initialisation.
[h264 @ 0x15c660c40] Reinit context to 720x480, pix_fmt: videotoolbox_vld
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[h264 @ 0x15c660c40] ct_type:0 pic_struct:2
[h264 @ 0x15c660c40] no picture 
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[h264 @ 0x15c660c40] ct_type:0 pic_struct:2
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture
[h264 @ 0x15c660c40] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 8(PPS), nal_ref_idc: 1
[h264 @ 0x15c660c40] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x15c660c40] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[h264 @ 0x15c660c40] VideoToolbox decoder needs reconfig, restarting..
[h264 @ 0x15c660c40] vt decoder cb: output image buffer is null
[h264 @ 0x15c660c40] hardware accelerator failed to decode picture

This log is from FFMPEG 5.0.1, not from the master commit referenced above. The last few commits add extra logging, so a log from that FFPMEG build will list the error code -12909 along with the line "vt decoder cb: output image buffer is null".

No particular FFMPEG options are used on iOS in the AVDictionary options dictionary, except for forcing the stream to be transported over TCP.

I have captured the raw RTSP stream into a small (1MB) MP4 file that is attached to this ticket. I am very confused as to why the issue no longer occurs in FFMPEG 5.0.1 running on MacOS 12.3.1, but it does occur in FFMPEG 5.0.1 running on iOS 15.4.1 linked against iOS 14.0 SDK.

If there is any more information I can provide please let me know. I am happy to contribute from a technical perspective to help solve this issue.