h264_qsv decoder does not work with RTSP

[Steve Sampson]

Summary of the bug:
How to reproduce:

using ffmpeg version: ffmpeg version 5.0.1-full_build from (https://www.gyan.dev/ffmpeg/builds/) built on 2022-04-07. (Windows version)

ffmpeg.exe -init_hw_device qsv:hw,child_device_type=d3d11va -hwaccel qsv -c:v h264_qsv -hwaccel_output_format qsv -i rtsp://user:password@172.16.1.105/axis-media/media.amp -f null -

Running on Windows 10 with an Intel GPU (Intel UHD Graphics 620, but I suspect this problem isn't related to the exact hardware version - it also reproduced on another machine). The RTSP stream comes from a camera. It looks like it has a pixel format of 720x576, pix_fmt: yuvj420p. If there is a way to capture this stream such that someone else can reproduce the error, please let me know. I suspect the reproduction of this problem requires a genuine RTSP server.

When running, it initializes the connection to the RTSP server and then logs an error

Error initializing the MFX video decoder: invalid video parameters (-15)

The same command line successfully decodes using an mp4 file instead of an RTSP input (captured from the camera using ffmpeg -i rtsp://user:password@172.16.1.105/axis-media/media.amp -acodec copy -vcodec copy c:/abc.mp4)

The RTSP stream decodes successfully without any hardware acceleration. That is, with a command line of "ffmpeg.exe -i rtsp://user:password@172.16.1.105/axis-media/media.amp -f null -". It also is correctly displayed in VLC. I tried many different combinations of command line parameters (for example just with -hwaccel qsv -c:v h264_qsv) and none were successful.

My commentary/analysis:

Note that I am not at all an expert on FFMPEG. I was tasked to GPU-enable our existing code and this is my first serious look at FFMPEG.

The use case is to display the video from a set of cameras (all RTSP, all H264). Our code uses the FFMPEG API and doesn't use the FFMPEG program but my tests have been using FFMPEG.

The actual build I wish(ed) to use is an older build of ffmpeg (branched in 2019 and slightly modified by my team - but the decoding wasn't touched). It gives a different error string but it may be a similar root cause. I tried to debug it to figure out what was going wrong. The error in that version was "non-existing PPS 0 referenced" (from parse_nal_units) and then "Decoding pixel format '(null)' is not supported" (from ff_qsv_process_data).

I can't easily make my own build of FFMPEG (the process of building with Windows is too complex for me to quickly do) or I would attempt to debug using the latest code.

The SPD returned by the camera is the following

SDP:
v=0
o=- 1651685682795324 1651685682795324 IN IP4 172.16.1.105
s=Media Presentation
e=NONE
b=AS:50000
t=0 0
a=control:*
a=range:npt=0.000000-
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:50000
a=framerate:25.0
a=transform:0.937500,0.000000,0.000000;0.000000,1.000000,0.000000;0.000000,0.000000,1.000000
a=control:trackID=1
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; profile-level-id=420029; sprop-parameter-sets=Z0IAKeKQFoJNgScFAQXh4kRU,aM48gA==

[Steve Sampson]

I managed to find some other cameras to test with. On one, it gives a few errors (-15) and then seems to work. On another, it just seems to work. It probably has something to do with the RTSP or RTP data. I don't know exactly how to go about debugging this to find out if there is something wrong with the source stream or if it is a bug in the code. It seems strange that it works when not using qsv (and in VLC) but I realize it could be something slightly out of spec with the original stream.

[Steve Sampson]

I tested with 5.1.2 and it still doesn't work, though this time FFMPEG just seems to wait for something that never comes.

[h264_qsv @ 00000251231a3480] More data is required to decode header

Last message repeated 154 times

Further investigation shows that there is a setting on the Axis camera "PS Enabled". When checked, FFMPEG works correctly and when unchecked it does not.

According to one of our customer's experts on video, this setting adds SPS and PPS packets to the RTP streams (NALU type 7 and 8). This information ought to be read from the SPD but it appears it isn't when using the QSV code.

[Carl Eugen Hoyos]

Please test current FFmpeg git head and provide the command line you tested together with the complete, uncut console output to make this a valid ticket.

[Steve Sampson]

I do not have the ability to build from head, but I did use a recent version.

The output (including command line) is the following

ffmpeg -hwaccel qsv -c:v h264_qsv -i rtsp://root:pass@10.255.200.235/axis-media/media.amp?streamprofile=rthd -f null -
ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers

built with gcc 12.1.0 (Rev2, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100

WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv".
Input #0, rtsp, from 'rtsp://root:pass@10.255.200.235/axis-media/media.amp?streamprofile=rthd':

Metadata:

title : Session streamed with GStreamer
comment : rtsp-server

Duration: N/A, start: 0.040022, bitrate: N/A
Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1920x1080, 25 fps, 25 tbr, 90k tbn

WARNING: defaulting child_device_type to AV_HWDEVICE_TYPE_DXVA2 for compatibility with old commandlines. This behaviour will be removed in the future. Please explicitly set device type via "-init_hw_device" option.
Stream mapping:

Stream #0:0 -> #0:0 (h264 (h264_qsv) -> wrapped_avframe (native))

Press [q] to stop, ? for help
[h264_qsv @ 000001b1753e5040] More data is required to decode header

Last message repeated 791 times

[h264_qsv @ 000001b1753e5040] video_get_buffer: image parameters invalid
[h264_qsv @ 000001b1753e5040] get_buffer() failed
Error while decoding stream #0:0: Invalid argument
Finishing stream 0:0 without any data written to it.
Output #0, null, to 'pipe:':

Metadata:

title : Session streamed with GStreamer
comment : rtsp-server
encoder : Lavf59.27.100

Stream #0:0: Video: wrapped_avframe, yuvj420p, 1920x1080, q=2-31, 200 kb/s, 25 fps, 25 tbn

Metadata:

encoder : Lavc59.37.100 wrapped_avframe

frame= 0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed= 0x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Exiting normally, received signal 2.

[Steve Sampson]

The output when PS Enabled (when it succeeds - same command line)

ffmpeg -hwaccel qsv -c:v h264_qsv -i rtsp://root:pass@10.255.200.235/axis-media/media.amp?streamprofile=rthd -f null -
ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers

built with gcc 12.1.0 (Rev2, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100

WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv".
Input #0, rtsp, from 'rtsp://root:pass@10.255.200.235/axis-media/media.amp?streamprofile=rthd':

Metadata:

title : Session streamed with GStreamer
comment : rtsp-server

Duration: N/A, start: 0.040022, bitrate: N/A
Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1920x1080, 25 fps, 25 tbr, 90k tbn

WARNING: defaulting child_device_type to AV_HWDEVICE_TYPE_DXVA2 for compatibility with old commandlines. This behaviour will be removed in the future. Please explicitly set device type via "-init_hw_device" option.
Stream mapping:

Stream #0:0 -> #0:0 (h264 (h264_qsv) -> wrapped_avframe (native))

Press [q] to stop, ? for help
Output #0, null, to 'pipe:':

Metadata:

title : Session streamed with GStreamer
comment : rtsp-server
encoder : Lavf59.27.100

Stream #0:0: Video: wrapped_avframe, qsv(pc, bt709, progressive), 1920x1080, q=2-31, 200 kb/s, 25 fps, 25 tbn

Metadata:

encoder : Lavc59.37.100 wrapped_avframe

frame= 200 fps= 29 q=-0.0 Lsize=N/A time=00:00:08.24 bitrate=N/A speed= 1.2x
video:92kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.

[Steve Sampson]

The only difference in the two runs is that PS was enabled on the camera. Everything else remains exactly the same. This is running under Windows. If needed, I can re-run with addition logging (please indicate the command line that will give more logs)

In the case where it fails (PS not enabled), the output basically repeats

[h264_qsv @ 000001b1753e5040] More data is required to decode header

Last message repeated 791 times

Until I hit Ctrl-C. The error message afterwards happens after Ctrl-C.

[Steve Sampson]

I just tried the build 2022-10-06-git-2c2aaa5bd0 downloaded from ​https://www.gyan.dev/ffmpeg/builds/

I see exactly the same behavior as seen in 5.1.2.

The camera model is AXIS P3265-LVE Dome Camera - 9mm. It has been updated with its latest firmware.

It successfully decodes when QSV is not specified. It successfully decodes when PS Enabled = true. It fails when using QSV and PS Enabled = false.

Note: I can get further information if it would be helpful (network trace, more detailed logs, ...)

ffmpeg -i rtsp://root:pass@10.255.200.235/axis-media/media.amp?streamprofile=rthd -f null

[Steve Sampson]

Log of latest FFMPEG with PS disabled

[Steve Sampson]

Log of latest FFMPEG with PS enabled

[Steve Sampson]

I just attached two logs with logging set to trace. The camera, computer, software and command line are all identical except the "PS Enabled" setting. When enabled, the decoding works. When disabled, decoding doesn't. I hope this helps narrow down the problem.

[rtsp qsv]

Behavior still present on ffmpeg version 4.4.2-0ubuntu0.22.04.1
Decoding rtsp stream fails with qsv_h264, works with h264 (native)

[rtsp qsv]

Error still present on:
ffmpeg 5.1.2-3ubuntu1
ffmpeg 6.0-6ubuntu1
ffmpeg 6.1-4ubuntu1

[wenbin,chen]

Tried to reproduce the error.

1. set up video server: using mediamtx : ​https://github.com/bluenviron/mediamtx
2. push stream: ffmpeg -v verbose -re -stream_loop -1 -i input.mp4 -c:v hevc_qsv -bsf:v extract_extradata=remove=1 -f rtsp rtsp://localhost:8554/mystream
3. ffmpeg -v verbose -f rtsp -i rtsp://localhost:8554/mystream -f null - success
4. ffmpeg -v verbose -hwaccel vaapi -f rtsp -i rtsp://localhost:8554/mystream -f null - success after skip many frames (it relates to gop size)
5. ffmpeg -v verbose -hwaccel qsv -f rtsp -i rtsp://localhost:8554/mystream -f null - fail ( error message

[hevc_qsv @ 0x559d42d69dd0] More data is required to decode header
    Last message repeated 3850 times

)

"remove=1" means remove sps and pps from bitstream after extracting data. If i set remove=0 when push stream, qsv can successfully decode video stream.
VPL may not be able to deal with stream without SPS and PPS, so it Keeps waiting for more data to decode header. I cannot reproduce the same error as reporter's.

I suggest to use vaapi/d3d11va to decode video stream without SPS/PPS header as they use ffmpeg build-in parser to parse stream.

[wenbin,chen]

According to ​https://spec.oneapi.io/onevpl/latest/API_ref/VPL_func_vid_decode.html#_CPPv431MFXVideoDECODE_DecodeFrameAsync10mfxSessionP12mfxBitstreamP16mfxFrameSurface1PP16mfxFrameSurface1P12mfxSyncPoint

“Before decoding the first frame, a sequence header (sequence parameter set in H.264 or sequence header in MPEG-2 and VC-1) must be present. The function skips any bitstreams before it encounters the new sequence header”

[wenbin,chen]

After commit ae06111d7406e541ac232318bcf50f8510a36f38. bsf can be used after demuxing.

Add dump_extra bsf and pull stream like this:

 ffmpeg -v verbose -bsf:v dump_extra -hwaccel qsv -i rtsp://localhost:8554/mystream -f null -

solve the problem on my side.