ffmpeg: Intel QSV and xstack_qsv: Error while filtering: Internal bug, should not have happened

[Shane Warren]

Summary of the bug:

I'm testing out an Intel Flex 140 card and I am attempting to transcode 4 inputs to 1 output using the xstack_qsv filter. My command takes in 4 udp multicast ts files (h264 encoded) and outputs one h264_qsv encoded udp transport stream.

The issue happens when one of the streams changes resolutions (from 1280x720 progressive, to 1920x1080 interlaced). I have a stream that changes resolutions available here: ​https://drive.google.com/file/d/1Mv4HUNCgY-_w9is8YMLb0SD8cs3NGbb_/view?usp=sharing

The stream will die with these errors if resolution changes:

[h264_qsv @ 0x55a240e68000] Error submitting the frame for encoding.
[vost#0:0/h264_qsv @ 0x55a240e30780] Error submitting video frame to the encoder
Error while filtering: Internal bug, should not have happened

How to reproduce:

% ffmpeg -nostats -loglevel verbose -init_hw_device qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1 -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@239.252.11.107:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.14:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.5:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -i "udp://@225.105.0.4:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" -noautoscale -filter_complex "\
[0:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v1]; \
[1:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v2]; \
[2:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v3]; \
[3:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v4]; \
[v1][v2][v3][v4] xstack_qsv=inputs=4:layout=0_0|0_h0|w0_0|w0_h0[mosiac]; \
[mosiac]vpp_qsv=w=1920:h=1080,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,hwupload=extra_hw_frames=64,format=qsv[v]"  -map "[v]" -map 0:a:0 -map 1:a:0 -map 2:a:0 -map 3:a:0 -c:v h264_qsv -noautoscale -async_depth 1 -scenario livestreaming -r:v 30000/1001 -b:v 6000k -minrate:v 6000k -maxrate:v 6000k -bufsize:v 12000k -threads 1 -profile:v high -bf:v 0 -g:v 15 -filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a 192k -vsync 1 -f mpegts -muxrate 7450599 -pes_payload_size 1528 "udp://@229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=7450599&burst_bits=10528"

ffmpeg version N-113112-gfb54c89a0d
Built on 12/27/2023 (pulled latest from git on the same day)
Built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04))

[Shane Warren]

This problem does not happen if this patch set is installed (​https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=10318), and the command line is changed to:

/tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1 -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -gpu_copy on -i "udp://@225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -gpu_copy on -i "udp://@225.105.0.14:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -gpu_copy on -i "udp://@225.105.0.5:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096 -async_depth 1 -gpu_copy on -i "udp://@225.105.0.4:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1" \
-noautoscale -filter_complex "\
[0:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=60000/1001[v1]; \
[1:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=60000/1001[v2]; \
[2:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=60000/1001[v3]; \
[3:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=60000/1001[v4]; \
[v1][v2][v3][v4] xstack_qsv=inputs=4:layout=0_0|0_h0|w0_0|w0_h0[mosiac]; \
[mosiac]vpp_qsv=w=1920:h=1080,pad=1920:1080:(ow-iw)/2:(oh-ih)/2[v]"  \
-map "[v]" -map 0:a:0 -map 1:a:0 -map 2:a:0 -map 3:a:0 \
-c:v h264_qsv -async_depth 1 -scenario livestreaming -r:v 60000/1001 -b:v 6000k -minrate:v 6000k -maxrate:v 6000k -bufsize:v 1500k -threads 1 -profile:v high  -bf:v 0 -g:v 15 \
-filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a 192k -fps_mode auto \
-f mpegts -muxrate 7450599 -pes_payload_size 1528 "udp://@225.105.0.83:10102?pkt_size=1316&fifo_size=90000&bitrate=7450599&burst_bits=10528"

Note: I removed the hwupload,hwdownload from my original commands, that patch seems to have negated the need for those commands.