using hardware encoding and decoding, change the resolution of videos, and merge them using concat

[elias tsolis]

Two videos with different resolution, framerates, and the rest, i would like to join them together by making framerates =25 , and resolution 1920x1080.

I want to use encoding and decoding hardware acceleration.
However, it fails.
Impossible to convert between the formats supported by the filter 'Parsed_scale_vaapi_2' and the filter 'auto_scale_1'

`
ffmpeg version 6.1.1-1 Copyright (c) 2000-2023 the FFmpeg developers

built with gcc 13 (Debian 13.2.0-9)
configuration: --prefix=/usr --extra-version=1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libharfbuzz --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libsvtav1 --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x564257deb200] Auto-inserting h264_mp4toannexb bitstream filter
Input #0, concat, from './vids.txt':

Duration: N/A, start: 0.000000, bitrate: 16895 kb/s
Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080, 16767 kb/s, 29.39 fps, 29.58 tbr, 90k tbn

Metadata:

handler_name : VideoHandle
vendor_id : [0][0][0][0]

Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 128 kb/s

Metadata:

handler_name : SoundHandle
vendor_id : [0][0][0][0]

File 'out_video.mp4' already exists. Overwrite? [y/N] y
Stream mapping:

Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_vaapi))
Stream #0:1 -> #0:1 (aac (native) -> aac (native))

Press [q] to stop, ? for help
[hevc_vaapi @ 0x564257d3e8c0] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Output #0, mp4, to 'out_video.mp4':

Metadata:

encoder : Lavf60.16.100

Stream #0:0(eng): Video: hevc (Main) (hev1 / 0x31766568), vaapi(tv, bt709, progressive), 1920x1080, q=2-31, 25 fps, 12800 tbn

Metadata:

handler_name : VideoHandle
vendor_id : [0][0][0][0]
encoder : Lavc60.31.102 hevc_vaapi

Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 69 kb/s

Metadata:

handler_name : SoundHandle
vendor_id : [0][0][0][0]
encoder : Lavc60.31.102 aac

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f113c00aa40] Auto-inserting h264_mp4toannexb bitstream filterrop=8758 speed=6.01x
[in#0/concat @ 0x564257de9a80] New (null) stream with index 2 at pos:48 and DTS:1996.03s
Impossible to convert between the formats supported by the filter 'Parsed_scale_vaapi_2' and the filter 'auto_scale_1'
[vf#0:0 @ 0x564257d3a340] Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while filtering: Function not implemented
[out#0/mp4 @ 0x564257d45640] video:5894436kB audio:17237kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.025196%
frame=49900 fps=150 q=-0.0 Lsize= 5913162kB time=00:33:16.65 bitrate=24260.9kbits/s dup=11 drop=8770 speed= 6x
[aac @ 0x564258c2ddc0] Qavg: 43.147
Conversion failed!
`

ffmpeg -hwaccel vaapi -hwaccel_output_format yuv420p -vaapi_device /dev/dri/renderD128 -f concat -safe 0 -i './vids.txt' -vf 'format=nv12,hwupload,scale_vaapi=1920:1080' -c:v hevc_vaapi -r 25 -rc_mode CQP -global_quality 22 -threads 12 out_video.mp4

[Player701]

I think I'm having a similar issue with overlay, also using hardware decoding and encoding.

The following command line will work if both streams have the same resolution, otherwise it fails:

ffmpeg -loglevel verbose \
 -hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream1 \
 -hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream2 \
 -filter_complex "[0:v][1:v]overlay_vaapi=w=500" \
 -c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an -f null -

ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14 (SUSE Linux)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --optflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --enable-version3 --enable-libsmbclient --disable-openssl --enable-gnutls --enable-ladspa --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libjack --enable-libjxl --enable-librist --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopenh264-dlopen --enable-libopus --enable-libpulse --enable-librav1e --enable-librubberband --enable-libsvtav1 --enable-libsoxr --enable-libspeex --enable-libssh --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lto --enable-lv2 --enable-libvpl --enable-vaapi --enable-vdpau --enable-version3 --enable-libfdk-aac-dlopen --enable-nonfree --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libx264 --enable-libx265 --enable-libxvid
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
[tcp @ 0x555f7fb057c0] Starting connection attempt to ::1 port 1935
[tcp @ 0x555f7fb057c0] Connection attempt to ::1 port 1935 failed: Connection refused
[tcp @ 0x555f7fb057c0] Starting connection attempt to 127.0.0.1 port 1935
[tcp @ 0x555f7fb057c0] Successfully connected to 127.0.0.1 port 1935
[h264 @ 0x555f7fb50480] Reinit context to 2688x1520, pix_fmt: yuvj420p
Selecting decoder 'h264' because of requested hwaccel method vaapi
Input #0, flv, from 'rtmp://localhost/stream1':
  Metadata:
    |RtmpSampleAccess: true
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 2688
    displayHeight   : 1520
    fps             : 26
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 17194.931000, bitrate: N/A
  Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
  Stream #0:1: Video: h264 (High), 1 reference frame, yuvj420p(pc, progressive, left), 2688x1520 [SAR 1:1 DAR 168:95], 26 fps, 26 tbr, 1k tbn
[tcp @ 0x555f7fb13740] Starting connection attempt to ::1 port 1935
[tcp @ 0x555f7fb13740] Connection attempt to ::1 port 1935 failed: Connection refused
[tcp @ 0x555f7fb13740] Starting connection attempt to 127.0.0.1 port 1935
[tcp @ 0x555f7fb13740] Successfully connected to 127.0.0.1 port 1935
[h264 @ 0x555f7fba4fc0] Reinit context to 2560x1440, pix_fmt: yuvj420p
Selecting decoder 'h264' because of requested hwaccel method vaapi
Input #1, flv, from 'rtmp://localhost/stream2':
  Metadata:
    |RtmpSampleAccess: true
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 2560
    displayHeight   : 1440
    fps             : 25
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 17200.176000, bitrate: N/A
  Stream #1:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
  Stream #1:1: Video: h264 (Main), 1 reference frame, yuvj420p(pc, bt709, progressive, left), 2560x1440, 25 fps, 25 tbr, 1k tbn
[out#0/null @ 0x555f7fb08100] Creating output stream from unlabeled output of complex filtergraph 0. This overrides automatic video mapping.
[vost#0:0/h264_vaapi @ 0x555f7fb29740] Created video stream from complex filtergraph 0:[overlay_vaapi:default]
[vost#0:0/h264_vaapi @ 0x555f7fb29740] 
[out#0/null @ 0x555f7fb08100] No explicit maps, mapping streams automatically...
[fc#0 @ 0x555f7fb02f40] Binding input with label '0:v' to input stream 0:1
[AVHWDeviceContext @ 0x555f811e3e00] Trying to use DRM render node for device 0.
[AVHWDeviceContext @ 0x555f811e3e00] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0x555f811e3e00] libva: Trying to open /usr/lib64/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x555f811e3e00] libva: Found init function __vaDriverInit_1_22
[AVHWDeviceContext @ 0x555f811e3e00] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x555f811e3e00] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext @ 0x555f811e3e00] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 25.1.1 ().
[AVHWDeviceContext @ 0x555f811e3e00] Driver not found in known nonstandard list, using standard behaviour.
[fc#0 @ 0x555f7fb02f40] Binding input with label '1:v' to input stream 1:1
Stream mapping:
  Stream #0:1 (h264) -> overlay_vaapi
  Stream #1:1 (h264) -> overlay_vaapi
  overlay_vaapi:default -> Stream #0:0 (h264_vaapi)
[vost#0:0/h264_vaapi @ 0x555f7fb29740] Starting thread...
[fc#0 @ 0x555f7fb02f40] Starting thread...
[vist#0:1/h264 @ 0x555f7fba3c40] [dec:h264 @ 0x555f7fb417c0] Starting thread...
[vist#1:1/h264 @ 0x555f7fb12ac0] [dec:h264 @ 0x555f7fb40180] Starting thread...
[in#0/flv @ 0x555f7fb04a40] Starting thread...
[in#1/flv @ 0x555f7fb16140] Starting thread...
Press [q] to stop, [?] for help
[h264 @ 0x555f812ca640] Reinit context to 2688x1520, pix_fmt: vaapi
[h264 @ 0x555f810132c0] Reinit context to 2560x1440, pix_fmt: vaapi
[graph 0 input from stream 0:1 @ 0x7f04f4005a80] w:2688 h:1520 pixfmt:vaapi tb:1/1000 fr:26/1 sar:1/1 csp:unknown range:pc
[graph 0 input from stream 1:1 @ 0x7f04f4006540] w:2560 h:1440 pixfmt:vaapi tb:1/1000 fr:25/1 sar:0/1 csp:bt709 range:pc
[auto_scale_0 @ 0x7f04f4008200] w:iw h:ih flags:'' interl:0
[Parsed_overlay_vaapi_0 @ 0x7f04f4005440] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 1:1' and the filter 'Parsed_overlay_vaapi_0'
Impossible to convert between the formats supported by the filter 'graph 0 input from stream 1:1' and the filter 'auto_scale_0'
[fc#0 @ 0x555f7fb02f40] Error reinitializing filters!
[fc#0 @ 0x555f7fb02f40] Task finished with error code: -38 (Function not implemented)
[fc#0 @ 0x555f7fb02f40] Terminating thread with return code -38 (Function not implemented)
[vost#0:0/h264_vaapi @ 0x555f7fb29740] Encoder thread received EOF
[vost#0:0/h264_vaapi @ 0x555f7fb29740] Could not open encoder before EOF
[vost#0:0/h264_vaapi @ 0x555f7fb29740] Task finished with error code: -22 (Invalid argument)
[vost#0:0/h264_vaapi @ 0x555f7fb29740] Terminating thread with return code -22 (Invalid argument)
[vist#0:1/h264 @ 0x555f7fba3c40] [dec:h264 @ 0x555f7fb417c0] Decoder returned EOF, finishing
[vist#0:1/h264 @ 0x555f7fba3c40] [dec:h264 @ 0x555f7fb417c0] Terminating thread with return code 0 (success)
[vist#0:1/h264 @ 0x555f7fba3c40] All consumers of this stream are done
[in#0/flv @ 0x555f7fb04a40] All consumers are done
[in#0/flv @ 0x555f7fb04a40] Terminating thread with return code 0 (success)
[vist#1:1/h264 @ 0x555f7fb12ac0] [dec:h264 @ 0x555f7fb40180] Decoder returned EOF, finishing
[vist#1:1/h264 @ 0x555f7fb12ac0] [dec:h264 @ 0x555f7fb40180] Terminating thread with return code 0 (success)
[vist#1:1/h264 @ 0x555f7fb12ac0] All consumers of this stream are done
[in#1/flv @ 0x555f7fb16140] All consumers are done
[in#1/flv @ 0x555f7fb16140] Terminating thread with return code 0 (success)
[out#0/null @ 0x555f7fb08100] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A    
[in#0/flv @ 0x555f7fb04a40] Input file #0 (rtmp://localhost/stream1):
[in#0/flv @ 0x555f7fb04a40]   Input stream #0:1 (video): 36 packets read (680287 bytes); 12 frames decoded; 0 decode errors; 
[in#0/flv @ 0x555f7fb04a40]   Total: 36 packets (680287 bytes) demuxed
[AVIOContext @ 0x555f7fb185c0] Statistics: 804302 bytes read, 0 seeks
[in#1/flv @ 0x555f7fb16140] Input file #1 (rtmp://localhost/stream2):
[in#1/flv @ 0x555f7fb16140]   Input stream #1:1 (video): 26 packets read (1154248 bytes); 2 frames decoded; 0 decode errors; 
[in#1/flv @ 0x555f7fb16140]   Total: 26 packets (1154248 bytes) demuxed
[AVIOContext @ 0x555f7fba2a80] Statistics: 1785858 bytes read, 0 seeks
Conversion failed!

It looks like FFmpeg tries to insert a software scaling filter, which obviously fails because there are only hardware frames to work with.

As of now, I have to resort to software decoding because of this issue. I can still use hardware encoding with hwupload:

ffmpeg -init_hw_device vaapi=hw -filter_hw_device hw \
 -i rtmp://localhost/stream1 \
 -i rtmp://localhost/stream2 \
 -filter_complex "[0:v]format=nv12,hwupload=extra_hw_frames=120[i0];[1:v]format=nv12,hwupload=extra_hw_frames=120[i1];[i0][i1]overlay_vaapi=w=500" \
 -c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an -f null -