zmq doesn't work in this filter chain

[Michael Koch]

This command line works as expected:

ffmpeg -f lavfi -i testsrc2=s=1920x1080 -f lavfi -i color=gray:s=960x540 -lavfi [0]zmq,scale@my=480:-1,chromakey@my=0x0000ff:0.35[FG];[1][FG]overlay,format=rgb24 -f sdl -

You can send zmq messages to the running ffmpeg process:
echo scale@my w 240 | zmqsend
or
echo scale@my w 480 | zmqsend
and you see that the size of the foreground video changes.
So far, everything is working as expected.

Now comes the problem. The only difference is that a "eq" filter (without any options) is inserted before the "chromakey" filter.

ffmpeg -f lavfi -i testsrc2=s=1920x1080 -f lavfi -i color=gray:s=960x540 -lavfi [0]zmq,scale@my=480:-1,eq,chromakey@my=0x0000ff:0.35[FG];[1][FG]overlay,format=rgb24 -f sdl -

When you now send the same messages as before, the size of the foreground video doesn't change.
Same problem also if the messages aren't sent by the "zmqsend" tool, but instead by a C# program with "NetMQ" library.

Below is the console output for the not-working case:

C:\Users\astro\Desktop\FFbluescreen>ffmpeg -f lavfi -i testsrc2=s=1920x1080 -f lavfi -i color=gray:s=960x540 -lavfi [0]zmq,scale@my=480:-1,eq,chromakey@my=0x0000ff:0.35[FG];[1][FG]overlay,format=rgb24 -f sdl -
ffmpeg version 2020-11-11-git-89429cf2f2-full_build-www.gyan.dev Copyright (c) 2000-2020 the FFmpeg developers

built with gcc 10.2.0 (Rev3, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 56. 60.100 / 56. 60.100
libavcodec 58.112.101 / 58.112.101
libavformat 58. 64.100 / 58. 64.100
libavdevice 58. 11.102 / 58. 11.102
libavfilter 7. 89.100 / 7. 89.100
libswscale 5. 8.100 / 5. 8.100
libswresample 3. 8.100 / 3. 8.100
libpostproc 55. 8.100 / 55. 8.100

Input #0, lavfi, from 'testsrc2=s=1920x1080':

Duration: N/A, start: 0.000000, bitrate: N/A

Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc

Input #1, lavfi, from 'color=gray:s=960x540':

Duration: N/A, start: 0.000000, bitrate: N/A

Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 960x540 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc

Stream mapping:

Stream #0:0 (rawvideo) -> zmq
Stream #1:0 (rawvideo) -> overlay:main
format -> Stream #0:0 (rawvideo)

Press [q] to stop, ? for help
Output #0, sdl,sdl2, to 'pipe:':

Metadata:

encoder : Lavf58.64.100
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24(pc, progressive), 960x540 [SAR 1:1 DAR 16:9], q=2-31, 311040 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:

encoder : Lavc58.112.101 rawvideo

f

[Michael Koch]

Another example that doesn't work: Use my first (working) example and replace "chromakey" by "colorkey".

[Elon Musk]

Is this really full log? Looks like send cmd is never sent, so no wonder why it does not work.

[Michael Koch]

Replying to richardpl:

Is this really full log? Looks like send cmd is never sent, so no wonder why it does not work.

Yes, I think this is the full log. When you send the message via zmqsend, this must of course be done from another console window, or from a batch file. I'm sure that the message is sent correctly. When I send the message from my C# program with NetMQ library, I get no error message, which means the message was sent and an answer was also received.
You can also see that something changes in the output video when the message is sent. The resolution of the foreground video changes. But its size remains unchanged.

[Michael Koch]

If I insert a combination of "copy" and "eq" filters before the chromakey filter, then FFmpeg exits with an error message when the message is sent:

C:\Users\astro\Desktop\FFbluescreen>ffmpeg -f lavfi -i testsrc2=s=1920x1080 -f lavfi -i color=gray:s=960x540 -lavfi [0]zmq,scale@my=480:-1,copy,eq,chromakey@my=0x0000ff:0.35[FG];[1][FG]overlay,format=rgb24 -f sdl -
ffmpeg version 2020-11-11-git-89429cf2f2-full_build-www.gyan.dev Copyright (c) 2000-2020 the FFmpeg developers

built with gcc 10.2.0 (Rev3, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 56. 60.100 / 56. 60.100
libavcodec 58.112.101 / 58.112.101
libavformat 58. 64.100 / 58. 64.100
libavdevice 58. 11.102 / 58. 11.102
libavfilter 7. 89.100 / 7. 89.100
libswscale 5. 8.100 / 5. 8.100
libswresample 3. 8.100 / 3. 8.100
libpostproc 55. 8.100 / 55. 8.100

Input #0, lavfi, from 'testsrc2=s=1920x1080':

Duration: N/A, start: 0.000000, bitrate: N/A

Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc

Input #1, lavfi, from 'color=gray:s=960x540':

Duration: N/A, start: 0.000000, bitrate: N/A

Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 960x540 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc

Stream mapping:

Stream #0:0 (rawvideo) -> zmq
Stream #1:0 (rawvideo) -> overlay:main
format -> Stream #0:0 (rawvideo)

Press [q] to stop, ? for help
Output #0, sdl,sdl2, to 'pipe:':

Metadata:

encoder : Lavf58.64.100
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24(pc, progressive), 960x540 [SAR 1:1 DAR 16:9], q=2-31, 311040 kb/s, 25 fps, 25 tbn, 25 tbc (default)
Metadata:

encoder : Lavc58.112.101 rawvideo

Assertion ((src_linesize) >= 0 ? (src_linesize) : (-(src_linesize))) >= bytewidth failed at C:/code/ffmpeg/src/libavutil/imgutils.c:349

[Elon Musk]

Sorry, but changing dimensions of video midstream is fully unsupported. It may work with some filters but with most filters it does not work.

[Michael Koch]

Replying to richardpl:

Sorry, but changing dimensions of video midstream is fully unsupported. It may work with some filters but with most filters it does not work.

Do you mean it's unsupported for all streams? Also for internal streams between filters? In my example the output stream (which is sent to the sdl device) has of course a constant size.

[Michael Koch]

ok, if mid-stream changing of size is unsupported, then this ticket makes no sense and can be closed. But it would be helpful to add a remark to the documentation of the zmq filter:

It's possible to send zmq messages to all options of all filters. However, not everything that's possible is also useful. For example, a zmq message can be sent to the "scale" filter to change the size of a video stream. But mid-stream changing of size isn't supported by many other filters. In some cases this may work, but in most cases it fails.

[Michael Koch]

After I've worked with zmq for about a week, I suggest to add the following revised note to the documentation of the zmq filter:

It's possible to send zmq messages to all those filters and options that support commands. Don't send messages to other options, as this may lead to malfunction.
However, not everything that's possible is also useful. For example, a message can be sent to the "width" option of the "scale" filter for changing the size of a video stream. But changing the size mid-stream isn't supported by many other filters (for example "eq", "colorkey" and "despill"). In some cases it works (for example the output of "scale" can be used as the second input of "overlay"), but in most other cases it fails.