x11grab fails to adjust to framebuffer resolution changes

[Mykola Mokhnach]

Summary of the bug:

OS: Linux vm 5.11.0-22-generic #23-Ubuntu SMP Thu Jun 17 00:34:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

ffmpeg version 4.3.2-0+deb11u1ubuntu1 Copyright (c) 2000-2021 the FFmpeg developers

built with gcc 10 (Ubuntu 10.2.1-20ubuntu1)
configuration: --prefix=/usr --extra-version=0+deb11u1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100

Hyper fast Audio and Video encoder

How to reproduce:

1. Install Xvfb and x11vnc on Ubuntu
2. Configure the virtual framebuffer:

% Xvfb :95 -screen 0 1920x1080x24
% x11vnc -display :95 -forever -shared

3. Add more supported resolutions to the framebuffer:

% xrandr -d :95 --newmode "1024x768" 63.50 1024 1072 1176 1328 768 771 775 798 -hsync +vsync
% xrandr -d :95 --addmode screen "1024x768"
% xrandr -d :95 --newmode "1440x900" 106.50 1440 1528 1672 1904 900 903 909 934 -hsync +vsync
% xrandr -d :95 --addmode screen "1440x900"

4. Start ffmpeg recording on the created display:

% ffmpeg -y -f 11grab -framerate 4 -r 4 -i :95.0 -filter_complex "[0:v]split=2[ts][capture];[ts]setpts=time(0)*FR[ts];[capture]scale=1280:960:force_original_aspect_ratio=decrease:eval=frame,pad=1280:960:-1:-1:color=black[capture]" -map "[ts]" -vsync 0 -flush_packets 1 -f 'mkvtimestamp_v2 output_ts.txt -map "[capture]" -c:v libx264 -tune zerolatency -preset veryfast -fflags nobuffer -pix_fmt yuv420p -movflags +faststart -f mp4 output.mp4

5. Change the display resolution while the recording is running:

% xrandr -d :95 -s "1024x768"

Actual result:
ffmpeg quits recording. the following output is observed in the log:
[x11grab @ 0x55bc5ca9c980] Cannot get the image data event_error: response_type:0 error_code:8 sequence:130 resource_id:23068672 minor_code:4 major_code:130.
[x11grab @ 0x55bc5ca9c980] Continuing without shared memory.
[x11grab @ 0x55bc5ca9c980] Cannot get the image data event_error: response_type:0 error_code:8 sequence:131 resource_id:1294 minor_code:0 major_code:73.
:95.0: Permission denied

If I set framebuffer resolution to a greater value than the inital one then the issue above is not being observed, although the video content does not get scaled properly and is cut off to the initial dimensions

Expected result:
The video must be properly scaled/padded after the resolution change, so the content of the full screen is always visible there.