Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#8450 closed defect (invalid)

x11grab doesn't work with headless X servers

Reported by: eero-t Owned by:
Priority: normal Component: avdevice
Version: git-master Keywords:
Cc: Marton Balint Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:

FFmpeg x11grab doesn't work with headless X server configurations.

How to reproduce:

  1. Start headless X server:
    Xvfb :99 -screen 0 1920x1080x24 -nolisten tcp
    
  2. Try streaming that with FFmepg:
    $ ffmpeg  -an -f x11grab -video_size 1920x1080 -framerate 2 -i :99 -c:v libx264 -tune zerolatency -pix_fmt nv12 -b:v 5000k -f flv rtmp://hostname/live/desktop
    ffmpeg version 4.1.4-1build2 Copyright (c) 2000-2019 the FFmpeg developers
      built with gcc 9 (Ubuntu 9.2.1-4ubuntu1)
    ...
    [x11grab @ 0x5652189c7b80] Cannot get the image data event_error: response_type:0 error_code:128 sequence:9 resource_id:4194304 minor_code:4 major_code:130.
    [x11grab @ 0x5652189c7b80] Stream #0: not enough frames to estimate rate; consider increasing probesize
    Input #0, x11grab, from ':99':
      Duration: N/A, bitrate: N/A
        Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 2 fps, 1000k tbr, 1000k tbn, 1000k tbc
    Stream mapping:
      Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
    ...
    Output #0, flv, to 'rtmp://192.168.0.8/live/desktop':
      Metadata:
        encoder         : Lavf58.20.100
        Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), nv12, 1920x1080, q=-1--1, 5000 kb/s, 2 fps, 1k tbn, 8 tbc
        Metadata:
          encoder         : Lavc58.35.100 libx264
        Side data:
          cpb: bitrate max/min/avg: 0/0/5000000 buffer size: 0 vbv_delay: -1
    frame=    2 fps=0.0 q=7.0 size=       2kB time=00:00:00.12 bitrate= 104.4kbits/s dup=0 drop=4 speed=0.2frame=    2 fps=1.8 q=7.0 size=       2kB time=00:00:00.12 bitrate= 104.4kbits/s dup=0 drop=9 speed=0.1frame=    2 fps=1.1 q=7.0 size=       2kB time=00:00:00.12 bitrate= 104.4kbits/s dup=0 drop=14 speed=0.frame=    2 fps=0.9 q=7.0 size=       2kB time=00:00:00.12 bitrate= 104.4kbits/s dup=0 drop=18 speed=0.
    ...
    

Outcome is that FFmpeg is forever stuck at dropping frames.

FFmpeg code outputs above warning from two places here:
https://github.com/FFmpeg/FFmpeg/blob/master/libavdevice/xcbgrab.c

Checking the X protocol headers, tells that "minor_code:4 major_code:130" is ShmGetImage(), i.e. culprit is xcbgrab_frame_shm() function.

In the Xvfb case, I can add "-shmem" option to get X root window to be available as shared memory:
https://www.x.org/archive/X11R7.5/doc/man/man1/Xvfb.1.html

HOWEVER, while that gets FFmpeg streaming working, then the use-cases which output I'd like to stream, don't work! (Those use-cases use GL with Mesa)

If I try to use Xdummy instead of Xvfb (to get 3D acceleration working for X clients):

FFmpeg fails with that too, but I didn't find any Xdummy option that would allow FFmpeg to work with it.

In summary, there are several issues:

  • Error message that's incomprehensible to normal users
  • Not stopping when things have failed completely (=FFmpeg can't acquire any input frames)
  • FFmpeg code seems to have also a non-SHM function for getting the frames, but FFmpeg doesn't seem to fallback to that, or that part is broken for some other reason which doesn't show up in FFmpeg output

PS. "xdpyinfo" shows following for Xdummy:

name of display:    :99
version number:    11.0
vendor string:    The X.Org Foundation
vendor release number:    12005000
X.Org version: 1.20.5
maximum request size:  16777212 bytes
motion buffer size:  256
bitmap unit, bit order, padding:    32, LSBFirst, 32
image byte order:    LSBFirst
number of supported pixmap formats:    7
supported pixmap formats:
    depth 1, bits_per_pixel 1, scanline_pad 32
    depth 4, bits_per_pixel 8, scanline_pad 32
    depth 8, bits_per_pixel 8, scanline_pad 32
    depth 15, bits_per_pixel 16, scanline_pad 32
    depth 16, bits_per_pixel 16, scanline_pad 32
    depth 24, bits_per_pixel 32, scanline_pad 32
    depth 32, bits_per_pixel 32, scanline_pad 32
keycode range:    minimum 8, maximum 255
focus:  PointerRoot
number of extensions:    28
    BIG-REQUESTS
    Composite
    DAMAGE
    DOUBLE-BUFFER
    DPMS
    DRI2
    DRI3
    GLX
    Generic Event Extension
    MIT-SCREEN-SAVER
    MIT-SHM
    Present
    RANDR
    RECORD
    RENDER
    SECURITY
    SHAPE
    SYNC
    X-Resource
    XC-MISC
    XFIXES
    XFree86-DGA
    XFree86-VidModeExtension
    XINERAMA
    XInputExtension
    XKEYBOARD
    XTEST
    XVideo
default screen number:    0
number of screens:    1

screen #0:
  dimensions:    1920x1080 pixels (508x285 millimeters)
  resolution:    96x96 dots per inch
  depths (7):    24, 1, 4, 8, 15, 16, 32
  root window id:    0x14a
  depth of root window:    24 planes
  number of colormaps:    minimum 1, maximum 1
  default colormap:    0x20
  default number of colormap cells:    256
  preallocated pixels:    black 0, white 16777215
  options:    backing-store WHEN MAPPED, save-unders NO
...

I.e. MIT-SHM extension is suppported, X root window just isn't using it.

Change History (11)

comment:1 by Carl Eugen Hoyos, 3 years ago

Component: avdeviceundetermined
Resolution: needs_more_info
Status: newclosed
Version: 4.1.4unspecified

comment:2 by eero-t, 3 years ago

Component: undeterminedavdevice
Resolution: needs_more_info
Status: closedreopened
Version: unspecifiedgit-master

cehoyos, when you close bug as needing more info, indicate what information is missing, and don't remove already given information.

In case the problem was testing a release, I tried now also git build of FFmpeg:

ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -an -f x11grab -video_size 1920x1080 -framerate 2 -i :99 -c:v h264_vaapi -vf format=nv12,hwupload -b:v 5000k -f flv rtmp://192.168.0.8/live/desktop_1
ffmpeg version N-95996-ge3dddf2142 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --prefix=/opt/ --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 36.101 / 56. 36.101
  libavcodec     58. 64.101 / 58. 64.101
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 68.100 /  7. 68.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[x11grab @ 0x55bf40067a40] Cannot get the image data event_error: response_type:0 error_code:128 sequence:9 resource_id:4194304 minor_code:4 major_code:130.
[x11grab @ 0x55bf40067a40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':99':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 2 fps, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_vaapi))
Press [q] to stop, [?] for help
Output #0, flv, to 'rtmp://192.168.0.8/live/desktop_1':
  Metadata:
    encoder         : Lavf58.35.101
    Stream #0:0: Video: h264 (h264_vaapi) (High) ([7][0][0][0] / 0x0007), vaapi_vld, 1920x1080, q=-1--1, 5000 kb/s, 2 fps, 1k tbn, 2 tbc
    Metadata:
      encoder         : Lavc58.64.101 h264_vaapi
frame=    2 fps=0.0 q=-0.0 size=       1kB time=-00:00:00.49 bitrate=N/A dup=0 drop=734 speed=N/A  

And as can be seen from above, that has exactly the same problem, all frames are "dropped" (because FFmpeg x11grab fails to acquire them).

comment:3 by eero-t, 3 years ago

In the Xvfb case, I can add "-shmem" option to get X root window to be available as shared memory:
HOWEVER, while that gets FFmpeg streaming working, then the use-cases which output I'd like to stream, don't work!

FYI: with Xvfb "-shmem" root window option, ShmPutImage() calls will fail in (fullscreen) applications.

comment:4 by Marton Balint, 3 years ago

Cc: Marton Balint added

Well, you should try the latest git master because there were some issues with XCB grab timestamps for the last few weeks. Also make sure you can reproduce this with local (non-networked) output and without hwaccel.

comment:5 by eero-t, 2 years ago

Well, you should try the latest git master because there were some issues with XCB grab timestamps for the last few weeks. Also make sure you can reproduce this with local (non-networked) output and without hwaccel.

With git-master from last evening, I'll see:

$ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -an -f x11grab -video_size 1920x1080 -framerate 2 -i :99 -c:v h264_vaapi -vf format=nv12,hwupload -b:v 5000k -f flv rtmp://192.168.0.8/live/desktop
ffmpeg version N-96204-gf651b18c19 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --prefix=/opt/ --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 38.100 / 56. 38.100
  libavcodec     58. 65.100 / 58. 65.100
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 70.101 /  7. 70.101
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[x11grab @ 0x556499d1fa80] Cannot get the image data event_error: response_type:0 error_code:128 sequence:9 resource_id:4194304 minor_code:4 major_code:130.
[x11grab @ 0x556499d1fa80] Continuing without shared memory.
[x11grab @ 0x556499d1fa80] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':99':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 2 fps, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_vaapi))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x556499d43d00] sws_param option is deprecated and ignored
Output #0, flv, to 'rtmp://192.168.0.8/live/desktop':
  Metadata:
    encoder         : Lavf58.35.101
    Stream #0:0: Video: h264 (h264_vaapi) (High) ([7][0][0][0] / 0x0007), vaapi_vld, 1920x1080, q=-1--1, 5000 kb/s, 2 fps, 1k tbn, 2 tbc
    Metadata:
      encoder         : Lavc58.65.100 h264_vaapi
[flv @ 0x556499d29c40] Failed to update header with correct duration.N/A dup=0 drop=63 speed=N/A    
[flv @ 0x556499d29c40] Failed to update header with correct filesize.
frame=    2 fps=0.1 q=-0.0 Lsize=       1kB time=00:00:00.00 bitrate=7848.0kbits/s dup=0 drop=64 speed=3.09e-05x 

Same issue without HW acceleration to local file:

$ ffmpeg -y -an -f x11grab -video_size 1920x1080 -framerate 2 -i :99  out.mp4
ffmpeg version N-96204-gf651b18c19 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --prefix=/opt/ --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 38.100 / 56. 38.100
  libavcodec     58. 65.100 / 58. 65.100
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 70.101 /  7. 70.101
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[x11grab @ 0x55a4dccf4380] Cannot get the image data event_error: response_type:0 error_code:128 sequence:9 resource_id:4194304 minor_code:4 major_code:130.
[x11grab @ 0x55a4dccf4380] Continuing without shared memory.
[x11grab @ 0x55a4dccf4380] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':99':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 2 fps, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x55a4dcd07d80] sws_param option is deprecated and ignored
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf58.35.101
    Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 1920x1080, q=2-31, 200 kb/s, 2 fps, 16384 tbn, 2 tbc
    Metadata:
      encoder         : Lavc58.65.100 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
frame=    3 fps=0.0 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=3 speed=1.83xframe=    3 fps=1.9 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=5 speed=0.647frame=    3 fps=1.5 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=6 speed=0.489frame=    3 fps=1.0 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=8 speed=0.328frame=    3 fps=0.7 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=10 speed=0.24frame=    3 fps=0.7 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=11 speed=0.22frame=    3 fps=0.5 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=13 speed=0.18frame=    3 fps=0.5 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=15 speed=0.15frame=    3 fps=0.4 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=16 speed=0.14frame=    3 fps=0.4 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=18 speed=0.12frame=    3 fps=0.3 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=20 speed=0.11frame=    3 fps=0.3 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=22 speed=0.09frame=    3 fps=0.3 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=23 speed=0.09frame=    3 fps=0.3 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=24 speed=0.09frame=    3 fps=0.2 q=2.0 size=       0kB time=00:00:01.00 bitrate=   0.4kbits/s dup=0 drop=26 speed=0.08frame=    3 fps=0.2 q=2.0 Lsize=      25kB time=00:00:01.00 bitrate= 203.6kbits/s dup=0 drop=27 speed=0.0797x    

Now Fmpeg says that it's continuing without shared memory, but all frames are still dropped.

What's worse, last night git-master version says that (and fails), even when Xvfb has been started with -shmem option.

comment:6 by Marton Balint, 2 years ago

Maybe you are using some nightly builds, but you have not tried the latest git master, because the change I was referring to was committed after f651b18c19 (the version which you were using). Please try the latest...

comment:7 by eero-t, 2 years ago

Ok, now it isn't anymore dropping frames with Xvfb:

$ Xvfb :99 -screen 0 1920x1080x24 -nolisten tcp &
$ ffmpeg -y -an -f x11grab -video_size 1920x1080 -framerate 8 -i :99 out.mp4
ffmpeg version N-96235-g3670a10c2d Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --prefix=/opt --enable-libmfx --enable-vaapi --enable-sdl2 --disable-libx265 --disable-libx264 --disable-libvpx --enable-libvorbis --enable-libopus --disable-libmp3lame --disable-libass --disable-sndio --enable-libfreetype --enable-gpl --disable-doc
  libavutil      56. 38.100 / 56. 38.100
  libavcodec     58. 65.102 / 58. 65.102
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.102 / 58.  9.102
  libavfilter     7. 70.101 /  7. 70.101
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[x11grab @ 0x564d93f4b3c0] Cannot get the image data event_error: response_type:0 error_code:128 sequence:9 resource_id:2097152 minor_code:4 major_code:130.
[x11grab @ 0x564d93f4b3c0] Continuing without shared memory.
[x11grab @ 0x564d93f4b3c0] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':99':
  Duration: N/A, start: 1577985187.298908, bitrate: N/A
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 8 fps, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
Press [q] to stop, [?] for help
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf58.35.101
    Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 1920x1080, q=2-31, 200 kb/s, 8 fps, 16384 tbn, 8 tbc
    Metadata:
      encoder         : Lavc58.65.102 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
frame=  192 fps=8.0 q=2.0 Lsize=     543kB time=00:00:23.87 bitrate= 186.2kbits/s speed=   1x

I'll check it more tomorrow (Xdummy, the actual output etc).

comment:8 by Marton Balint, 2 years ago

Resolution: fixed
Status: reopenedclosed

Reopen if you can reproduce the original issue with the git master.

comment:9 by Carl Eugen Hoyos, 2 years ago

Resolution: fixedinvalid

comment:10 by eero-t, 2 years ago

Resolution: invalidfixed

Whoever is running the "cehoyos" bot, please fix it, it's randomly messing up bug statuses. Correcting it again...

comment:11 by Carl Eugen Hoyos, 2 years ago

Resolution: fixedinvalid
Note: See TracTickets for help on using tickets.