x11grab leaks shared memory if X server shuts down

[Levi]

Summary of the bug:

If the X server supports the shared memory extension and is shut down while ffmpeg x11grab is running then x11grab_read_close() is not called and so the shared memory segment is not deleted.

It is cleaned up correctly in the event of SIGINT/SIGTERM on the ffmpeg process -- it's only a broken X11 connection that has a problem.

How to reproduce:
Running on debian stable using git master HEAD compilation.

Minimal x11grab build:

./configure \
        --cpu=core2 \
        --arch=x86_64 \
        \
        --disable-doc \
        --disable-bsfs \
        --disable-outdevs \
        --disable-parsers \
        --disable-hwaccels \
        --disable-devices \
        --disable-protocols \
        --disable-indevs \
        --disable-filters \
        --disable-decoders \
        --disable-encoders \
        --disable-demuxers \
        --disable-muxers \
        --disable-ffplay \
        --disable-ffprobe \
        --disable-ffserver \
        \
        --enable-ffmpeg \
        --enable-gpl \
        --enable-yasm \
        \
        --enable-muxer=rawvideo \
        --enable-demuxer=rawvideo \
        --enable-encoder=rawvideo \
        --enable-decoder=rawvideo \
        --enable-protocol=file \
        --enable-protocol=pipe \
        --enable-filter=fps \
        --enable-filter=null \
        --enable-indev=x11grab \
        --enable-x11grab

No existing shared memory segments:

$ ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Start X

Start ffmpeg:

$ .ffmpeg.x11grab  -loglevel info  -f x11grab  -s 720x576  -framerate 25  -i :0  -an  -f rawvideo -y /dev/null
ffmpeg version N-58440-g841c0aa Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 25 2013 18:38:31 with gcc 4.7 (Debian 4.7.2-5)
  configuration: --cpu=core2 --arch=x86_64 --disable-doc --disable-bsfs --disable-outdevs --disable-parsers --disable-hwaccels --disable-devices --disable-protocols --disable-indevs --disable-filters --disable-decoders --disable-encoders --disable-demuxers --disable-muxers --disable-ffplay --disable-ffprobe --disable-ffserver --enable-ffmpeg --enable-gpl --enable-yasm --enable-muxer=rawvideo --enable-demuxer=rawvideo --enable-encoder=rawvideo --enable-decoder=rawvideo --enable-protocol=file --enable-protocol=pipe --enable-filter=fps --enable-filter=null --enable-indev=x11grab --enable-x11grab
  libavutil      52. 54.100 / 52. 54.100
  libavcodec     55. 44.100 / 55. 44.100
  libavformat    55. 21.101 / 55. 21.101
  libavdevice    55.  5.100 / 55.  5.100
  libavfilter     3. 91.100 /  3. 91.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
[x11grab @ 0xcd3a20] device: :0 -> display: :0 x: 0 y: 0 width: 720 height: 576
[x11grab @ 0xcd3a20] shared memory extension found
Input #0, x11grab, from ':0':
  Duration: N/A, start: 1385365236.018051, bitrate: 331776 kb/s
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 720x576, 331776 kb/s, 25 tbr, 1000k tbn, 25 tbc
Output #0, rawvideo, to '/dev/null':
  Metadata:
    encoder         : Lavf55.21.101
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 720x576, q=2-31, 200 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
Press [q] to stop, [?] for help
...

Stop X while leaving ffmpeg running

XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"kbits/s
      after 9714 requests (9714 known processed) with 0 events remaining.

Shared memory segment was not released:

$ ipcs
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 29622272   levi       777        1658880    0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Repeat until the system runs out of shared memory.

[Carl Eugen Hoyos]

Is avformat_close_input() in libavformat/utils.c called when the X server shuts down?
(Is ffmpeg_cleanup() in ffmpeg.c called?)

[Levi]

Extra av_log debugging output with SIGINT:

Press [q] to stop, [?] for help
video:25920kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.000000%
fmpeg_cleanup() called
    avformat_close_input() called
      x11grab_read_close() called
      x11grab_read_close() done
    avformat_close_input() done
Received signal 2: terminating.
ffmpeg_cleanup() done.

With X shutdown:

Press [q] to stop, [?] for help
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"kbits/s
      after 339 requests (339 known processed) with 0 events remaining.

I created a partial solution that has been tested on linux here:
​https://github.com/levic/FFmpeg/compare/FFmpeg:master...master
It's only a partial solution because while it solves SIGKILL/X disconnect for x11grab, if any other filters/input devices need to be cleanly shut down they still won't be if x11grab encounters an X disconnect.

On linux and freebsd, you can mark the segment for deletion while processes are still attached and it will automatically clean up when the attached process count goes to zero.
​http://linux.die.net/man/2/shmctl
​http://www.freebsd.org/cgi/man.cgi?query=shmctl&sektion=2

Unfortunately posix & solaris say nothing along these lines and may delete the segment immediately (or at least prevent X from attaching to it)
​http://docs.oracle.com/cd/E19109-01/tsolaris8/835-8003/6ruu1b0qp/index.html
​http://pubs.opengroup.org/onlinepubs/007904975/functions/shmctl.html
​http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2005-04/0599.html

[Carl Eugen Hoyos]

I cannot comment on your patch but if it fixes an issue (like for example a part of this ticket), please send it to the ffmpeg-devel mailing list, patches on this bug tracker are typically ignored.