Opened 6 years ago

Last modified 5 years ago

#3192 new defect

Dropped frames if x11grab and webcam are used in filter_complex

Reported by: ian_m Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords: x11grab
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Using the overlay filter to overlay video from a webcam onto x11grab causes many dropped frames. The frames seem to come in short bursts followed by a large number of dropped frames. This is easily seen by inserting showinfo into the x11grab stream before it is sent to the overlay filter. The attached file shows this behaviour. If the overlay filter isn't used, the frames appear to have approximately the correct timestamps as shown in the second attachment.

Attachments (3)

ffmpeg-overlay.log (61.5 KB) - added by ian_m 6 years ago.
ffmpeg-nooverlay.log (214.5 KB) - added by ian_m 6 years ago.
x11grab-nonblock.diff (1.9 KB) - added by ian_m 6 years ago.

Download all attachments as: .zip

Change History (17)

Changed 6 years ago by ian_m

Changed 6 years ago by ian_m

comment:1 Changed 6 years ago by cehoyos

What happens if you significantly decrease the x11grab resolution (640x480 or 400x200)?

comment:2 Changed 6 years ago by ian_m

I get the same behavior, even at 400x200.

comment:3 Changed 6 years ago by ubitux

Wouldn't that be related to the nanosleep hack used to sync with the framerate?

Changed 6 years ago by ian_m

comment:4 Changed 6 years ago by ian_m

I wrote the attached patch to make x11grab respect AVFMT_FLAGS_NONBLOCK but it didn't seem to fix the issue.

comment:5 Changed 6 years ago by cehoyos

Is the problem also reproducible if you replace -f x11grab -video_size 1680x1050 -framerate 30 -i :0.0 in your command line with -f lavfi -i testsrc=s=1680x1050:r=30?

The following works fine here:

$ ffmpeg -f x11grab -s 1920x1200 -i :0 -f v4l2 -s 320x176 -i /dev/video1 -filter_complex overlay -qscale 2 out.avi

comment:6 Changed 6 years ago by ian_m

No, if I use the testsrc instead of x11grab it seems to work.

comment:7 Changed 6 years ago by cehoyos

frame= 551 fps= 22 q=-1.0 Lsize= 3615kB time=00:00:24.63 bitrate=1202.2kbits/s

As pointed out on the mailing list, this indicates the problem is also reproducible without overlay: Your system does not reach the requested frame rate. Try with overlay and a framerate of ~15.

comment:8 Changed 6 years ago by ian_m

Lowering the framerate does not seem to help. It seems that in the nonoverlay version, it takes a second or so for everything to initialize so the framerate x264 sees starts low and converges towards 30. In the overlay version the framerate is stuck at around 5-6 fps.

comment:9 Changed 6 years ago by cehoyos

  • Keywords x11grab added

comment:10 Changed 6 years ago by jnvsor

Switching the input order is a workaround

ffmpeg \
-f x11grab -framerate 30 -s hd1080 -i :0.0 \
-f v4l2 -framerate 30 -i /dev/video0 \
-filter_complex '[0][1]overlay=x=300:y=300[cap]' \
-map '[cap]' \
-c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 \
-threads 2 -y out.avi
# frame=  225 fps= 18 q=-1.0 Lsize=   36782kB time=00:00:12.56 bitrate=23977.3kbits/s dup=0 drop=74

Switch the input order and switch it on the overlay filter again:

ffmpeg \
-f v4l2 -framerate 30 -i /dev/video0 \
-f x11grab -framerate 30 -s hd1080 -i :0.0 \
-filter_complex '[1][0]overlay=x=300:y=300[cap]' \
-map '[cap]' \
-c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 \
-threads 2 -y out.avi
# frame=  208 fps= 30 q=-1.0 Lsize=   25993kB time=00:00:06.93 bitrate=30711.3kbits/s

comment:11 Changed 5 years ago by jnvsor

I can confirm it occurs when both streams are passed through filter_complex, even if they are null filters that shouldn't effect performance at all.

Low fps:

ffmpeg -f x11grab -framerate 30 -s hd1080 -i :0.0 -f v4l2 -framerate 30 -i /dev/video0 -filter_complex '[0]null[cap];[1]null[webcam]' -map '[cap]' -map '[webcam]' -c:v libx264 -pix_fmt yuv444p -preset ultrafast -qp 0 -threads 2 -y out.avi

Remove either of the null filters and it will work at full framerate:

- '[0]null[cap];[1]null[webcam]'
- -map '[cap]'
+ '[1]null[webcam]'
+ -map 0
OR
+ '[0]null[cap]'
+ -map 1

Add the v4l2 input first and it will work at full framerate:

- -f x11grab -framerate 30 -s hd1080 -i :0.0 -f v4l2 -framerate 30 -i /dev/video0
+ -f v4l2 -framerate 30 -i /dev/video0 -f x11grab -framerate 30 -s hd1080 -i :0.0

It only occurs if that filter output is actually used, piping one to a nullsink also makes it work at full framerate:

- '[0]null[cap];[1]null[webcam]'
- -map '[cap]'
+ '[0]null[cap];[cap]nullsink;[1]null[webcam]'
+ -map 0

comment:12 Changed 5 years ago by cehoyos

  • Summary changed from Dropped Frames With Overlay Filter to x11grab drops frames if overlay filter is used

comment:13 Changed 5 years ago by jnvsor

  • Summary changed from x11grab drops frames if overlay filter is used to Dropped frames if v4l2 and x11grab are used in filter_complex

Changing the title since it can be reproduced without overlay, it just so happens overlay requires both streams to be passed through filter_complex which seems to be closer to the root of the problem

ffmpeg \
-f x11grab -i :0.0 \
-f v4l2 -i /dev/video0 \
-filter_complex '[0]null[cap];[1]null[webcam]' \
-map '[cap]' -map '[webcam]' \
-c:v libx264 -y out.mkv

comment:14 Changed 5 years ago by jnvsor

  • Summary changed from Dropped frames if v4l2 and x11grab are used in filter_complex to Dropped frames if x11grab and webcam are used in filter_complex

Happens with webcam audio too!

ffmpeg \
-f x11grab -i :0.0 \
-f pulse -i alsa_input.usb-046d_0802_F1851F50-02-U0x46d0x802.analog-mono -channel_layout stereo \
-filter_complex '[0]null[cap];[1]anull[webcam]' \
-map '[cap]' -map '[webcam]' \
-c:v libx264 -c:a libmp3lame -y test.mkv
Note: See TracTickets for help on using tickets.