Opened 10 years ago

Last modified 3 years ago

#4437 open defect

AVFoundation recording has consistent intermittent frame drops

Reported by: LordHDL Owned by: mateo
Priority: normal Component: avdevice
Version: git-master Keywords: avfoundation bounty
Cc: matthieu.bouron@gmail.com Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Summary: 60 FPS screen capture on OS X results in intermittent drops every 5-8 seconds.
How to reproduce: Record a video with AVFoundation using -framerate 60.

ffmpeg -f avfoundation -pix_fmt 0rgb -framerate 60 -video_device_index 3 -i “” -c:v libx264rgb -crf 0 -vf crop=426:240:507:339 ~/Desktop/avfoundation.avi

Additional notes:

  • I have a SATA 3 SSD; plenty of disk write speed.
  • The source does not drop frames during recording, only the video does.
  • The source frame rate is exactly 60 FPS.
  • The frame drops have a fairly consistent timing.
  • Recording the same source in Windows does not produce these frame drops.
  • Media players will report an average rate of around 57.13 (it varies).
  • The -r 60 output option will of course force the rate to 60, but not prevent the frame drops.
  • Example recording: ​https://www.dropbox.com/s/0dthf652g644ldq/avfoundation.avi.xz?dl=0

Attachments (2)

Terminal Saved Output​ (20.7 KB ) - added by LordHDL 10 years ago.
Terminal output from the video example provided.
avf.patch​ (11.6 KB ) - added by Thilo Borgmann 9 years ago.
First attempt to solve the issue send to ffmpeg-devel

Download all attachments as: .zip

Change History (52)

by LordHDL, 10 years ago

Attachment: Terminal Saved Output​ added

Terminal output from the video example provided.

comment:1 by Thilo Borgmann, 10 years ago

Please provide the full uncut console output of the command you show to reproduce this behaviour.

What device is it, you are capturing from?

comment:2 by mateo, 10 years ago

Cc: matthieu.bouron@gmail.com added
Owner: set to mateo
Status: new → open

in reply to:  1 comment:3 by LordHDL, 10 years ago

Replying to thilo.borgmann:

Please provide the full uncut console output of the command you show to reproduce this behaviour.

What device is it, you are capturing from?

The full output is in the attached text file. "Capture screen 0" is the selected device.

comment:4 by Thilo Borgmann, 10 years ago

I cannot reproduce with libx264rgb...

Please strip the command line / features down to the minimum required to trigger these frameskips you're reporting. Try a different internal video codec an/or raw video and to skip the crop filter.

comment:5 by mateo, 10 years ago

I'm not able to reproduce the issue either.
What is your OSX version ? Maybe the CPU is the bottleneck at some point. Is your machine under heavy load ?

comment:6 by LordHDL, 10 years ago

I'm on 10.10.2. My CPU usage is around 15-18% while recording. I have an i7 2720QM 2.2 GHz and a Radeon 6750M (MacBook Pro 8,2). As I said in my notes, the drops don't happen when recording with ffmpeg in Windows on the same hardware. On OS X the drops also happen when using huffyuv.

Also I know I specified 60 FPS but this happens even if I don't use the -framerate option. The recording defaults to 30 FPS and will still exhibit the intermittent drops. The only time hardware appears to be a bottleneck is if I don't use the crop filter and record full screen.

Edit: This is perhaps a better example video to showcase this issue: ​https://www.dropbox.com/s/71c37uq91xqyvzr/huffyuv.avi.xz?dl=0

The flickering of the objects happens every frame, so any frame drops are immediately obvious. Here is the output from that recording (total CPU usage was lower, around 10-12%):

Last login: Tue Apr  7 17:25:40 on ttys000
HDLs-MacBook-Pro:~ HDL$ ffmpeg -f avfoundation -framerate 60 -i "3:none" -c:v huffyuv -vf crop=320:240:392:177 ~/Desktop/avtest.avi
ffmpeg version N-71282-g0968180-HDL Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --as=yasm --extra-version=HDL --disable-shared --enable-static --enable-gpl --enable-nonfree --enable-pthreads --enable-postproc --enable-libmp3lame --enable-libx264 --enable-bzlib --enable-zlib --enable-version3 --enable-libfdk-aac --enable-filters --enable-runtime-cpudetect
  libavutil      54. 22.100 / 54. 22.100
  libavcodec     56. 34.100 / 56. 34.100
  libavformat    56. 29.100 / 56. 29.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 13.101 /  5. 13.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
2015-04-07 17:28:12.639 ffmpeg[7183:171297] alloc
2015-04-07 17:28:12.640 ffmpeg[7183:171297] query 2
2015-04-07 17:28:12.640 ffmpeg[7183:171297] release 2
2015-04-07 17:28:12.640 ffmpeg[7183:171297] ** MyPlugInitializeWithObjectID
2015-04-07 17:28:12.640 ffmpeg[7183:171297] init
2015-04-07 17:28:12.640 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.640 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.640 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.715 ffmpeg[7183:171297] <CMVideoFormatDescription 0x7fe8bae13d50 [0x7fff774fdcf0]> {
	mediaType:'vide' 
	mediaSubType:'BGRA' 
	mediaSpecific: {
		codecType: 'BGRA'		dimensions: 426 x 240 
	} 
	extensions: {<CFBasicHash 0x7fe8bae162d0 [0x7fff774fdcf0]>{type = immutable dict, count = 4,
entries =>
	1 : <CFString 0x7fff78b74f08 [0x7fff774fdcf0]>{contents = "CVFieldCount"} = <CFNumber 0x137 [0x7fff774fdcf0]>{value = +1, type = kCFNumberSInt64Type}
	2 : <CFString 0x10bfa57f0 [0x7fff774fdcf0]>{contents = "CVBytesPerRow"} = <CFNumber 0x6a837 [0x7fff774fdcf0]>{value = +1704, type = kCFNumberSInt64Type}
	4 : <CFString 0x7fff77bc3e50 [0x7fff774fdcf0]>{contents = "com.apple.cmio.format_extension.video.only_has_i_frames"} = <CFBoolean 0x7fff774fe6c8 [0x7fff774fdcf0]>{value = true}
	5 : <CFString 0x7fff794ae6f0 [0x7fff774fdcf0]>{contents = "FormatName"} = <CFString 0x10bfa5810 [0x7fff774fdcf0]>{contents = "Component Video - CCIR-601 RGB"}
}
}
}
2015-04-07 17:28:12.715 ffmpeg[7183:171297] alloc
2015-04-07 17:28:12.716 ffmpeg[7183:171297] query 2
2015-04-07 17:28:12.716 ffmpeg[7183:171297] release 2
2015-04-07 17:28:12.716 ffmpeg[7183:171297] ** MyPlugInitializeWithObjectID
2015-04-07 17:28:12.716 ffmpeg[7183:171297] init
2015-04-07 17:28:12.716 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.716 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.716 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.716 ffmpeg[7183:171297] <CMVideoFormatDescription 0x7fe8bad236c0 [0x7fff774fdcf0]> {
	mediaType:'vide' 
	mediaSubType:'2vuy' 
	mediaSpecific: {
		codecType: '2vuy'		dimensions: 426 x 240 
	} 
	extensions: {<CFBasicHash 0x7fe8bad23f50 [0x7fff774fdcf0]>{type = immutable dict, count = 6,
entries =>
	0 : <CFString 0x7fff78b75088 [0x7fff774fdcf0]>{contents = "CVImageBufferYCbCrMatrix"} = <CFString 0x7fff78b750c8 [0x7fff774fdcf0]>{contents = "ITU_R_601_4"}
	1 : <CFString 0x7fff78b74f08 [0x7fff774fdcf0]>{contents = "CVFieldCount"} = <CFNumber 0x137 [0x7fff774fdcf0]>{value = +1, type = kCFNumberSInt64Type}
	2 : <CFString 0x7fff78b75188 [0x7fff774fdcf0]>{contents = "CVImageBufferTransferFunction"} = <CFString 0x7fff78b750a8 [0x7fff774fdcf0]>{contents = "ITU_R_709_2"}
	4 : <CFString 0x7fff77bc3e50 [0x7fff774fdcf0]>{contents = "com.apple.cmio.format_extension.video.only_has_i_frames"} = <CFBoolean 0x7fff774fe6c8 [0x7fff774fdcf0]>{value = true}
	5 : <CFString 0x7fff78b75108 [0x7fff774fdcf0]>{contents = "CVImageBufferColorPrimaries"} = <CFString 0x7fff78b75148 [0x7fff774fdcf0]>{contents = "SMPTE_C"}
	6 : <CFString 0x7fff794ae6f0 [0x7fff774fdcf0]>{contents = "FormatName"} = <CFString 0x10bfa5870 [0x7fff774fdcf0]>{contents = "Component Video - CCIR-601 uyvy"}
}
}
}
2015-04-07 17:28:12.819 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.819 ffmpeg[7183:171297] enog
2015-04-07 17:28:12.819 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.819 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.819 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.820 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.820 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.820 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.820 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.820 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.821 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.821 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.821 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.821 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.821 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.821 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.822 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.822 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.822 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.822 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.822 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.823 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.823 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.823 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.823 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.823 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.823 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.823 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.824 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.824 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.824 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.824 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.824 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.824 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.825 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.825 ffmpeg[7183:171297] grrf
2015-04-07 17:28:12.825 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.825 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.825 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.825 ffmpeg[7183:171297] grrf
2015-04-07 17:28:12.826 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.826 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.826 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.826 ffmpeg[7183:171297] trfn
2015-04-07 17:28:12.826 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.826 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.826 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.827 ffmpeg[7183:171297] trfm
2015-04-07 17:28:12.827 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.827 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.827 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.827 ffmpeg[7183:171297] trfn
2015-04-07 17:28:12.827 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.827 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.827 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.828 ffmpeg[7183:171297] dulp
2015-04-07 17:28:12.828 ffmpeg[7183:171297] tpni
2015-04-07 17:28:12.828 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.828 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.828 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.828 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.828 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.828 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.829 ffmpeg[7183:171297] enog
2015-04-07 17:28:12.829 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.829 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.829 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.829 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.829 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.829 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.829 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.829 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.829 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.830 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.830 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.830 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.830 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.830 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.830 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.830 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.830 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.830 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.830 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.830 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.830 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.831 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.831 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.831 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.831 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.831 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.831 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.831 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.831 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.831 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.832 ffmpeg[7183:171297] elt 0
2015-04-07 17:28:12.832 ffmpeg[7183:171297] MyPlugObjectSetPropertyData
2015-04-07 17:28:12.832 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.832 ffmpeg[7183:171297] grrf
2015-04-07 17:28:12.832 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.832 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.832 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.832 ffmpeg[7183:171297] grrf
2015-04-07 17:28:12.832 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.833 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.833 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.833 ffmpeg[7183:171297] trfn
2015-04-07 17:28:12.833 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.833 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.833 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.833 ffmpeg[7183:171297] trfm
2015-04-07 17:28:12.833 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.833 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.833 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.833 ffmpeg[7183:171297] trfn
2015-04-07 17:28:12.834 ffmpeg[7183:171297] bolg
2015-04-07 17:28:12.834 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.834 ffmpeg[7183:171297] MyPlugObjectHasProperty
2015-04-07 17:28:12.834 ffmpeg[7183:171297] dulp
2015-04-07 17:28:12.834 ffmpeg[7183:171297] tpni
2015-04-07 17:28:12.834 ffmpeg[7183:171297] 0
2015-04-07 17:28:12.834 ffmpeg[7183:171297] MyPlugObjectGetPropertyDataSize
2015-04-07 17:28:12.834 ffmpeg[7183:171297] sel dnwo
2015-04-07 17:28:12.834 ffmpeg[7183:171297] scope bolg
2015-04-07 17:28:12.834 ffmpeg[7183:171297] elt 0
[avfoundation @ 0x7fe8bb02bc00] Selected pixel format (yuv420p) is not supported by the input device.
[avfoundation @ 0x7fe8bb02bc00] Supported pixel formats:
[avfoundation @ 0x7fe8bb02bc00]   uyvy422
[avfoundation @ 0x7fe8bb02bc00]   yuyv422
[avfoundation @ 0x7fe8bb02bc00]   nv12
[avfoundation @ 0x7fe8bb02bc00]   0rgb
[avfoundation @ 0x7fe8bb02bc00]   bgr0
[avfoundation @ 0x7fe8bb02bc00] Overriding selected pixel format to use uyvy422 instead.
[avfoundation @ 0x7fe8bb02bc00] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, avfoundation, from '3:none':
  Duration: N/A, start: 30205.076500, bitrate: N/A
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 1440x900, 1000k tbr, 1000k tbn, 1000k tbc
[avi @ 0x7fe8bc820800] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2
Output #0, avi, to '/Users/HDL/Desktop/avtest.avi':
  Metadata:
    ISFT            : Lavf56.29.100
    Stream #0:0: Video: huffyuv (HFYU / 0x55594648), yuv422p, 320x240, q=2-31, 200 kb/s, 1000k fps, 600 tbn, 1000k tbc
    Metadata:
      encoder         : Lavc56.34.100 huffyuv
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> huffyuv (native))
Press [q] to stop, [?] for help
frame=   34 fps=0.0 q=0.0 size=    3123kB time=00:00:00.53 bitrate=47965.5kbits/frame=   64 fps= 63 q=0.0 size=    5964kB time=00:00:01.03 bitrate=47278.6kbits/frame=   95 fps= 62 q=0.0 size=    8892kB time=00:00:01.55 bitrate=46995.9kbits/frame=  126 fps= 62 q=0.0 size=   11824kB time=00:00:02.06 bitrate=46870.6kbits/frame=  157 fps= 61 q=0.0 size=   14766kB time=00:00:02.58 bitrate=46824.3kbits/frame=  187 fps= 61 q=0.0 size=   17602kB time=00:00:03.08 bitrate=46765.0kbits/frame=  218 fps= 61 q=0.0 size=   20538kB time=00:00:03.60 bitrate=46735.3kbits/frame=  249 fps= 61 q=0.0 size=   23464kB time=00:00:04.11 bitrate=46693.0kbits/frame=  280 fps= 61 q=0.0 size=   26391kB time=00:00:04.63 bitrate=46660.0kbits/frame=  311 fps= 61 q=0.0 size=   29330kB time=00:00:05.15 bitrate=46654.5kbits/frame=  341 fps= 61 q=0.0 size=   32161kB time=00:00:05.65 bitrate=46630.9kbits/frame=  372 fps= 61 q=0.0 size=   35094kB time=00:00:06.16 bitrate=46619.9kbits/frame=  403 fps= 61 q=0.0 size=   38030kB time=00:00:06.68 bitrate=46614.5kbits/frame=  434 fps= 60 q=0.0 size=   40956kB time=00:00:07.20 bitrate=46598.8kbits/frame=  464 fps= 60 q=0.0 size=   43798kB time=00:00:07.70 bitrate=46596.1kbits/frame=  495 fps= 60 q=0.0 size=   46724kB time=00:00:08.21 bitrate=46583.5kbits/frame=  526 fps= 60 q=0.0 size=   49654kB time=00:00:08.73 bitrate=46576.1kbits/frame=  556 fps= 60 q=0.0 size=   52498kB time=00:00:09.23 bitrate=46577.6kbits/frame=  587 fps= 60 q=0.0 size=   55422kB time=00:00:09.75 bitrate=46566.1kbits/frame=  617 fps= 60 q=0.0 size=   58264kB time=00:00:10.25 bitrate=46565.4kbits/frame=  648 fps= 60 q=0.0 size=   61192kB time=00:00:10.76 bitrate=46559.0kbits/frame=  679 fps= 60 q=0.0 size=   64118kB time=00:00:11.28 bitrate=46551.3kbits/frame=  710 fps= 60 q=0.0 size=   67054kB time=00:00:11.80 bitrate=46551.5kbits/frame=  740 fps= 60 q=0.0 size=   69881kB time=00:00:12.30 bitrate=46541.8kbits/frame=  770 fps= 60 q=0.0 size=   72702kB time=00:00:12.80 bitrate=46529.2kbits/frame=  800 fps= 60 q=0.0 size=   75533kB time=00:00:13.30 bitrate=46523.7kbits/frame=  831 fps= 60 q=0.0 size=   78461kB time=00:00:13.81 bitrate=46520.3kbits/frame=  862 fps= 60 q=0.0 size=   81400kB time=00:00:14.33 bitrate=46522.7kbits/frame=  892 fps= 60 q=0.0 size=   84229kB time=00:00:14.83 bitrate=46517.2kbits/frame=  923 fps= 60 q=0.0 size=   87154kB time=00:00:15.35 bitrate=46512.6kbits/frame=  954 fps= 60 q=0.0 size=   90095kB time=00:00:15.86 bitrate=46516.2kbits/frame=  985 fps= 60 q=0.0 size=   93020kB time=00:00:16.38 bitrate=46511.7kbits/frame= 1015 fps= 60 q=0.0 size=   95862kB time=00:00:16.88 bitrate=46513.4kbits/frame= 1046 fps= 60 q=0.0 size=   98797kB time=00:00:17.40 bitrate=46514.1kbits/frame= 1077 fps= 60 q=0.0 size=  101727kB time=00:00:17.91 bitrate=46512.3kbits/frame= 1107 fps= 60 q=0.0 size=  104571kB time=00:00:18.41 bitrate=46514.9kbits/frame= 1138 fps= 60 q=0.0 size=  107495kB time=00:00:18.93 bitrate=46510.5kbits/frame= 1169 fps= 60 q=0.0 size=  110424kB time=00:00:19.45 bitrate=46508.8kbits/frame= 1200 fps= 60 q=0.0 size=  113365kB time=00:00:19.96 bitrate=46511.6kbits/frame= 1230 fps= 60 q=0.0 size=  116201kB time=00:00:20.46 bitrate=46510.8kbits/frame= 1261 fps= 60 q=0.0 size=  119133kB time=00:00:20.98 bitrate=46510.1kbits/frame= 1292 fps= 60 q=0.0 size=  122066kB time=00:00:21.50 bitrate=46510.0kbits/frame= 1322 fps= 60 q=0.0 size=  124898kB time=00:00:22.00 bitrate=46507.5kbits/frame= 1352 fps= 60 q=0.0 size=  127738kB time=00:00:22.50 bitrate=46507.8kbits/frame= 1383 fps= 60 q=0.0 size=  130676kB time=00:00:23.01 bitrate=46509.7kbits/frame= 1407 fps= 60 q=0.0 Lsize=  133337kB time=00:00:23.43 bitrate=46612.9kbits/s    
video:133002kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.252035%
2015-04-07 17:28:36.367 ffmpeg[7183:171297] MyPlugTeardown
2015-04-07 17:28:36.368 ffmpeg[7183:171297] MyPlugTeardown
HDLs-MacBook-Pro:~ HDL$
Last edited 10 years ago by LordHDL (previous) (diff)

comment:7 by LordHDL, 10 years ago

Here's a test without the -vf option: ​https://www.dropbox.com/s/a982jatnw7n6d45/nofilter.tar.xz?dl=0

I didn't include the lossless recording since, even compressed via xz, my connection is far too slow to upload within a reasonable time. But you can get the idea from the encoded video and included Terminal outputs. I'm actually surprised the recording is using that much CPU. I thought lossless was supposed to be more taxing on disk write speed, while compression depends more on CPU?

comment:8 by LordHDL, 9 years ago

Just an update to say that I've added a bounty to this on Bountysource; I'm still interested in a fix for this.

I also have recent videos here: ​https://www.dropbox.com/s/0ovhm3h3dl1fhx9/Tests.tar.xz?dl=0

comment:9 by mateo, 9 years ago

I'll take another look at the issue in one week as I won't have access to a mac until then (and if no one fixes the issue).

comment:10 by LordHDL, 9 years ago

I increased the bounty for this. Let me know if you could use any other tests or information from my part.

comment:12 by mateo, 9 years ago

I'm currently working on the issue. A patch can be expected by the end of the week.

Last edited 9 years ago by mateo (previous) (diff)

comment:13 by mateo, 9 years ago

A patch has been sent to the mailing list [1], however it does not get rid of the issue but might help a bit. I beleive the issue is on the avfoundation side as the screen recording feature might not be in sync with the display (or the display compositor might not be able to stay at a constant 60fps ?).

Can you try the patch and let me know if it improves a bit your issue ?
ffmpeg will output a warning if a drop happens on the avdevice side.

[1] ​http://ffmpeg.org/pipermail/ffmpeg-devel/2015-August/177845.html

comment:14 by Thilo Borgmann, 9 years ago

Thanks for your patch, mateo!

If you extend your buffer size to way bigger than 4 frames you should be able to see if your assumptions are correct and we just do not consume the frames fast enough (and your buffer warning occurs later).

Otherwise, if there is something out of sync on the system-side, your buffering extension could be useless and memory requirements raise. In that case we have to figure out if we can somehow acquire these missing frames in any way or if we don't even have a chance.

I'm eager to hear feedback on the patch from LordHDL.

in reply to:  14 comment:15 by mateo, 9 years ago

Replying to thilo.borgmann:

Thanks for your patch, mateo!

If you extend your buffer size to way bigger than 4 frames you should be able to see if your assumptions are correct and we just do not consume the frames fast enough (and your buffer warning occurs later).

While testing the patch, ffmpeg did not output the new warning telling the user that a frame is being dropped because the queue is full while still seeing the issue on the output file (what LordHDL has described / what is mentionned in the commit message).

Otherwise, if there is something out of sync on the system-side, your buffering extension could be useless and memory requirements raise. In that case we have to figure out if we can somehow acquire these missing frames in any way or if we don't even have a chance.

I've tried to set the setAlwaysDiscardsLateVideoFrames to NO but it was leading to a performance decrease, ie: the capture was running at 54fps instead of 60.

I also experience the same issue using gstreamer.

Anyway, I don't think the buffering extension is useless as the new code reduce the locking area and give more guarantee that in case of a sudden slowdown we don't drop any frames (unless the queue gets full). If the memory requirements raise is an issue i can reduce the default queue size. Even add an option to configure it.
What do you think ?

[...]

comment:16 by LordHDL, 9 years ago

I've never applied patches in this manner before so I'm not actually sure how to do it. Could you give me a brief rundown?

Additionally as I said in comment 6, this issue does also happen with the default 30 FPS. I initially thought it was specific to 60 but it happens regardless.

comment:17 by Thilo Borgmann, 9 years ago

Briefly:

You might want to do it in a seperate branch, so create one with git branch,
next you apply the patch file with git am.
Then you rerun configure and make.
After that you can test the new behaviour.

comment:18 by LordHDL, 9 years ago

What option am I supposed to use with git am? This is my first time using this command.

comment:19 by Thilo Borgmann, 9 years ago

You might use git am < file.patch.

comment:20 by LordHDL, 9 years ago

That does nothing for me. I'm not a git expert so I'm likely just doing it wrong. Can you give an example of how that is supposed to look?

comment:21 by Thilo Borgmann, 9 years ago

Use:

git apply ../patches/avf.patch

If you further want it to be committed to your local repository:

git add libavdevice/avfoundation.m
git commit -m "avf patch"

Then git log -1 shows you your patch to be the last change.
The avf.patch is just a copy&paste from the mail on ffmpeg-devel.

by Thilo Borgmann, 9 years ago

Attachment: avf.patch​ added

First attempt to solve the issue send to ffmpeg-devel

comment:22 by Thilo Borgmann, 9 years ago

You can validate that your patch is applied if you lastly type:
git diff

The output you get should look (almost) similar to the patch file.

comment:23 by LordHDL, 9 years ago

New test with patch: ​https://www.dropbox.com/s/wbp01bcxhrrauxi/AVFTest.tar.xz?dl=0

Ignore the registration prompt (click outside or the X), it's not needed to download.

The source chosen is a recent PC game (not emulated, low CPU & GPU demands) so it runs at 60 FPS exactly as opposed to older standards like 59.94. Drops in smoothness are very obvious. The patch does seem to help with the audio tempo issue I've experienced, however. It doesn't seem to cut off or warp anymore.

comment:24 by Thilo Borgmann, 9 years ago

Unfortunately I will be offline for the whole weekend so maybe mateo can comment on how the patch work for him. If it just fixes an audio issue (got a ticket for that?), then it might help with that but does not fix this issue.

Still I wonder if the FFmpeg side causes the frame drops or if we never come to see these frames and Apple drops them.

I will also need some more time to review the patch on the ml.

comment:25 by LordHDL, 9 years ago

I'm hoping it's not a rooted issue in AV Foundation itself. As I said before, recordings in Windows do not exhibit the drops (on the same hardware). I know OS X is capable of perfectly smooth capture, as evidenced by the recording app ScreenFlow, so it's not a limitation of the OS so much as something janky in the software somewhere.

The audio bug ticket is #4463.

comment:26 by Thilo Borgmann, 9 years ago

Reproduced by developer: set

I can reproduce it. There are missing frames and duplicated frames during my testing. However, I'm still not sure what causes these drops.

If I understand it correctly, the patch sent does not fix the issue even partially. I will have a look at it and review it on the ML asap. And I will look at #4463 again and see what this patch achieves there.

So, LordHDL, you possess the ScreenFlow software and it captures the very same resolution at the very same framerate with unique frames?
Can you link to another software capable of that, maybe one that's open-source (or at least free)?

comment:27 by LordHDL, 9 years ago

Correct, ScreenFlow seems to nail both frame and lossless RGB capture (too bad the export options suck). The only other software I've used that gets frames right is Norichan, but that won't help much. It's only usable with a certain capture device and was designed to record interlaced video so that the resulting deinterlaced encoding would be perfect.

Some notable mentions, however: ​CocoaSplit, ​OBS, and ​Syphon Recorder. All do exhibit some drops (Syphon Recorder has a similar problem to ffmpeg), but so far these are the best of the bunch that I've encountered. As far as video capture (not recording), ​CamTwist is also excellent. These are all free. If you're looking to test Syphon Recorder a good complement to it would be ​OpenEmu, as it has Syphon output built-in.

comment:28 by LordHDL, 9 years ago

I recorded more examples that compare the current git master to the patch: ​https://www.dropbox.com/s/p3na7szw6d8ds9r/PatchComparison.tar.xz?dl=0

You'll notice that the audio plays correctly in the patched version, but video behaves the same. Commands and outputs are in the archive for each.

comment:29 by LordHDL, 9 years ago

As per Thilo's request I'm posting my test results for his mirror here. This is what I got: ​https://www.dropbox.com/s/apkdhi4liixgdl8/avf2.tar.xz?dl=0

Full console outputs included in the archive.

Some observations:

  • Recording now forces my device's sample rate to max, which is 192,000 Hz. I'm using Soundflower (2ch) for audio and usually leave it at 48,000.
  • Audio is just all static, probably due to the sample rate thing, although when I use -ar 48000 it's still static.
  • The audio/video queue warning messages are gone.
  • Frame rate capture seems unaffected.

comment:30 by Thilo Borgmann, 9 years ago

For the audio:

I forgot to mentioned there are new options for that.
"-list_audio_formats true" shows the formats available.
"-audio_format_index N" selects the one you want.
I tested with my phone connected to line-in and it works for me with any of the formats given there (which are not marked as unsupported).
Please post the output of -list_audio_formats here and try some another formats for recording - static (silence) seems to be just another problem I did only encounter with microphone so far.

For the video:

I don't see any problems in the first few hundred frames, visually checked.
However, there are duplicated frames like 790-791. This looks to me that the applications window was not redrawn by the window server. I encounter similar behaviour when capturing the entire screen where some parts were redrawn while the 60fps application shows the very same pixels. The PTS timestamps from the Apple provided sample buffers show different values in these cases. Thus I think that it is not the case that ffmpeg is loosing frames for latency somewhere.

To prove this assumption, another capture of more than one 60fps application should be helpful. Is this application you're using somehting I can download for free somewhere (are you actually playing or is it a self-running demo)? I would like to capture it next to PixelToy to be sure that there is nothing more we could do here.

comment:31 by Thilo Borgmann, 9 years ago

I found it non-free on Steam so I can't test with that.
My tests with PixelToy validate my window server assumption.
I would appreciate it if you would reevaluate using PixelToy (free at Mac App Store, the snowfall set works pretty well). The best setup would be to have them running next to each other, capture the screen and see, if there are pixel updates in either of them while the other one has a duplicated frame.

For the audio part I will have another look at higher bitrates than my current devices offer. I would also appreciate to see the format listing of you audio input devices.

Version 1, edited 9 years ago by Thilo Borgmann (previous) (next) (diff)

comment:32 by Thilo Borgmann, 9 years ago

I tested with Soundflower and it seems like not being capable of doing more than 96kHz during passthrough - means it is not the avfoundation device to blame for 192kHz not working.

I will polish up the patch on github and post it to devel asap (means next week, unfortunately).

Thanks.

comment:33 by LordHDL, 9 years ago

I am not trying to get 192 kHz working, I am using 48 but ffmpeg is forcing it to change to 192. It only does this in the build from the mirror.

comment:34 by Thilo Borgmann, 9 years ago

Did you try to use -audio_format_index N to chose a 49kHz format? Does this work for you?

Fro me, FFmpeg choses the first format available if no index is given which corresponds to 192kHz for the SoundFlower device. It sounds like FFmpeg resets the format within the SoundFlower audio configuration for you?

comment:35 by fthiery, 9 years ago

For what it's worth, the single file patch here solves the issue in our case, but not the bigger overhaul 3-part patch that is suggested in #4513 and available here: ​http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/203959/focus=203966

comment:36 by Thilo Borgmann, 9 years ago

Thanks for the feedback!

The overhaul patch is going to be void soon since we're currently trying to get a common codebase with Libav. This will unfortunately postpone a real update on this device for some time.

However, thanks to your feedback, I will reevaluate again the corresponding tickets for the avfoundation device.

Thanks!

in reply to:  36 comment:37 by LordHDL, 9 years ago

Replying to thilo.borgmann:

Thanks for the feedback!

The overhaul patch is going to be void soon since we're currently trying to get a common codebase with Libav. This will unfortunately postpone a real update on this device for some time.

However, thanks to your feedback, I will reevaluate again the corresponding tickets for the avfoundation device.

Thanks!

Is there currently any way to test? I am unfamiliar with Libav. I'm also curious if this problem (this ticket) is the result of limitations within AVFoundation itself. Thanks.

comment:38 by Thilo Borgmann, 9 years ago

Unfortunately there has been no progress for a long time.
You can expect me to get back to the overhaul patch soon.

comment:39 by LordHDL, 9 years ago

Came across something curious in a recent test. I did a lossless RGB recording at 720p and when played back in ffplay the FPS doesn't seem smooth (as much as usual, anyway). However when played back in VLC it seems fine. Still has the intermittent drops but the rest was smooth. Here's an example: ​https://www.dropbox.com/s/du4bxjltwi0uae3/720p60.mkv?dl=0

comment:40 by LordHDL, 8 years ago

I know this issue is on the back burner for now but I have some more information that might be helpful to you guys. I have a couple recorded tests using Pixel Toy here: ​http://puu.sh/pX4FU/8119e38474.xz

Included are the console outputs and screen shots that show the write speed during the screen capture as well as what my SSD is capable of.

Now here's another tidbit that might be a factor. I have an app called RefreshRate that displays the rate of the GPU currently selected. I forget where I got it but I've uploaded it here: ​http://puu.sh/pX4GV/594298bee7.tar

According to that, my AMD Radeon 6750M has a rate of 59.901460. I'm not sure if this is playing a part but there it is just in case.

comment:41 by LordHDL, 8 years ago

All right so I have yet more information that might be helpful. I have these 2 videos recorded via Syphon Recorder (couldn't use PixelToy because it didn't work with Syphon Inject):

(Ignore registration prompt and download directly.)

​https://www.dropbox.com/s/2wg1hwydxomdgaa/60.mkv?dl=0
​https://www.dropbox.com/s/l56iuma9hrz9uum/AFAP.mkv?dl=0

The first one is very similar to the videos I get when recording using ffmpeg. The second is perfectly smooth. The difference is in choosing a number frame rate versus "As Fast As Possible" in the preferences. Here are their average rates respectively:

https://puu.sh/rgumQ/57d2f5abed.png
https://puu.sh/rgumk/7e86c11bd3.png

The source in question does not run higher than 60 FPS anyway, so choosing "As Fast As Possible" nets every possible frame it can deliver, resulting in a perfect video.

Is there a way to set ffmpeg to record "as fast as possible" with AVFoundation screen capture? I think that may be a good workaround for this issue for the time being.

comment:42 by Eugene Ware, 8 years ago

Hi Thilo, I know this on the backburner, but do you have an estimate on when you're likely to get time to get back to this?

There's quite a few projects like ​https://github.com/wulkano/aperture.js which could be using ffmpeg and thus produce a platform-independent screen recording code-base which are having to fork their code for OS X and use Swift/Objective-C directly to get a quality screen recording with audio which is a shame for both performance reasons and also the audio glitching.

Hope everything is going well with the code-base merge. Good luck, and thanks so much for your incredible contributions. My mind was blown when I discovered last week that this was even possible!

comment:43 by Thilo Borgmann, 8 years ago

Expect me to continue this year.
Unfortunately, I can't make any more promises.

comment:44 by LordHDL, 8 years ago

I wasn't aware at the time of posting this ticket, but I've recently learned about VFR, CFR, and how they relate to each other. Based on my testing with other software, it seems I can only capture every frame properly when recording VFR, as CFR will guarantee that frames are lost as the sources in question (contrary to what I previously thought) are not a constant rate. Their advertised rate is 60 but in practice the FPS always fluctuates, making CFR unreliable for recording.

I believe recording with the native input frame rate as variable might be the solution (as it was in other software) but I haven't been able to test it. I tried -vsync 0 (input, output, and also both at once) but the resulting video is always considered constant. How do I tell ffmpeg to use VFR for input (and really that should be the default behavior)?

comment:45 by Thilo Borgmann, 8 years ago

I'm note sure if this actually comes into the game here. If so, I would why frame drops only happen so "rarely".

comment:46 by Thiago A. Correa, 5 years ago

It's not just the video frames that has drops but the audio as well (see #4513), and then it's very noticeable. Been trying to find a solution to the problem for a week in a program using ffmpeg as a library and only applying this patch that I was able to hear intelligible audio. Unfortunately after the commit with support for raw video this patch doesn't apply cleanly anymore on top of HEAD.

in reply to:  description comment:47 by bgreen965, 5 years ago

Priority: normal → important

This is still an issue on 4.3. Is there an update for the patch?The bug causes audio dropouts that make AVFoundation audio unusable.

comment:48 by Carl Eugen Hoyos, 5 years ago

Priority: important → normal

comment:49 by varenc, 3 years ago

For those struggling with the avfoundation audio stuttering issue, I resolved it just using the older ffmpeg 4.2-tessus to record. That predates this ticket so clearly YMMV. Specifically I was getting audio stuttering with avfoundation only when using a virtual audio loopback device. (Loopback/Soundflower/etc)

comment:50 by varenc, 3 years ago

For those struggling with the avfoundation audio stuttering issue, I resolved it just using the older ffmpeg 4.2-tessus to record. That predates this ticket so clearly YMMV. Specifically I was getting audio stuttering with avfoundation only when using a virtual audio loopback device. (Loopback/Soundflower/etc)

Note: See TracTickets for help on using tickets.