Opened 18 months ago
Last modified 18 months ago
#11525 reopened enhancement
Better frame dropping logic for live streams to avoid unjustifiable freezing?
| Reported by: | Kirill A. Korinsky | Owned by: | |
|---|---|---|---|
| Priority: | normal | Component: | avdevice |
| Version: | git-master | Keywords: | video4linux2 |
| Cc: | MasterQuestionable | Blocked By: | |
| Blocking: | Reproduced by developer: | no | |
| Analyzed by developer: | yes |
Description
Summary of the bug:
When I use Elgato Facecam Pro is frozen an attempt to consume 60fps stream from this device lead to frozen picture. 30 fps and adding -noframedrop helps, but the last one introduced a significant delay for couple of seconds.
I had captured two video on my laptop on OpenBSD from this device as:
tmp $ ffmpeg -input_format mjpeg -video_size 3840x2160 -framerate 30 -i /dev/video1 -codec copy -t 10 framerate-30.mp4
ffmpeg version 6.1.2 Copyright (c) 2000-2024 the FFmpeg developers
built with OpenBSD clang version 16.0.6
configuration: --enable-shared --arch=amd64 --cc=cc --cxx=c++ --enable-debug --disable-stripping --disable-indev=jack --disable-outdev=sdl2 --disable-vulk
an --enable-fontconfig --enable-frei0r --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libdav1d --enable-libfontconfig --enable-libfre
etype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-libopus --enable-libspeex --enable-libtheora --enable-libv4l2 --enable-libvorbis --en
able-libvpx --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nonfree --enable-openssl --enable-libvidstab --ext
ra-cflags='-I/usr/local/include -I/usr/X11R6/include' --extra-libs='-L/usr/local/lib -L/usr/X11R6/lib' --extra-ldsoflags= --mandir=/usr/local/man --objcc=/u
sr/bin/false --optflags='-O2 -pipe -g -Wno-redundant-decls'
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
Input #0, video4linux2,v4l2, from '/dev/video1':
Duration: N/A, start: 50567.670002, bitrate: N/A
Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 3840x2160 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1000k tbn
Output #0, mp4, to 'framerate-30.mp4':
Metadata:
encoder : Lavf60.16.100
Stream #0:0: Video: mjpeg (Baseline) (mp4v / 0x7634706D), yuvj420p(pc, bt470bg/unknown/unknown), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 30 tbr, 100
0k tbn
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[out#0/mp4 @ 0xb2000de8700] video:98822kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.004377%
size= 98826kB time=00:00:09.98 bitrate=81120.5kbits/s speed= 1x
tmp $
tmp $ ffmpeg -input_format mjpeg -video_size 3840x2160 -framerate 60 -i /dev/video1 -codec copy -t 10 framerate-60.mp4
ffmpeg version 6.1.2 Copyright (c) 2000-2024 the FFmpeg developers
built with OpenBSD clang version 16.0.6
configuration: --enable-shared --arch=amd64 --cc=cc --cxx=c++ --enable-debug --disable-stripping --disable-indev=jack --disable-outdev=sdl2 --disable-vulk
an --enable-fontconfig --enable-frei0r --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libdav1d --enable-libfontconfig --enable-libfre
etype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-libopus --enable-libspeex --enable-libtheora --enable-libv4l2 --enable-libvorbis --en
able-libvpx --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nonfree --enable-openssl --enable-libvidstab --ext
ra-cflags='-I/usr/local/include -I/usr/X11R6/include' --extra-libs='-L/usr/local/lib -L/usr/X11R6/lib' --extra-ldsoflags= --mandir=/usr/local/man --objcc=/u
sr/bin/false --optflags='-O2 -pipe -g -Wno-redundant-decls'
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
Input #0, video4linux2,v4l2, from '/dev/video1':
Duration: N/A, start: 50588.050002, bitrate: N/A
Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 3840x2160 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 1000k tbn
Output #0, mp4, to 'framerate-60.mp4':
Metadata:
encoder : Lavf60.16.100
Stream #0:0: Video: mjpeg (Baseline) (mp4v / 0x7634706D), yuvj420p(pc, bt470bg/unknown/unknown), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 60 fps, 60 tbr, 100
0k tbn
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[out#0/mp4 @ 0x90d2c605500] video:168313kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.004573%
size= 168321kB time=00:00:09.99 bitrate=138026.3kbits/s speed= 1x
tmp $
Here files:
- https://kirill.korins.ky/pub/ffmpeg-framerate-30.mp4
- https://kirill.korins.ky/pub/ffmpeg-framerate-60.mp4
I had tried Linux as well with the same result, with ffmpeg 7.1.
And on the same setup Jabra PanaCast 20 produces same 4K 60fps strem which doesn't create any issue.
Attachments (1)
Change History (24)
comment:1 by , 18 months ago
| Cc: | added |
|---|---|
| Component: | undetermined → avdevice |
| Keywords: | video4linux2 added |
comment:2 by , 18 months ago
- The same device works on macOS laptop, I haven't tried ffmpeg, but it works with elgato cam hub, and in Safari.
- OpenBSD and Linux has two different implementation of V4L2 stack, I doubt that both of them has the same issue.
Also, this webcam supports both isoc and bulk endpoint, and such behaviour can be observed via both endpoints.
And, finally, I had tried mplayer and it works on OpenSBD.
I had run it as: mplayer tv:// -tv driver=v4l2:device=/dev/video1:width=3840:height=2160:fps=60
and mplayer playes ffmpeg-framerate-60.mp4 as well.
All of this smells like a bug in ffplay / ffmpeg.
comment:3 by , 18 months ago
| Analyzed by developer: | set |
|---|
͏ Probably indeed problems in FFmpeg code.
͏ Need device author's assistance.
͏ Try running with "-v debug" for potentially more info.
comment:4 by , 18 months ago
Here an output of:
ffplay -autoexit -v debug ffmpeg-framerate-60.mp4: https://kirill.korins.ky/pub/ffmpeg-framerate-60.mp4-debug.logffplay -autoexit -noframedrop -v debug ffmpeg-framerate-60.mp4: https://kirill.korins.ky/pub/ffmpeg-framerate-60.mp4-noframedrop-debug.log
In the first case video is frozen at "27,34" on the clock, on the second case it works but slower than it should.
I doubt that the issue is device itself because mplayer and ffplay -noframedrop works as expected from a file which I had capture before by ffmpeg.
comment:5 by , 18 months ago
͏ "-v debug" with the capture command...
͏ Playing the pre-captured MP4 is pointless.
comment:6 by , 18 months ago
Here an output of env AV_LOG_FORCE_NOCOLOR=1 ffplay -input_format mjpeg -video_size 3840x2160 -v debug -framerate 60 -i /dev/video1 2>/tmp/ffplay-framerate-60.log https://kirill.korins.ky/pub/ffplay-framerate-60.log
Also, I not sure but it can be related I had suceffuly achieve crash of ffplay / ffmpeg when I detach device from USB port. It isn't reproduced each detach but often enough and I can't reproduce it by different device.
Stacktrace is here: https://marc.info/?l=openbsd-ports&m=174248074730073&w=2
comment:7 by , 18 months ago
Interesting, if I switch my laptop's CPU to the high performance... it plays without an issue.
comment:8 by , 18 months ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
That pretty much means your CPU is too weak to decode mjpeg of that quality in real time.
Try one of the various hardware decoders that support mjpeg.
In any case, not a bug.
comment:9 by , 18 months ago
Not sure that "to weak".
I had ping point the threshold. 1500 Mhz isn't enough, and 1600 Mhz is enough.
comment:10 by , 18 months ago
͏ Seems it's the OS didn't switch to High Performance on need?
͏ Log content of interest:
[mjpeg] Found EOI before any SOF, ignoring
[mjpeg] marker parser used 0 bytes (0 bits)
[mjpeg] No JPEG data found in image
͏ Better-looking link of previous mail:
͏ https://mail-archive.com/ports@openbsd.org/msg132177.html
comment:11 by , 18 months ago
OpenBSD switches to the High Performance on need.
Interesting that on this machine ffplay loads it to 25%..30% CPU and manual switching to high performance doesn't solve the issue each time. Usually it does but at least once (right now) it doesn't help.
Thus, manually switch to the high performance doesn't unfreeze the video.
comment:12 by , 18 months ago
͏ Feels like OS/environment bug.
͏ Further monitor the CPU utilization meanwhile? (including frequency stats)
comment:13 by , 18 months ago
MasterQuestionable, if it is only OS/env bug, when the picture should unfreeze as soon as I enforce the high speed in CPU, isn't it?
Right now I see that if I start streaming by ffplay with CPU speed in 1600 MHz or more, it works, but as soon as I switched to slower speed... the picture is frozen, and switch back to higher CPU speed does not unfreeze the picture.
It really looks like that ffplay / ffmpeg has an issue on this stream, and some missed frames due to "slow CPU" leads to frozen picture completley, and -noframedrop prevents it.
comment:14 by , 18 months ago
| Summary: | 60fps stream from Elgato Facecam Pro is frozen → Better frame dropping logic for live streams to avoid unjustifiable freezing? |
|---|
͏ Is alike reproducible with other input sources?
comment:15 by , 18 months ago
| Resolution: | invalid |
|---|---|
| Status: | closed → reopened |
| Type: | defect → enhancement |
| Version: | unspecified → git-master |
͏ Would you try further down-frequency the CPU..?
͏ (may cause system instability; or might irrevocable havoc if did too outrageously)
͏ Or running other CPU-heavy software meanwhile to simulate alike?
͏ Or limit max CPU FF-* can use via OS's task management?
comment:16 by , 18 months ago
MasterQuestionable, I do have access only to Jabra PanaCast 20, but it turns that it can produce only 30 fps, and 30 fps doesn't trigger an issue.
comment:17 by , 18 months ago
I mean besides this P010 capture does not work #8454. Patches are there...
comment:18 by , 18 months ago
MasterQuestionable, I see you update but I feel that I don't understnad you.
Used CPU supports only spet by 100 Mhz, so I can't try 1550 Mhz.
Anyway, compiling the kernel allows to reproduce the issue when machine in the high maximal frequency.
comment:20 by , 18 months ago
͏ With CPU restraints on FF-* tools:
͏ Would you reproduce the same problem on 30 FPS? (and different devices?)
comment:21 by , 18 months ago
I had tried 15 fps and 30 fps from both Elgato and Jabra device, and I can confirm that I can reproduce the same issue with threshold 1600 Mhz.
I see the same behaviour on the Fedora Linux and OpenBSD.
comment:22 by , 18 months ago
͏ Is it exactly what described in comment:13?
͏ “missed frames due to "slow CPU" leads to frozen picture completely, and "-noframedrop" prevents it.”
͏ ; and reproducible across devices?
͏ Consult the equipment's manufacturer (or its driver author) perhaps?