Opened 4 years ago

Closed 4 years ago

#8982 closed defect (invalid)

Video Sync Support

Reported by: villmer Owned by:
Priority: normal Component: undetermined
Version: unspecified Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Synchronously acquiring two camera streams (such as V4L2) is critical for both recording (and live streaming of) stereoscopic video and, in our case, 360° video (using two back-to-back cameras). Both 360° video and stereo video utilize ffmpeg framepack filter to place the two streams side-by-side.

Unfortunately, the synchronous acquisition of two V4L2 video streams is currently not possible with ffmpeg. While two streams can be requested, there is always a delay of at least ~1second. Despite using various filters, such as itsoffset, setpts etc, we have been unable to synchronize two streams. As the ffmpeg team has already taken steps implementing various 360° video support, we would like to see dual-stream synchronization capabilities added to ffmpeg. In this way, ffmpeg can be directly used to create 360° video - instead of just performing post-production effects to it.

Our current ffmpeg pipeline to capture two V4L2 streams and one alsa audio stream looks like this:

ffmpeg -y \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20 -thread_queue_size 32k -i /dev/video0 \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20 -thread_queue_size 32k -i /dev/video1 \
-f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 32k -async 1 -itsoffset 0.3 -i hw:1,0 \
-filter_complex "[0:v][1:v]framepack=sbs" \
-f mp4 \
-c:v libx264 \
-b:v 2M \
-b:a 128k \
-preset ultrafast \
-pix_fmt yuv420p \

The code above successfully records two V4L2 videos into a side-by-side formation, synchronously merges one alsa audio stream (via -itsoffset 0.3) and exports everything out to an mp4 file on-the-fly!

The ONLY (unsolvable) problem is the ~1 second lag between videos. We are, quite literally, one second from a workable solution. However, removing this issue has not been successful despite many days of effort.

On the Raspberry Pi (Compute Module 4) we can use the Raspivid application to capture frame-accurate, synchronous video from two cameras @ 30fps by simply adding the -3d sbs tag. We believe ffmpeg oils potentially use similar techniques / code to synchronize the videos with a modified V4L2 driver. We have a new post on the Raspberry Pi forums describing this:

We are attaching a file showing the results of the code above. As you see, FFMPEG successfully places two (circular) videos side-by-side. The ONLY issue is the ~1second time lag.

Attachments (1)

FFMPEG-TEST.png (887.4 KB ) - added by villmer 4 years ago.
Stereo Video using FFMPEG

Download all attachments as: .zip

Change History (8)

comment:1 by villmer, 4 years ago

Keywords: Stereo 3d 360 added

comment:2 by villmer, 4 years ago

Keywords: synchronize added
Priority: normalimportant

by villmer, 4 years ago

Attachment: FFMPEG-TEST.png added

Stereo Video using FFMPEG

comment:3 by villmer, 4 years ago

Cc: added

comment:4 by villmer, 4 years ago

Version: unspecifiedgit-master

comment:5 by villmer, 4 years ago

The following code above produces the following:

Input #0, video4linux2,v4l2, from '/dev/video0':

Duration: N/A, start: 377.919269, bitrate: N/A

Stream #0:0: Video: h264 (High), yuv420p(progressive), 1024x1024, 20 fps, 20 t

br, 1000k tbn, 2000k tbc
Input #1, video4linux2,v4l2, from '/dev/video1':

Duration: N/A, start: 378.792375, bitrate: N/A

Stream #1:0: Video: h264 (High), yuv420p(progressive), 1024x1024, 20 fps, 20 t

br, 1000k tbn, 2000k tbc
Stream mapping:

Stream #0:0 (h264) -> framepack:left
Stream #1:0 (h264) -> framepack:right
framepack -> Stream #0:0 (libx264)

Press [q] to stop, ? for help
hroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0

threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 blura

y_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=20 scenec
ut=0 intra_refresh=0 rc=abr mbtree=0 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 q
pmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to 'video.mp4':


encoder : Lavf58.20.100
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 2048x1024, q=

-1--1, 2000 kb/s, 20 fps, 10240 tbn, 20 tbc (default)


encoder : Lavc58.35.100 libx264

Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1

[libx264 @ 0xa8fd00] non-strictly-monotonic PTS
[mp4 @ 0xa8f3d0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0;

changing to 1. This may result in incorrect timestamps in the output file.

frame= 11 fps=0.0 q=29.0 size= 0kB time=00:00:00.10 bitrate= 3.8kbits/s
frame= 25 fps= 23 q=26.0 size= 0kB time=00:00:00.80 bitrate= 0.5kbits/s
frame= 35 fps= 22 q=24.0 size= 256kB time=00:00:01.30 bitrate=1613.4kbits/s
frame= 45 fps= 21 q=25.0 size= 256kB time=00:00:01.80 bitrate=1165.2kbits/s
frame= 49 fps= 20 q=-1.0 Lsize= 505kB time=00:00:02.35 bitrate=1759.0kbits/s


video:504kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing ove
rhead: 0.199724%
[libx264 @ 0xa8fd00] frame I:1 Avg QP:36.00 size: 20426
[libx264 @ 0xa8fd00] frame P:48 Avg QP:26.79 size: 10306
[libx264 @ 0xa8fd00] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0xa8fd00] mb P I16..4: 3.9% 0.0% 0.0% P16..4: 21.1% 0.0% 0.0% 0
.0% 0.0% skip:75.0%
[libx264 @ 0xa8fd00] final ratefactor: 27.71
[libx264 @ 0xa8fd00] coded y,uvDC,uvAC intra: 10.7% 27.0% 1.4% inter: 7.6% 10.0% 0
[libx264 @ 0xa8fd00] i16 v,h,dc,p: 51% 22% 12% 15%
[libx264 @ 0xa8fd00] i8c dc,h,v,p: 60% 20% 14% 5%
[libx264 @ 0xa8fd00] kb/s:1716.99
Exiting normally, received signal 2.

Last edited 4 years ago by villmer (previous) (diff)

comment:6 by villmer, 4 years ago

The output from the terminal shows:
Input #0 has start: 377.919269 while Input #1 has start: 378.792375

  • - - -

As there is approximately a ~1 second time lag between the 1st and 2nd inputs, we tried adding a filter to change the 2nd input’s time stamp:

-filter_complex "[1:v]setpts=(PTS-2)/TB[mod];[0:v][mod]framepack=sbs" \

This did not work.

We also tried adding the -vsync option to both streams:

-f v4l2 -input_format h264 -video_size 1024x1024 -framerate 20 -thread_queue_size 32k -vsync 0 -i /dev/video0 \
-f v4l2 -input_format h264 -video_size 1024x1024 -framerate 20 -thread_queue_size 32k -vsync 0 -i /dev/video1 \

This did not work.

Last edited 4 years ago by villmer (previous) (diff)

comment:7 by Carl Eugen Hoyos, 4 years ago

Keywords: Stereo 3d 360 synchronize removed
Priority: importantnormal
Resolution: invalid
Status: newclosed
Type: enhancementdefect
Version: git-masterunspecified

This has no similarities with a bug report, please post all user questions on the FFmpeg user mailing list.

Note: See TracTickets for help on using tickets.