Video Sync Support

[villmer]

Synchronously acquiring two camera streams (such as V4L2) is critical for both recording (and live streaming of) stereoscopic video and, in our case, 360° video (using two back-to-back cameras). Both 360° video and stereo video utilize ffmpeg framepack filter to place the two streams side-by-side.

Unfortunately, the synchronous acquisition of two V4L2 video streams is currently not possible with ffmpeg. While two streams can be requested, there is always a delay of at least ~1second. Despite using various filters, such as itsoffset, setpts etc, we have been unable to synchronize two streams. As the ffmpeg team has already taken steps implementing various 360° video support, we would like to see dual-stream synchronization capabilities added to ffmpeg. In this way, ffmpeg can be directly used to create 360° video - instead of just performing post-production effects to it.

Our current ffmpeg pipeline to capture two V4L2 streams and one alsa audio stream looks like this:

ffmpeg -y \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20 -thread_queue_size 32k -i /dev/video0 \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20 -thread_queue_size 32k -i /dev/video1 \
-f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 32k -async 1 -itsoffset 0.3 -i hw:1,0 \
-filter_complex "[0:v][1:v]framepack=sbs" \
-f mp4 \
-c:v libx264 \
-b:v 2M \
-b:a 128k \
-preset ultrafast \
-pix_fmt yuv420p \
video.mp4

The code above successfully records two V4L2 videos into a side-by-side formation, synchronously merges one alsa audio stream (via -itsoffset 0.3) and exports everything out to an mp4 file on-the-fly!

The ONLY (unsolvable) problem is the ~1 second lag between videos. We are, quite literally, one second from a workable solution. However, removing this issue has not been successful despite many days of effort.

On the Raspberry Pi (Compute Module 4) we can use the Raspivid application to capture frame-accurate, synchronous video from two cameras @ 30fps by simply adding the -3d sbs tag. We believe ffmpeg oils potentially use similar techniques / code to synchronize the videos with a modified V4L2 driver. We have a new post on the Raspberry Pi forums describing this: ​https://www.raspberrypi.org/forums/viewtopic.php?f=61&t=290197&p=1754295#p1754295

We are attaching a file showing the results of the code above. As you see, FFMPEG successfully places two (circular) videos side-by-side. The ONLY issue is the ~1second time lag.

[villmer]

Stereo Video using FFMPEG

[villmer]

UPDATED
The following code above produces the following:

Input #0, video4linux2,v4l2, from '/dev/video0':

Duration: N/A, start: 377.919269, bitrate: N/A

Stream #0:0: Video: h264 (High), yuv420p(progressive), 1024x1024, 20 fps, 20 t

br, 1000k tbn, 2000k tbc
Input #1, video4linux2,v4l2, from '/dev/video1':

Duration: N/A, start: 378.792375, bitrate: N/A

Stream #1:0: Video: h264 (High), yuv420p(progressive), 1024x1024, 20 fps, 20 t

br, 1000k tbn, 2000k tbc
Stream mapping:

Stream #0:0 (h264) -> framepack:left
Stream #1:0 (h264) -> framepack:right
framepack -> Stream #0:0 (libx264)

Press [q] to stop, ? for help
hroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0

threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 blura

y_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=20 scenec
ut=0 intra_refresh=0 rc=abr mbtree=0 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 q
pmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to 'video.mp4':

Metadata:

encoder : Lavf58.20.100
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 2048x1024, q=

-1--1, 2000 kb/s, 20 fps, 10240 tbn, 20 tbc (default)

Metadata:

encoder : Lavc58.35.100 libx264

Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1

[libx264 @ 0xa8fd00] non-strictly-monotonic PTS
[mp4 @ 0xa8f3d0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0;

changing to 1. This may result in incorrect timestamps in the output file.

frame= 11 fps=0.0 q=29.0 size= 0kB time=00:00:00.10 bitrate= 3.8kbits/s
frame= 25 fps= 23 q=26.0 size= 0kB time=00:00:00.80 bitrate= 0.5kbits/s
frame= 35 fps= 22 q=24.0 size= 256kB time=00:00:01.30 bitrate=1613.4kbits/s
frame= 45 fps= 21 q=25.0 size= 256kB time=00:00:01.80 bitrate=1165.2kbits/s
frame= 49 fps= 20 q=-1.0 Lsize= 505kB time=00:00:02.35 bitrate=1759.0kbits/s

speed=0.971x

video:504kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing ove
rhead: 0.199724%
[libx264 @ 0xa8fd00] frame I:1 Avg QP:36.00 size: 20426
[libx264 @ 0xa8fd00] frame P:48 Avg QP:26.79 size: 10306
[libx264 @ 0xa8fd00] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0xa8fd00] mb P I16..4: 3.9% 0.0% 0.0% P16..4: 21.1% 0.0% 0.0% 0
.0% 0.0% skip:75.0%
[libx264 @ 0xa8fd00] final ratefactor: 27.71
[libx264 @ 0xa8fd00] coded y,uvDC,uvAC intra: 10.7% 27.0% 1.4% inter: 7.6% 10.0% 0
.2%
[libx264 @ 0xa8fd00] i16 v,h,dc,p: 51% 22% 12% 15%
[libx264 @ 0xa8fd00] i8c dc,h,v,p: 60% 20% 14% 5%
[libx264 @ 0xa8fd00] kb/s:1716.99
Exiting normally, received signal 2.

[villmer]

The output from the terminal shows:
Input #0 has start: 377.919269 while Input #1 has start: 378.792375

• - - -

As there is approximately a ~1 second time lag between the 1st and 2nd inputs, we tried adding a filter to change the 2nd input’s time stamp:

-filter_complex "[1:v]setpts=(PTS-2)/TB[mod];[0:v][mod]framepack=sbs" \

This did not work.

We also tried adding the -vsync option to both streams:

-f v4l2 -input_format h264 -video_size 1024x1024 -framerate 20 -thread_queue_size 32k -vsync 0 -i /dev/video0 \
-f v4l2 -input_format h264 -video_size 1024x1024 -framerate 20 -thread_queue_size 32k -vsync 0 -i /dev/video1 \

This did not work.

[Carl Eugen Hoyos]

This has no similarities with a bug report, please post all user questions on the FFmpeg user mailing list.