Video Sync Support

[villmer]

Synchronously acquiring two camera streams (such as V4L2) is critical for both recording (and live streaming of) stereoscopic video and, in our case, 360° video (using two back-to-back cameras). Both 360° video and stereo video utilize ffmpeg framepack filter to place the two streams side-by-side.

Unfortunately, the synchronous acquisition of two V4L2 video streams is currently not possible with ffmpeg. While two streams can be requested, there is always a delay of at least ~1second. Despite using various filters, such as itsoffset, setpts etc, we have been unable to synchronize two streams. As the ffmpeg team has already taken steps implementing various 360° video support, we would like to see dual-stream synchronization capabilities added to ffmpeg. In this way, ffmpeg can be directly used to create 360° video - instead of just performing post-production effects to it.

Our current ffmpeg pipeline to capture two V4L2 streams and one alsa audio stream looks like this:

ffmpeg -y \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20 -thread_queue_size 32k -i /dev/video0 \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20 -thread_queue_size 32k -i /dev/video1 \
-f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 32k -async 1 -itsoffset 0.3 -i hw:1,0 \
-filter_complex "[0:v][1:v]framepack=sbs" \
-f mp4 \
-c:v libx264 \
-b:v 2M \
-b:a 128k \
-preset ultrafast \
-pix_fmt yuv420p \
video.mp4

The code above successfully records two V4L2 videos into a side-by-side formation, synchronously merges one alsa audio stream (via -itsoffset 0.3) and exports everything out to an mp4 file on-the-fly!

The ONLY (unsolvable) problem is the ~1 second lag between videos. We are, quite literally, one second from a workable solution. However, removing this issue has not been successful despite many days of effort.

On the Raspberry Pi (Compute Module 4) we can use the Raspivid application to capture frame-accurate, synchronous video from two cameras @ 30fps by simply adding the -3d sbs tag. We believe ffmpeg oils potentially use similar techniques / code to synchronize the videos with a modified V4L2 driver. We have a new post on the Raspberry Pi forums describing this: ​https://www.raspberrypi.org/forums/viewtopic.php?f=61&t=290197&p=1754295#p1754295

We are attaching a file showing the results of the code above. As you see, FFMPEG successfully places two (circular) videos side-by-side. The ONLY issue is the ~1second time lag.

[mkver]

Duplicate of #8982.