Opened 7 months ago
Last modified 7 months ago
#11034 new defect
Stream copy with -use_editlist 0 introduces delay by default, causes A/V desync without warning
Reported by: | pchu | Owned by: | |
---|---|---|---|
Priority: | normal | Component: | avformat |
Version: | git-master | Keywords: | mp4 editlist |
Cc: | MasterQuestionable | Blocked By: | |
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
Summary of the bug:
At first, I was trying to fix some mp4 files' problematic editlists (preferably get rid of them), only to find that if I extract and remux the audio/video streams, the resulting mp4 files still and always have an editlist.
Later, I tested on mp4 files that don't have editlists in the first place. It seems that FFmpeg copy-codec introduces editlists by default, unless force -use_editlist 0
.
I notice that, with the default -use_editlist auto
option, ffprobe -v trace out.mp4
shows a non-zero media_time in the video stream editlist, meaning that A/V would desync (if FFmpeg really copies the streams without any modification, which is not true as I'll show below).
Instead, with -use_editlist 0
option, ffprobe -show_streams out.mp4
shows a non-zero start_pts
and start_time
in the video stream, again meaning that A/V would desync (which is real).
How to reproduce:
ffmpeg -i "elst_test.mp4" -codec copy -copyts -copytb 1 -fps_mode passthrough -avoid_negative_ts disabled "out_with_elst.mp4" ffmpeg -i "elst_test.mp4" -codec copy -copyts -copytb 1 -fps_mode passthrough -avoid_negative_ts disabled -use_editlist 0 "out_without_elst.mp4"
All those options -copyts -copytb 1 -fps_mode passthrough -avoid_negative_ts disabled
are me trying to tell FFmpeg not to process input timestamps, although they have no effect. The output files are the same with or without these options. The resulting files always have timestamps modified.
I inspect the first frames/packets with ffprobe -select_streams v -read_intervals 0%0.5 -show_packets elst_test.mp4
and out.mp4
, with and without -ignore_editlist 1
option. To summarize, their timestamps are:
% elst_test.mp4 (no editlist): % time_base=1/600, start_pts=0 frame I P B B B ... DTS -40 -20 0 20 40 ... CTS 0 80 40 20 60 ... PTS 0 80 40 20 60 ... % out_with_elst.mp4 (with an editlist saying media_time=1280): % time_base=1/19200, start_pts=0 frame I P B B B ... DTS 0 640 1280 1920 2560 ... (not counting editlist shift) CTS 1280 3840 2560 1920 3200 ... (not counting editlist shift) PTS 0 2560 1280 640 1920 ... (counting editlist shift, A/V synched) % out_without_elst.mp4 (no editlist): % time_base=1/19200, start_pts=1280 frame I P B B B ... DTS 0 640 1280 1920 2560 ... CTS 1280 3840 2560 1920 3200 ... PTS 1280 3840 2560 1920 3200 ... (out-of-sync by 1280/19200 seconds)
Finally, I googled a lot and was able to narrow down the reason why FFmpeg insisted on shifting all timestamps forward and making start_pts non-zero : version 0 of the CTTS box doesn't accept negative CTS-DTS offsets. With this constraint, out_without_elst.mp4
has to start at PTS = 1280.
I realized at this point that ffprobe -show_packets
did not report 'correct' or 'raw' DTS:
% elst_test.mp4 (no editlist): % time_base=1/600, start_pts=0 frame I P B B B ... DTS 0 20 40 60 80 ... ('raw' DTS from STTS box) CTS 0 80 40 20 60 ... (calculated from DTS and CTTS box) DTS -40 -20 0 20 40 ... (then shift DTS by 40) <- reported by ffprobe
My workaround to produce no-delay v1 CTTS video:
ffmpeg -i "elst_test.mp4" -codec copy -movflags +negative_cts_offsets "out_with_elst_CTTSv1.mp4" ffmpeg -i "elst_test.mp4" -codec copy -movflags +negative_cts_offsets -use_editlist 0 "out_without_elst_CTTSv1.mp4"
But there're still problems:
- No warning is shown when FFmpeg chooses to follow v0 CTTS box specification and introduces unwanted delays.
- The only method
-movflags +negative_cts_offsets
that allows FFmpeg to use v1 spec (to keep input timestamps untouched) is not documented/mentioned neither under-copyts
option section nor under-vsync/-fps_mode
or-avoid_negative_ts
option sections, so that I had to dig deeper to find it. - I also wonder why
-avoid_negative_ts disabled
does not imply the use of v1 spec (which is also the input file timestamps spec! ). Why-copyts
does not imply using the same version of CTTS box in output as in input file? - When no other options except for
-use_editlist 0
is given, shouldn't the default behaviour be using v1 CTTS (same as input) with a "using-v1" warning shown in cmdline, instead of using v0 CTTS with an "A/V-desync" warning? - ISO/IEC 14496-12:2015(E) explicitely states that "The DT axis has a zero origin". Then why
ffprobe -show_packets
deliberately shifts DTS to a non-zero origin in order to keep CTS-DTS offsets non-negative, pretending to be a v0 CTTS box (which is not)? - When
negative_cts_offsets
flag is set,out_with_elst_CTTSv1.mp4
no longer needs an editlist to shift its start_pts, but the implicit option-use_editlist auto
still creates an editlist for no reason, with a media_time = 0. What's the point? -copytb 1
does not stop FFmpeg from modifying thetime_base=1/600 -> 1/19200
. How can I stop this and why is it doing so?
Attachments (2)
Change History (3)
by , 7 months ago
Attachment: | ffmpeg-20240527-193020.log added |
---|
by , 7 months ago
Attachment: | elst_test_sample100KB.mp4 added |
---|
dd if=elst_test.mp4 of=elst_test_sample100KB.mp4 bs=1024 count=100
comment:1 by , 7 months ago
Cc: | added |
---|---|
Component: | undetermined → avformat |
Keywords: | mp4 editlist added |
͏ There are potentially implementation problems with "-copyts" ͏"-avoid_negative_ts":
͏ Seemingly overlapping.
͏ I do not yet have the time to further analyze.
͏ See also:
͏ https://trac.ffmpeg.org/wiki/Seeking?action=diff&version=43
͏ https://trac.ffmpeg.org/ticket/11007
͏ As for the spec-compliance: I fear many of the specs are illy-defined.
͏ See:
͏ https://trac.ffmpeg.org/ticket/2325#comment:24
͏ https://trac.ffmpeg.org/ticket/5309#comment:16
͏ https://trac.ffmpeg.org/wiki/WikiFormatting#Images ("MasterQuestionable")
͏ https://trac.ffmpeg.org/attachment/wiki/WikiFormatting/raw.txt
ffmpeg -v 9 -loglevel 99 -i elst_test.mp4 -report