When copying video from .avi to .mp4, frames have incorrect pts values

[Misaki]

Step 1, generate a video with numbered frames:

ffmpeg -filter_complex "color=white:160x160:1,drawtext='fontcolor=black:fontsize=96:x=(W-tw)/2+4:y=(H-th)/2+4:text=%{n}:alpha=0.5',boxblur,drawtext='fontcolor=black:fontsize=96:x=(W-tw)/2:y=(H-th)/2:text=%{n}'" -preset veryslow -refs 8 -aq-strength 0.5 -threads 1 -crf 20 -frames:v 30 -movflags faststart ordered-frames.mp4

Step 2, convert it to avi:

ffmpeg -i ordered-frames.mp4 -c copy ordered-frames.avi

Step 3, convert back to mp4:

ffmpeg -i ordered-frames.avi -c copy -movflags faststart ordered-frames-avicopy.mp4

Examine the frames using ffprobe:

ffprobe -hide_banner -show_entries frame=pkt_pts_time,pkt_dts_time,pkt_size,coded_picture_number ordered-frames.mp4 -select_streams v | less

In another terminal tab, compare the new mp4:

ffprobe -hide_banner -show_entries frame=pkt_pts_time,pkt_dts_time,pkt_size,coded_picture_number ordered-frames-avicopy.mp4 -select_streams v | less

The pkt_dts_time values are just two more than the original file, but the pts values are now completely different. They are the packet dts values, which are different from the frame's pkt_dts values. This can be seen by looking at the packets and looking for similar sizes:

ffprobe -hide_banner -show_entries packet=pts_time,dts_time,size ordered-frames-avicopy.mp4 -select_streams v | less

Could also make arrays and by doing so sort everything by size so you wouldn't have to search to compare packets to frames, but it's easy to look for two or three packets.

Most programs seem to effectively ignore the pts for a frame shown by ffprobe, because they pay attention to the dts for a frame, which is different from the packet dts but usually the same as the frame pts and the packet pts.

So the 'bugged' mp4 files, copied from avi, appear to have a delay of a couple frames at most at the start, but then play normally with maybe a slight audio-video sync mismatch.

But it appears that the Chrome browser and Chrome OS's video player do care about the pts values, because they read the frames earlier than the dts values. This results in stuttering output: ​https://bugs.chromium.org/p/chromium/issues/detail?id=793635

They stutter the same way with the original avi file. This is probably unrelated to any ffmpeg libraries, but it does seem possible to make the copied mp4 files have better timestamps and so play correctly in Chrome.

[Misaki]

Converting from avi gives the error message, [mp4 @ 0x5572f6242e80] pts has no value (Last message repeated 29 times).

(I don't know if '-vsync drop' is working correctly. When I try to use it, it gives the error 'av_interleaved_write_frame(): Invalid argument' and 'Conversion failed!', whether codec is copy or x264, though x264 only stops after reading a little more than rc_lookahead frames. Not making a new bug report unless sure it's a bug.)

But anyway, it seems that the mp4 muxer is probably giving pts values to the packets that don't have any, based on the packet order instead of the frame order.

In all cases, it seems like the packet's 'pts' is the same as the frame's 'pkt_pts', even though for normal files this is also the same as the frame's 'pkt_dts', while in the bugged files it's the same as the packet's 'dts', which is different from the frame's 'pkt_dts'.

If the mp4 muxer is writing these pts values for the frame, it should be possible for it to assume pts values based on the frame order (or using the frame dts), not the packet order.

This gives a consistent explanation for the stuttering in Chrome: for an avi file, Chrome is making up pts values based on the packet order. These just happen to be the same as the dts values for those packets (which is different from the dts values for the frames contained in the packets, as reported by ffprobe). The mp4 muxer component of ffmpeg makes up pts values if frames don't have them, and it also does this based on the packet order instead of the frame order.

[Carl Eugen Hoyos]

Replying to Misaki:

ffmpeg -i ordered-frames.mp4 -c copy ordered-frames.avi

Why is this supposed to produce a working file?

[Misaki]

What do you mean by working file? Many programs play it back just fine. ffplay does it normally the first time; if you seek back to the start, frame 0 does display for an extra 2~3 seconds. totem plays it completely normally. vlc has a short delay, plays it normally, then starts dropping frames because of "more than 5 seconds of late video".

(Edit: Note that playback behavior is slightly different between ordered-frames.avi and ordered-frames-avicopy.mp4; common programs are using the frame (not packet) dts values for avicopy.mp4, which are a bit different from those in the .avi. Specifically, the second frame is displayed longer; in totem, the 0 is played for a long time, then the 1 is shown for little or no time, then it plays 'normally'; if you seek back to the start, it does play normally. But in ffplay, both the 0 and 1 are displayed for extra time, with the 0 still getting more time if you seek back to start. vlc is just two seconds 'late', no dropped frames; not sure if vlc apparently requiring two frames as a buffer is related.)

ffmpeg doesn't give any warnings or errors for that command.

Using MP4Box, part of the 'gpac' package, I was able to get a non-bugged mp4 that doesn't stutter in Chrome OS. That program doesn't seem to accept the h.264 (avc1) codec, but it's still an interim solution for someone wishing to convert old avis (using mpeg4 codec) to mp4s that won't stutter in Chrome. ​https://bugs.chromium.org/p/chromium/issues/attachment?aid=315836&inline=1

[Carl Eugen Hoyos]

I suggest we shorten this discussion: H.264 timestamps are not always correct with FFmpeg, this is a known limitation, it is apparently very difficult to fix, several tickets exist.
H.264 in avi may have issues, this may be unrelated though.

[Misaki]

Mpeg4 streams work the same way. The reasons for using h.264 were that I was sure it would lead to B-pyramids and unpredictable frame order, and also that it would work in browsers like Chrome on Linux so verifying an error would be easier.

As I said, MP4Box does it correctly. I don't know how accurate ffprobe is, but if it is accurate, then ffmpeg just needs to generate pts's for a frame that are the same as the dts's that already exist for that frame, when converting an avi to mp4.

There's also the issue of timestamps being bumped up when converting to and from avi; in a 30-fps file this isn't noticeable, but at 1 fps it was more so. Although the packet dts as reported by ffprobe started from 0, the frames' pkt_dts_time starts at 2 for the ordered-frames.avi file (pkt_dts=4). Even if .avi can't have negative timestamps, could still start at 0 instead of higher than that. The original mp4 does start at 0 dts and pts for frames.

May I ask why the bug was closed? There's nothing wrong with leaving a bug open if no one is willing to fix it.

[Misaki]

The workaround, MP4Box, stopped working on episode 6 when the encoder switched from XVID to DX50. The resulting file was half the size and had lots of missing frames (and pts not matching dts for frames). Playback was fine for the first 12 seconds, with no missing frames.

Copying using ffmpeg starting at 13 seconds to another avi, and then copying that to mp4 using MP4Box, led to good output. Two videos had been combined, with different encoding parameters. For the original, MP4Box indicated 'Has B-Frames (8 max consecutive B-VOPs)'. For the video starting at the keyframe before 13 sec, it said 'Has B-Frames (2 max consecutive B-VOPs)'. (no packed bitstream.)

This information is for anyone trying to get this to work when ffmpeg is bugged.

[Carl Eugen Hoyos]

Replying to Misaki:

Mpeg4 streams work the same way.

Then please explain how to reproduce and reopen.

Does -fflags +genpts make a difference?

[Misaki]

[Edit] When used as an input option, it does generate pts and there's no warning messages, but the generated pts are just 1 second (at 1 fps) higher than those generated by mp4 with the warning message. (dts is the same, so pts are all 1 second higher than dts with -show_packets.) It's still using the packet pts to generate timestamps, rather than the frame dts as reported by ffprobe. I don't know if the 'frame dts' for .avi format is used the same way as 'frame pts' is used for other formats like mp4, or if accessing the frame dts for an avi requires some decoding while it's available without decoding for an mp4; basically whether the problems that ffmpeg and Chrome are experiencing are historical and due to the concept of 'pts' possibly arising after 'dts' had been around for some time. 'pts' and 'dts' are both not in Wiktionary.

Originally:
I didn't know about that option, but it does not make a difference. The warning messages '[mp4 @ 0x5569f707bea0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly' and '[mp4 @ 0x5569f707bea0] pts has no value' x30 for the test file are still there, and timestamps in the output file are the same as without the option.

Reproducing is easy, just copy the video stream in any avi with B-frames to mp4 format. Confirming that the output is such that it will stutter in Chrome, even though it has almost no problems in most players, is more difficult, and that's where it helps to start with an mp4 so you can see that the start and end files are clearly different.

Symptom is that if examined with ffprobe, a stutter-prone mp4 will have same dts as pts with -show_packets, while normal mp4s have same with -show_frames. (.avi's will also stutter in Chome specifically because there are no pts values, or it should treat the frame dts as the frame pts but doesn't, or something.)