Jumping frames (wrong presentation order) in QT for avi(h264)->mp4 streamcopy (missing h264 pts interpolation code)

[Alex]

Description: Jumping frames when copying video of an h264 file
Version: latest trunk
URL: see below

Steps that will reproduce the problem?
ffmpeg -i sample.avi -vcodec copy sample-copy.m4v

What is the expected result?
Converted file w/o any issues.

What happens instead?
The frames "jump" in the converted file when playing it with QuickTimePlayer X 10.1 (more) or VLC (less).

Possible workaround:
ffmpeg -i sample.avi sample-copy.m4v

Any additional information:

• Example: see below
• Command:

  $ ffmpeg -y -i sample.avi -vcodec copy sample-copy.m4v
  ffmpeg version N-30168-g6d789f5, Copyright (c) 2000-2011 the FFmpeg developers
    built on Sep 23 2011 09:20:16 with clang 2.1 (tags/Apple/clang-163.7.1)
    configuration: --enable-zlib --disable-debug --enable-libfaac --enable-libx264 --enable-gpl --enable-nonfree --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libxvid --cc=clang
    libavutil    51. 17. 0 / 51. 17. 0
    libavcodec   53. 17. 0 / 53. 17. 0
    libavformat  53. 13. 0 / 53. 13. 0
    libavdevice  53.  4. 0 / 53.  4. 0
    libavfilter   2. 43. 3 /  2. 43. 3
    libswscale    2.  1. 0 /  2.  1. 0
    libpostproc  51.  2. 0 / 51.  2. 0
  Input #0, avi, from 'sample.avi':
    Metadata:
      encoder         : transcode-1.0.7rc1
    Duration: 02:02:15.00, start: 0.000000, bitrate: 0 kb/s
      Stream #0.0: Video: h264 (Main) (H264 / 0x34363248), yuv420p, 720x576 [SAR 16:15 DAR 4:3], 25 fps, 25 tbr, 25 tbn, 50 tbc
      Stream #0.1: Audio: mp3 (U[0][0][0] / 0x0055), 48000 Hz, stereo, s16, 192 kb/s
  Output #0, ipod, to 'sample-copy.m4v':
    Metadata:
      encoder         : Lavf53.13.0
      Stream #0.0: Video: h264 (avc1 / 0x31637661), yuv420p, 720x576 [SAR 16:15 DAR 4:3], q=2-31, 25 tbn, 25 tbc
      Stream #0.1: Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, s16, 128 kb/s
  Stream mapping:
    Stream #0.0 -> #0.0 (copy)
    Stream #0.1 -> #0.1 (mp3 -> libfaac)
  Press [q] to stop, [?] for help
  [ipod @ 0x7ff8f32d1c00] pts has no value
      Last message repeated 64 times
  frame=   65 fps=  0 q=-1.0 Lsize=     479kB time=00:00:02.60 bitrate=1509.5kbits/s    
  video:436kB audio:40kB global headers:0kB muxing overhead 0.564932%
  

[Alex]

• ​http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/sample-issue-502.avi
• ​http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/sample-copy-issue-502.m4v
• ​http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/sample-recode-issue-502.m4v

[Carl Eugen Hoyos]

Not reproducible with MPlayer and ffplay.

[Alex]

Too bad, so it's Apple specific again. Also I've to admin that VLC plays the file w/o issues. Here is a screencast example:

• ​http://dl.dropbox.com/u/1358485/bugs/ffmpeg/502/screencast-issue-502.m4v

[Alex]

Why is it invalid? I described in detail that this issue is valid, does really exist, and also added example files.

[reimar]

I am not entirely sure, and the "[ipod @ 0x7ff8f32d1c00] pts has no value" is suspicious, however if only QuickTime has issues it might just be a QuickTime bug.
It certainly does mean that most developers will not be able to reproduce it. So you'll have to explain what "jumping" exactly means.
If it means that it plays at irregular speed, that might be something wrong in FFmpeg, if however it means the frames play out of order or decode completely wrong that is most likely a bug in the QuickTime H.264 decoder and you'll just have to re-encode. In that case you should be able to also see the issue when playing the AVI file (I think there are plugins that should allow playing it directly in QuickTime).
I'd like to ask developers to avoid changing bug state without writing a few words though.

[Alex]

Thank you for the feedback.

It certainly does mean that most developers will not be able to reproduce it.

I can start an old VM to test it using Windows with QuickTime.

So you'll have to explain what "jumping" exactly means.

Therefore I added the above linked screencast.

if however it means the frames play out of order

I guess this is the case.

that is most likely a bug in the QuickTime H.264 decoder

Hard to say which fault it is: wrong encoding with a resilient decoder (VLC) or correct encoding with a bug in the decoder (QuickTime).

In that case you should be able to also see the issue when playing the AVI file (I think there are plugins that should allow playing it directly in QuickTime).

The AVI file plays fine.

I'd like to ask developers to avoid changing bug state without writing a few words though.

If you need any other feedback, please let me know.

[reimar]

Looks like a reorder issue. I guess it's possible that FFmpeg mixes up pts and dts and then QuickTime uses those time stamps as they are even when they contradict the decoder reordering, though it seems like a strange design.

[RyanS]

I am seeing this when using ffmpeg to mux to mp4 with: ffmpeg -i x.264 -vcodec copy x.mp4
I am using the windows (zeranoe) build of ffmpeg:

ffmpeg version N-45279-g1a104bf Copyright (c) 2000-2012 the FFmpeg developers

built on Oct 10 2012 19:21:13 with gcc 4.7.2 (GCC)

After this ffmpeg mux, Quicktime plays the frames back out of order. It is presenting frames in decode order rather than presentation order. It does this for two different h.264 encoders I am working with, so does not appear to be an h.264 encoder issue. This only happens around B frames, since those cause the difference between decode and presentation order.

It does seem like pts's are wrong. Perhaps pts is simply ascending despite frames being in decode order.

[Carl Eugen Hoyos]

This is at least missing complete, uncut console output (together with your command line) and probably a sample, but also some indication that what you claim (wrong pts) is true, ffprobe allows to print pts for each packet.

[RyanS]

My command line and ffmpeg version output was in my prior post. My complete console output does not appear to be substantively different from the original bug report, but I can include it if you think it is worth the extra post size.

Note that I do not see the symptom in VLC (Windows version 2.0.1) just Quicktime.

Using ffprobe, it looks like my speculation was backwards. The ffmpeg-muxed mp4 has pts's that are not in order, they match the coded_picture_number order. However, an mp4box-muxed mp4 (that quicktime plays correctly) has pts's that are simply ascending. Here is ffprobe -show_frames output for the mp4box mp4 that plays correctly (notice coded picture numbers 7,9,8):

[FRAME]
media_type=video
key_frame=0
pkt_pts=32032
pkt_pts_time=0.444889
pkt_dts=36036
pkt_dts_time=0.500500
pkt_duration=4004
pkt_duration_time=0.055611
pkt_pos=107727
width=720
height=480
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=P
coded_picture_number=7
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=3
[FRAME]
[FRAME]
media_type=video
key_frame=0
pkt_pts=36036
pkt_pts_time=0.500500
pkt_dts=40040
pkt_dts_time=0.556111
pkt_duration=4004
pkt_duration_time=0.055611
pkt_pos=128869
width=720
height=480
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=B
coded_picture_number=9
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=0
[FRAME]
[FRAME]
media_type=video
key_frame=0
pkt_pts=40040
pkt_pts_time=0.556111
pkt_dts=44044
pkt_dts_time=0.611722
pkt_duration=4004
pkt_duration_time=0.055611
pkt_pos=117491
width=720
height=480
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=P
coded_picture_number=8
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=3
[FRAME]

I notice that the pts is always < dts, and that pts is always equal to prior frame's dts (in stream order).

[Carl Eugen Hoyos]

Replying to RyanS:

My command line and ffmpeg version output was in my prior post. My complete console output does not appear to be substantively different from the original bug report, but I can include it if you think it is worth the extra post size.

I would say it is always required when reopening a ticket, but see below.

[...]

I notice that the pts is always < dts, and that pts is always equal to prior frame's dts (in stream order).

Do you agree that the ticket can be closed as invalid or do I miss something?
(Consider testing vanilla WMP, if you are unable, please point me to a sample.)

[RyanS]

My last comment was not clear. I think ffmpeg mux is getting the pts's wrong. Here is ffprobe -show_frames on the ffmpeg-muxed mp4 from the same x.264 file:

[FRAME]
key_frame=0
pkt_pts=448399
pkt_dts=581865
pkt_duration=66733
pkt_pos=106296
pict_type=P
coded_picture_number=7
reference=3
[FRAME]
[FRAME]
key_frame=0
pkt_pts=581865
pkt_dts=648599
pkt_duration=66734
pkt_pos=127438
pict_type=B
coded_picture_number=9
reference=0
[FRAME]
[FRAME]
key_frame=0
pkt_pts=515132
pkt_dts=715332
pkt_duration=66733
pkt_pos=116060
pict_type=P
coded_picture_number=8
reference=3
[FRAME]
(I cut a bunch of lines that are irrelevant for space.)

You can see here from pkt_pos that the frames are in the same order here as they were above in the mp4box mux. (Note that the pkt_pos values are offset by 1431 bytes probably because the different muxers put in a different header or something, but still the correspondence is clear.) So the frames here appear in the correct presentation order. So the pts values should simply be ascending, just like in the mp4box mux. But they aren't. So it looks like ffmpeg correctly re-orders the frames to presentation order but doesn't set the pts's correctly.

If that is true, then the playback behavior would make sense if Quicktime pays attention to the pts's and ffplay/vlc doesn't. ffplay/vlc may just get the frames from libavformat/libavcodec and play them in the order received, because the decoder is supposed to (and does) put them in presentation order.

Note that because I'm using ffprobe to inspect the mp4's, and ffprobe uses the same libs as ffmpeg, there's some question about what is being done by the mux step vs by the ffprobe decode step. But given the other evidence I think it's the mux step that has the problem.

Here is the x.264 file I'm using:
​https://dl.dropbox.com/u/62496791/x.264

[martinr92]

I've got the same problem with my video files.
If you need more video-samples, let me know.

[Michael Niedermayer]

avi doesnt have pts
libavformat isnt able to (reliably) calculate pts values for h264 in avi
thus no pts values are provided to the muxer

To fix this someone has to implement more complete pts generation support for h264. It would require changes in the H264 parser, see h222 (mpeg systems) and h264 spec. PTS could amongth other things be calculated from POC values and or various SEIs, some of this is implemented since a long time some not.
Iam happy to help/mentor this if theres some volunteer who wants to work on it

[Selur]

btw. using ​http://dl.free.fr/bvisTe0mD
(AVCHD mts from a Sony DCR-CX115E)
and

ffmpeg -i test_mvt.mts -vcodec copy -acodec pcm_s16be test_mvt.mov

I experienced the same problem, with the stuttering output.
Here's the ffmpeg output

ffmpeg version git-2014-07-30-ff9a154 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 30 2014 13:45:30 with gcc 4.9.1 (Rev1, Built by MSYS2 project)
  configuration: --arch=x86_64 --disable-debug --disable-shared --disable-doc --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avfilter --enable-bzlib --enable-zlib --enable-librtmp --enable-gnutls --enable-avisynth --enable-frei0r --enable-filter=frei0r --enable-libbluray --enable-libcaca --enable-libopenjpeg --enable-fontconfig --enable-libfreetype --enable-libass --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libschroedinger --enable-libsoxr --enable-libtwolame --enable-libutvideo --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvo-aacenc --enable-openal --enable-libopus --enable-libvidstab --enable-libvpx --enable-libwavpack --enable-libxavs --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzvbi
  libavutil      52. 93.100 / 52. 93.100
  libavcodec     55. 71.100 / 55. 71.100
  libavformat    55. 50.100 / 55. 50.100
  libavdevice    55. 13.102 / 55. 13.102
  libavfilter     4. 11.102 /  4. 11.102
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mpegts, from 'c:\Users\Selur\Desktop\test_mvt.mts':
  Duration: 00:00:06.24, start: 1.040000, bitrate: 17014 kb/s
  Program 1 
    Stream #0:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x1100]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 256 kb/s
    Stream #0:2[0x1200]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
Output #0, mov, to 'h:\Output\test_mvt.mov':
  Metadata:
    encoder         : Lavf55.50.100
    Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 25 fps, 90k tbn, 90k tbc
    Stream #0:1: Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
      encoder         : Lavc55.71.100 pcm_s16be
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (ac3 (native) -> pcm_s16be (native))
Press [q] to stop, [?] for help
[mov @ 000000000066e680] pts has no value
    Last message repeated 155 times
frame=  312 fps=0.0 q=-1.0 Lsize=   13215kB time=00:00:06.24 bitrate=17349.1kbits/s    

video:12038kB audio:1170kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.053760%

when using ffmbc instead of ffmpeg, I got:

FFmbc version 0.7-rc8
Copyright (c) 2008-2013 Baptiste Coudurier and the FFmpeg developers
Input #0, mpegts, from 'test_mvt.mts':
  Duration: 00:00:06.27, bitrate: 16922 kb/s
  Program 1
    Stream #0.0[0x1011](und): Video: h264 (High), yuv420p, 1920x1080i tff [PAR 1:1 DAR 16:9], 25.00 fps
    Stream #0.1[0x1100](und): Audio: ac3, 48000 Hz, stereo, s16, 256 kb/s
    Stream #0.2[0x1200](und): Subtitle: pgssub
[pcm_s16be @ 03bfc640] The requested thread algorithm is not supported with this thread library.
Output #0, mov, to 'test_mvt.mov':
  Metadata:
    encoder: FFmbc 0.7
    Stream #0.0(und): Video: h264, yuv420p, 1920x1080i tff [PAR 1:1 DAR 16:9], stream copy, 25.00 fps
    Stream #0.1(und): Audio: pcm_s16be, 48000 Hz, stereo, s16, 1536 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1
Press [q] to stop, [?] for help
[mov @ 04b314a0] color primaries unspecified, assuming bt709
[mov @ 04b314a0] replacing header in front, copying 12.90MB
frame=  156 fps=2500 Lsize=   13212kB time=00:00:06.24 bitrate=17344.4kbits/s
video:12038kB audio:1170kB global headers:0kB muxing overhead 0.026529%

and the output seems to be fine.

thought about reporting the problem and found this thread, so not sure if it helps, but for some users, using ffmbc might be a working workaround.
(since ffmbc is based on an older ffmpeg version, this might be a regression bug,..)

[Elon Musk]

This is regression since removal of pts guessing code.

[dustinkerstein]

Are there any updates on this issue? I am able to reproduce this when demuxing using h264_mp4toannexb and then remuxing to mp4 on any file that contains B frames.

[Balling]

Will be fixed in ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220830014653.2477-1-jamrial@gmail.com/

Wow.

[James]

Commit b92e14147f8efb32f3dbbb112dbaa5a5ab9bcd03 addresses this with a new bitstream filter to reorder DTS.

[Balling]

ffmpeg -i sample-issue-502.avi -c:v copy -bsf:v dts2pts sample-copy2.mp4

does indeed fix quicktime.