Wrong first frame using new decode API

[s0m3]

I'm developping a multimedia player based on ffmpeg.

To do frame stepping, I need to associate frame numbers and frame times (timecodes).
If AVStream::nb_index_entries is != 0, I use AVIndexEntry structure, otherwise, I build an "index" manually.

Building the "index" is as simple as decoding frames until we reach the end of the file. When decoding frames, I can retrieve frame times (pts) and store them.

My issue is that the first frame I get is not the first one, its pts is not 0, its the (correct) pts of the third frame so I miss 2 frames.

I suspect frames content are correct (what is displayed) but their pts is not. It's hard to tell because first frames of the sample file are identical.

I use the new decoding API so it may be a bug there or I may not use it correctly.

Anyway, a test case is provided.
The sample is "simpsons_movie_1080p_trailer.zip" from ​http://www.h264info.com/clips.html. MD5 is ff37f121ea26b09c926926a6ba737194

I tested it on windows with ffmpeg v3.2.4 and git version 20170404-1229007-win32 (win32 build)

[s0m3]

Test case to trigger the bug

[s0m3]

The test case decodes and exports (as BMP files) all frames of the specified video file.
The first exported frame is "0_834166.bmp" instead of "0_0.bmp".
The second number "831466" is the pts in 100ns unit.

I know from a dedicated software (elecard StreamEye) this is the pts of frame 3, not 1.

[Carl Eugen Hoyos]

What time stamps are shown by ffprobe for the first few frames?

[s0m3]

D:\>"D:\ffmpeg-3.2.4-win32-shared\bin\ffprobe.exe" -i "D:\Samples\The Simpsons Movie - 1080p Trailer.mp4"
ffprobe version 3.2.4 Copyright (c) 2007-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
<snip>
[mov,mp4,m4a,3gp,3g2,mj2 @ 004772e0] stream 0, timescale not set
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\Samples\The Simpsons Movie - 1080p Trailer.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2007-07-17T09:18:37.000000Z
    genre           : Trailer
    artist          : Fox
    title           : The Simpsons Movie
    date            : 2007
  Duration: 00:02:17.25, start: 0.000000, bitrate: 8591 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x800,
 8486 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default)
    Metadata:
      creation_time   : 2007-07-17T09:18:37.000000Z
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, 5.1, fltp,
107 kb/s (default)
    Metadata:
      creation_time   : 2007-07-17T09:18:45.000000Z
      handler_name    : GPAC ISO Audio Handler
    Stream #0:2: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 101x150 [S
AR 72:72 DAR 101:150], 90k tbr, 90k tbn, 90k tbc

The file has 3 streams. Stream #0:0 is the video part, #0:1 the audio part and #0:2 is an embedded picture ("cover art" as VLC calls it).

D:\>"D:\ffmpeg-3.2.4-win32-shared\bin\ffprobe.exe" -i "D:\Samples\The Simpsons Movie - 1080p Trailer.mp4" -show_frames
<output cut to only keep 4 frames of stream 0>
[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=2002
pkt_pts_time=0.083417
pkt_dts=2002
pkt_dts_time=0.083417
best_effort_timestamp=2002
best_effort_timestamp_time=0.083417
pkt_duration=1001
pkt_duration_time=0.041708
pkt_pos=71545
pkt_size=65391
<snip>
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=3003
pkt_pts_time=0.125125
pkt_dts=3003
pkt_dts_time=0.125125
best_effort_timestamp=3003
best_effort_timestamp_time=0.125125
pkt_duration=1001
pkt_duration_time=0.041708
pkt_pos=137164
pkt_size=56
<snip>
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=4004
pkt_pts_time=0.166833
pkt_dts=4004
pkt_dts_time=0.166833
best_effort_timestamp=4004
best_effort_timestamp_time=0.166833
pkt_duration=1001
pkt_duration_time=0.041708
pkt_pos=137103
pkt_size=61
<snip>
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=5005
pkt_pts_time=0.208542
pkt_dts=5005
pkt_dts_time=0.208542
best_effort_timestamp=5005
best_effort_timestamp_time=0.208542
pkt_duration=1001
pkt_duration_time=0.041708
pkt_pos=136936
pkt_size=167
<snip>

So on ffmpeg side, start time is 0.083417 as shown with the following excerpt:

D:\>"D:\ffmpeg-3.2.4-win32-shared\bin\ffprobe.exe" -i "D:\Samples\The Simpsons Movie - 1080p Trailer.mp4" -show_streams
[STREAM]
index=0
<snip>
timecode=N/A
<snip>
start_pts=2002
start_time=0.083417
<snip>

But with StreamEye v3.1, I have:

stream=0, display=0, size=65391, type=I, key=yes, offset=0x11779, time=00:00:00.000
stream=1, display=3, size=  167, type=P, key= no, offset=0x216e8, time=00:00:00.125
stream=2, display=2, size=   61, type=B, key= no, offset=0x2178f, time=00:00:00.083
stream=3, display=1, size=   56, type=B, key= no, offset=0x217cc, time=00:00:00.041

If you look at size and offset of each frame, we're talking about the same ones but their timestamps differ.
ffmpeg start timecode is frame 3 timestamp, not the first one (if StreamEye is correct).

I do think decoded frames are ok but their associated timestamp is not.
My question is: who's right? Is first frame timestamp 0 or 0.083417 ?

[Carl Eugen Hoyos]

Please test current FFmpeg git head to make this a valid ticket.

[s0m3]

I compiled ffmpeg from Git on my Ubuntu box.
I tried compiling my own testcase but it took me ages to get it working because I didn't want to install this ffmpeg version system wide.

The first BMP file is 0_417083.bmp and second one is 1_834166.bmp so it's better in git but still not correct.

[Carl Eugen Hoyos]

Replying to s0m3:

I compiled ffmpeg from Git on my Ubuntu box.
I tried compiling my own testcase but it took me ages to get it working because I didn't want to install this ffmpeg version system wide.

But didn't the ffprobe output you provided confirm that if there is an issue, it is reproducible with FFmpeg itself and doesn't need another testcase or do I misunderstand?

[Hendrik]

There is definitely no video frame at time = 0 in that stream (there is an audio frame there, though), the first video frame I can see is at 1001 stream time, so the Git output you mention seems to be correct.