Incorrect video framerate due to no timestamp interpolation for H.264/HEVC

[Andriy Gelman]

Summary of the bug:

The uploaded video is playing at low frame rate, due to missing timestamps after mpegts demuxer.

How to reproduce:

➜  ffmpeg git:(master) ✗ ffplay -vf showinfo example_missing_frames.ts 
ffplay version n4.1.3 Copyright (c) 2003-2019 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20181127
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[hevc @ 0x7f7c80005d40] PPS id out of range: 00KB sq=    0B f=0/0   
    Last message repeated 8 times
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #1.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[hevc @ 0x7f7c80005d40] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x7f7c80005d40] Error parsing NAL unit #0.
[mpegts @ 0x7f7c80000b80] start time for stream 1 is not set in estimate_timings_from_pts
Input #0, mpegts, from 'example_missing_frames.ts':
  Duration: 00:00:53.70, start: 9275.578422, bitrate: 8 kb/s
  Program 1 
    Stream #0:0[0x100]: Video: hevc (Main) (HEVC / 0x43564548), yuv420p(tv), 426x240, 8 fps, 0.83 tbr, 90k tbn, 8 tbc
    Stream #0:1[0x101]: Data: bin_data ([6][0][0][0] / 0x0006)
[hevc @ 0x7f7c80008400] Could not find ref with POC -2
[hevc @ 0x7f7c80008400] Could not find ref with POC -3
[hevc @ 0x7f7c80008400] Could not find ref with POC -4
[Parsed_showinfo_0 @ 0x7f7c74044780] config in time_base: 1/90000, frame_rate: 5/6
[Parsed_showinfo_0 @ 0x7f7c74044780] config out time_base: 0/0, frame_rate: 0/0
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   0 pts:834802058 pts_time:9275.58 pos:      940 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:F87EA414 plane_checksum:[C3D6CB92 8545EF8F 0723E8D5] mean:[128 128 128] stdev:[2.5 0.3 0.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   1 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:15308A58 plane_checksum:[F42FA775 2215F9DD 4415E8E8] mean:[128 128 128] stdev:[3.2 1.2 0.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   2 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:D2F676F4 plane_checksum:[60709564 11BDF889 BEC4E8E9] mean:[128 128 128] stdev:[4.3 1.1 0.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   3 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:C20F85DD plane_checksum:[4660A9CB E3D4F18D 70B8EA67] mean:[128 128 128] stdev:[5.4 1.1 0.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   4 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:5C24987E plane_checksum:[8642BC7C E3D4F18D 7F71EA57] mean:[128 128 128] stdev:[6.4 1.1 0.8]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   5 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:3023D784 plane_checksum:[C05DFA5C 25B6F17B 5EF2EB8F] mean:[128 128 128] stdev:[6.6 1.2 0.8]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   6 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:5049A677 plane_checksum:[9C7DC958 7868F199 A8A4EB68] mean:[128 128 128] stdev:[7.9 1.2 0.8]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   7 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:CAF1926E plane_checksum:[8203C1CE 0A39F802 500AD880] mean:[128 128 128] stdev:[9.2 1.8 1.6]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   8 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:47D2D960 plane_checksum:[D29E08F8 1D60F497 A0A7DBC2] mean:[128 128 128] stdev:[9.4 1.3 1.8]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:   9 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:B039E32F plane_checksum:[BF42BAEA 0C75F820 8B333016] mean:[70 138 103] stdev:[38.3 6.7 16.2]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  10 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:5CC7DAFC plane_checksum:[D96DABD7 43F70144 BAA52DE1] mean:[70 138 103] stdev:[38.3 6.5 16.2]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  11 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:ADE6A6A2 plane_checksum:[D2697330 803E0B19 4D1B2859] mean:[70 139 103] stdev:[38.1 6.4 16.2]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  12 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:F568AFB0 plane_checksum:[49347A84 0E5C0C68 34F528C4] mean:[70 139 103] stdev:[38.0 6.4 16.2]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  13 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:A7002363 plane_checksum:[18F8EDF2 3EE00C37 DF2A292B] mean:[70 139 103] stdev:[38.2 6.4 16.2]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  14 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:A9A64F56 plane_checksum:[AF2F180D 700C18E2 D6A51E67] mean:[70 139 103] stdev:[38.3 6.2 16.1]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  15 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:75018734 plane_checksum:[D4BD4C1B 74081608 32462511] mean:[70 139 103] stdev:[38.9 6.2 16.0]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  16 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:61883693 plane_checksum:[F8BC0376 048A1993 1DE8198A] mean:[71 139 103] stdev:[39.2 6.2 16.0]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  17 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:D300F4F7 plane_checksum:[DB1CC37E 2F4C1C94 7FF514E5] mean:[71 139 103] stdev:[39.9 6.1 16.1]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  18 pts:835029233 pts_time:9278.1  pos:     4700 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:D869C4DF plane_checksum:[55C38C73 16933601 DC71026B] mean:[73 139 103] stdev:[40.5 6.1 15.9]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  19 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:4A803B32 plane_checksum:[3E700EF4 82223B2E D400F101] mean:[76 139 102] stdev:[41.1 6.1 15.9]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  20 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:81D8DCCB plane_checksum:[4B23D1CF 04A94989 BE66C164] mean:[78 139 102] stdev:[41.7 6.1 15.9]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  21 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:6195A409 plane_checksum:[C0F8B2AE 2E2456D7 C71B9A75] mean:[82 139 102] stdev:[42.9 6.1 15.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  22 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:802698B1 plane_checksum:[A5CB9122 3D6A6ED3 218198AD] mean:[89 140 102] stdev:[44.7 6.0 14.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  23 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:948F9177 plane_checksum:[D3B89C39 5D2B6E24 2A47870B] mean:[94 140 101] stdev:[46.0 5.9 13.9]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  24 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:34A167F4 plane_checksum:[F8E78A6A 8352709F BF5F6CDC] mean:[99 140 101] stdev:[47.7 5.9 13.6]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  25 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:0749E276 plane_checksum:[D1822F35 416B4317 1B66702A] mean:[105 139 101] stdev:[49.2 5.9 13.0]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  26 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:3ABE60A9 plane_checksum:[C77EA2AC DF913B9C 9D1D8252] mean:[114 139 101] stdev:[51.3 5.3 12.6]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  27 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:90807773 plane_checksum:[779CC8B6 6A701579 A3969935] mean:[119 139 102] stdev:[52.1 4.8 12.1]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  28 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:3C4BD642 plane_checksum:[AFC4720F 423EAB89 5DBAB89B] mean:[123 138 102] stdev:[52.6 4.8 11.7]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  29 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:C9C15CD5 plane_checksum:[359EDC0D 2CBEB1F6 9941CEB4] mean:[127 138 102] stdev:[53.6 4.7 11.5]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
[Parsed_showinfo_0 @ 0x7f7c74044780] n:  30 pts:  NOPTS pts_time:NOPTS   pos:       -1 fmt:yuv420p sar:0/1 s:426x240 i:P iskey:0 type:P checksum:1A2068F2 plane_checksum:[1B26E0B3 6C36AD04 3133DB1D] mean:[130 138 102] stdev:[54.6 4.7 11.3]
[Parsed_showinfo_0 @ 0x7f7c74044780] color_range:tv color_space:unknown color_primaries:unknown color_trc:unknown
9289.52 M-V: -0.000 fd=   3 aq=    0KB vq=    5KB sq=    0B f=0/0  

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Carl Eugen Hoyos]

Is the issue not reproducible with ffmpeg, only with ffplay?
Please test current FFmpeg git head, nothing else is supported on this bug tracker.

[Andriy Gelman]

I've tested on git master and observed the same.

The issue is that the call to av_read_frame returns many NAL packets with unknown timestamps on this input. I assume that different muxers (if the stream is going to be re-muxed) will handle this differently.
But in the above case ffplay does not handle missing timestamps.

I've sent in a patch that fixes the problem for this dataset:
​http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/243877.html

[Carl Eugen Hoyos]

Is this ticket related to #4813? Is it really the demuxers job to fill in the missing information?

[Andriy Gelman]

Replying to cehoyos:

Is this ticket related to #4813? Is it really the demuxers job to fill in the missing information?

I'll have a look at #4813
Does the mpegts standard specify whether each decoded PES must have a valid timestamp?

[Andriy Gelman]

Replying to cehoyos:

Is this ticket related to #4813? Is it really the demuxers job to fill in the missing information?

I looked at #4813

The problem is that the raw demuxer for hevc and h264 streams does not provide any timestamps which mpegts needs.
This means that we cannot remux any h264 and hevc streams with the copy option... actually not just for mpegts but matrsoka, too.
Of course, we do not have the same problem during transcoding because the encoder outputs pts/dts timestamps.
My initial patch was targeting only the mpegts demuxer (​http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/243877.html), but this seems a bigger issue to me.

I feel the only way to properly get pts/dts timestamps without transcoding is to read the POC from each NAL. Otherwise, I do not see how to deal with b-frames.

What are your thoughts?