Invalid B-frames timestamps importing .h264 or .hevc stream

[slydiman]

ffmpeg calculates invalid timestamps importing .h264 or .hevc stream with B-frames. Note the video decoder will reorder frames and fix the frames timestamps. So it is possible to get the valid file only reencoding the video stream. It is impossible to mux the valid file without reencoding. Note the invalid file may be played correctly if the video decoder is smart enough (MediaPlayerClassic, VLC, etc.). You must playback the invalid H264 MP4 in Google Chrome or QuickTime to see the issue.

Note ffmpeg muxes an invalid file from HEVC B-frames yuv420p10le. But everything seems ok in case of HEVC B-frames yuv422p10le with the same GOP. Google Chrome cannot playback the file hevc_ffmpeg_copy_yuv422p10le_ok.mp4, so I tested this file in Adobe Premiere Pro 2022.

How to reproduce (latest ffmpeg with Lavf60.5.100):

% ffmpeg -i b-frames.h264 -c copy h264_ffmpeg_copy_invalid.mp4
% ffmpeg -i b-frames.h264 -c h264 h264_ffmpeg_reencode_valid.mp4
% mp4box -add b-frames.h264 -new h264_mp4box_copy_valid.mp4

% ffmpeg -i b-frames.hevc -c copy hevc_ffmpeg_copy_invalid.mp4
% ffmpeg -i b-frames.hevc -c hevc hevc_ffmpeg_reencode_valid.mp4
% mp4box -add b-frames.hevc -new hevc_mp4box_copy_valid.mp4

Samples:
H264 1920x1080 50fps, High L4.2, yuv420p, recorded by SONY 7SM3
​http://slydiman.me/ffmpeg/b-frames.h264 (12MB)
​http://slydiman.me/ffmpeg/h264_ffmpeg_copy_invalid.mp4 (12MB)
​http://slydiman.me/ffmpeg/h264_ffmpeg_reencode_valid.mp4 (1.4MB)
​http://slydiman.me/ffmpeg/h264_mp4box_copy_valid.mp4 (12MB)

HEVC 4K 23.976fps, Main 10 L5.0, yuv420p10le, recorded by SONY 7M4
​http://slydiman.me/ffmpeg/b-frames.hevc (14MB)
​http://slydiman.me/ffmpeg/hevc_ffmpeg_copy_invalid.mp4 (14MB)
​http://slydiman.me/ffmpeg/hevc_ffmpeg_reencode_valid.mp4 (1.1MB)
​http://slydiman.me/ffmpeg/hevc_mp4box_copy_valid.mp4 (14MB)

HEVC 4K 23.976fps, rext L5.0, yuv422p10le, recorded by SONY 7M4
​http://slydiman.me/ffmpeg/b-frames-yuv422p10le.hevc (13MB)
​http://slydiman.me/ffmpeg/hevc_ffmpeg_copy_yuv422p10le_ok.mp4 (13MB)

[Balling]

You must playback the invalid MP4 in Google Chrome

Yes, Chrome Canary is horrific playback. Still, you were not using last version: Lavf58.51.101 is old. Latest version is a little different:

ffmpeg -i ​http://slydiman.me/ffmpeg/invalid.mp4 -vf vfrdet -f null -

[null @ 0000016fe18d5880] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 99 >= 98
[out#0/null @ 0000016fdfa58480] video:47kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=  100 fps= 26 q=-0.0 Lsize=N/A time=00:00:01.96 bitrate=N/A speed=0.51x
[Parsed_vfrdet_0 @ 0000016fe185e400] VFR:0.030303 (3/96) min: -24000 max: 48000 avg: 16000

while newest version gives this:

ffmpeg -i invli.mp4 -vf vfrdet -f null -

[null @ 000001a403501a00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 98 >= 97
[out#0/null @ 000001a4035e83c0] video:46kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=   99 fps=0.0 q=-0.0 Lsize=N/A time=00:00:01.94 bitrate=N/A speed= 3.4x
[Parsed_vfrdet_0 @ 000001a407079640] VFR:0.020408 (2/96) min: -24000 max: 48000 avg: 0

Also:

ffmpeg -i ​http://slydiman.me/ffmpeg/b-frames.h264 -vf vfrdet -f null -

gives perfect VFR:0.000000 (0/100) but says:

[h264 @ 00000186dc07f8c0] non-existing SPS 0 referenced in buffering period

Extracting ffmpeg -i invli.mp4 -c copy nceaca.264 back gives bitperfect file with original Annex B.

There are many bugs, e.g. just now: ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230505113123.28404-1-jeebjp@gmail.com/

[slydiman]

I have remuxed b-frames.h264 to mp4 using today's ffmpeg with Lavf60.5.100 - nothing changed.
The problem is invalid PTS of B-frames, not DTS.
BTW,

%ffmpeg -i b-frames.h264 -c copy invalid.mp4

[mp4 @ 000001933f6ed800] Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
[mp4 @ 000001933f6ed800] pts has no value
    Last message repeated 100 times

Note this problem is related to most modern cameras SONY including SONY FX3.

It seems ffmpeg can calculate PTS for B-frames if they have been encoded with ffmpeg itself.

The sample ​http://slydiman.me/ffmpeg/b-frames.h264 has been recorded by the camera SONY 7SM3 and GOP is
IIII-BBBB-BBBB-PPPP-BBBB-BBBB-PPPP-... (7 blocks)

I see the following GOP for H265 recorded by SONY FX3
IIIIIIII-BBBBBBBB-BBBBBBBB-PPPPPPPP-BBBBBBBB-BBBBBBBB-PPPPPPPP-... (7 blocks)

Something is unexpected for ffmpeg in such GOPs and ffmpeg just does not calculate PTS at all.

[Gyan]

Have you tried the dts2pts bsf?

[slydiman]

Replying to Gyan:

Have you tried the dts2pts bsf?

I have no idea what is dts2pts. Can you provide some details/instructions?

I tried the latest mp4box and it can remux b-frames.h264 to mp4 just fine (the resulting file can be playback in Chrome without any problems).

[pdr0]

works ok for chrome playback
ffmpeg -f h264 -r 50 -i b-frames.h264 out.mp4

In the old days, you always had to input format and framerate for CFR elementary streams

[slydiman]

Replying to pdr0:

works ok for chrome playback
ffmpeg -f h264 -r 50 -i b-frames.h264 out.mp4

No, this means libx264 encoding by default. It is the equivalent of
%ffmpeg -i b-frames.h264 -c h264 valid.mp4

The problem is that remuxing without reencoding produces an invalid file
%ffmpeg -i b-frames.h264 -c copy invalid.mp4

[Balling]

I have remuxed b-frames.h264 to mp4 using today's ffmpeg with Lavf60.5.100 - nothing changed.

Some timestamps did.

I see the following GOP for H265 recorded by SONY FX3
IIIIIIII-BBBBBBBB-BBBBBBBB-PPPPPPPP-BBBBBBBB-BBBBBBBB-PPPPPPPP-... (7 blocks)

Where is that hevc sample??

BTW, I also used JM 18 reference encoder, it produces the same file (crazy.yuv) with md5 87DFBE705624E76A746F90871B78CEFC as ffmpeg itself.

ldecod.exe -p InputFile=b-frames.h264 -p OutputFile=crazy.yuv

[Balling]

Replying to Gyan:

Have you tried the dts2pts bsf?

ffmpeg -i ​http://slydiman.me/ffmpeg/b-frames.h264 -c:v copy -bsf:v dts2pts dts2ptsstrange.mp4

makes it a little better, but the end is still bad, also, that still does not fix the

[h264 @ 00000186dc07f8c0] non-existing SPS 0 referenced in buffering period

and still gives VFR file (mediainfo app):

Frame rate mode             : Variable
Frame rate                  : 52.604 FPS
Minimum frame rate          : 50.000 FPS
Maximum frame rate          : 1200 000.000 FPS
Original frame rate         : 50.000 FPS

[slydiman]

Replying to Balling:

Where is that hevc sample?

I have updated the description and added
​​http://slydiman.me/ffmpeg/b-frames.hevc

[Balling]

Yep, same thing with hevc. ​http://slydiman.me/ffmpeg/hevc_ffmpeg_copy_invalid.mp4 has horrible playback with Chrome Canary, while ​http://slydiman.me/ffmpeg/hevc_mp4box_copy_valid.mp4 is valid.

Though Chrome still takes time to load ​https://slydiman.me/ffmpeg/hevc_mp4box_copy_valid.mp4 maybe because the file produces these warnings if you -c copy it (or just ffplay it), file after -c copy opens super fast in chrome, so GPAC/mp4box still has some bug there apparently:

[mov,mp4,m4a,3gp,3g2,mj2 @ 000001db95c67ec0] st: 0 edit list: 1 Missing key frame while searching for timestamp: 10010
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001db95c67ec0] st: 0 edit list 1 Cannot find an index entry before timestamp: 10010.

[slydiman]

So, the message like

[mp4 @ 000001933f6ed800] Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
[mp4 @ 000001933f6ed800] pts has no value

during copying is the sign of the problem.
Probably it must be fixed somewhere in ​https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/demux.c

I don't see any problem around the warning non-existing SPS 0 referenced in buffering period. MP4 is playable anyway because SPS and PPS will be inside avcC atom which is read before mdat.

[slydiman]

Note ffmpeg muxes an invalid file from HEVC B-frames yuv420p10le. But everything seems ok in case of HEVC B-frames yuv422p10le with the same GOP. Google Chrome cannot playback the file hevc_ffmpeg_copy_yuv422p10le_ok.mp4, so I tested this file in Adobe Premiere Pro 2022. I have updated the description.

[Balling]

Since you touched this code...

I also opened an issue with ​https://github.com/gpac/gpac/issues/2462

Since its mp4box hevc file is not actually perfect apparently, it warns on it and says

0, 50, 50, 1, 24883200, 6293ededaf5a936d9af4b648a6174c82
0, 51, 51, 1, 24883200, 7786f45387f7e2072f7c25597224bd87
0, 53, 53, 1, 24883200, 44f16bcbaed6644d40680fd9db2f25f3

[Balling]

Apparently hevc 420 10 bit sample has what is called stream access point (SAP) of very rare type SAP 2. See patent: ​https://patentimages.storage.googleapis.com/3c/b6/be/2e367218e89e47/WO2014056435A1.pdf

We do support many of RAPs like CRA (clean random access) but not others like BLA, see ab77b878f1205225c6de1370fb0e998dbcc8bc69 (quote from there: "We should probably add BLA frames as well, but I don't have any sample so I prefered to leave that for later"). But that is again different with SAP 2.

[Balling]

Okay. Got a fix in ​https://github.com/gpac/gpac/commit/fe02e73f7e1e6caf74a826bdd64b49eff1e5bdc5
that fixes SAP 2 in both hevc samples, only for gpac.exe file.hevc inspect:deep

[Balling]

BTW, do you have original mp4 files or whatever Sony uses? I need to look at how they signal editlist and sdtp box.

[slydiman]

Replying to Balling:

Okay. Got a fix in ​https://github.com/gpac/gpac/commit/fe02e73f7e1e6caf74a826bdd64b49eff1e5bdc5 that fixes SAP 2 in both hevc samples, only for gpac.exe file.hevc inspect:deep

Can it help somehow to fix ffmpeg? Usually I don't use gpac. Sometimes mp4box crashes on huge files in the middle of process. But I expected to get ffmpeg fixed.

Replying to Balling:

BTW, do you have original mp4 files or whatever Sony uses? I need to look at how they signal editlist and sdtp box.

Sure
​http://slydiman.me/ffmpeg/sony_hevc_yuv422p10le_4k.mp4

I don't see any sdtp. Note SONY uses MXF-like metadata and fragmented old kkad metadata inside with SPS and many PPS. Note I can decode these metadata. I have not seen specs of these metadata in the public domain.

[Balling]

Please also attach yuv420p10le hevc.mp4

[slydiman]

Replying to Balling:

Please also attach yuv420p10le hevc.mp4

It is huge 5.6GB. I can dump any part if necessary.
​http://slydiman.me/ffmpeg/sony_hevc_yuv420p10le_4k.txt