Opened 13 months ago

Last modified 12 months ago

#10348 open defect

Invalid B-frames timestamps importing .h264 or .hevc stream

Reported by: slydiman Owned by: Sasi Inguva
Priority: important Component: undetermined
Version: unspecified Keywords: frames timestamp h264 hevc
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by slydiman)

ffmpeg calculates invalid timestamps importing .h264 or .hevc stream with B-frames. Note the video decoder will reorder frames and fix the frames timestamps. So it is possible to get the valid file only reencoding the video stream. It is impossible to mux the valid file without reencoding. Note the invalid file may be played correctly if the video decoder is smart enough (MediaPlayerClassic, VLC, etc.). You must playback the invalid H264 MP4 in Google Chrome or QuickTime to see the issue.

Note ffmpeg muxes an invalid file from HEVC B-frames yuv420p10le. But everything seems ok in case of HEVC B-frames yuv422p10le with the same GOP. Google Chrome cannot playback the file hevc_ffmpeg_copy_yuv422p10le_ok.mp4, so I tested this file in Adobe Premiere Pro 2022.

How to reproduce (latest ffmpeg with Lavf60.5.100):

% ffmpeg -i b-frames.h264 -c copy h264_ffmpeg_copy_invalid.mp4
% ffmpeg -i b-frames.h264 -c h264 h264_ffmpeg_reencode_valid.mp4
% mp4box -add b-frames.h264 -new h264_mp4box_copy_valid.mp4

% ffmpeg -i b-frames.hevc -c copy hevc_ffmpeg_copy_invalid.mp4
% ffmpeg -i b-frames.hevc -c hevc hevc_ffmpeg_reencode_valid.mp4
% mp4box -add b-frames.hevc -new hevc_mp4box_copy_valid.mp4

Samples:
H264 1920x1080 50fps, High L4.2, yuv420p, recorded by SONY 7SM3
http://slydiman.me/ffmpeg/b-frames.h264 (12MB)
http://slydiman.me/ffmpeg/h264_ffmpeg_copy_invalid.mp4 (12MB)
http://slydiman.me/ffmpeg/h264_ffmpeg_reencode_valid.mp4 (1.4MB)
http://slydiman.me/ffmpeg/h264_mp4box_copy_valid.mp4 (12MB)

HEVC 4K 23.976fps, Main 10 L5.0, yuv420p10le, recorded by SONY 7M4
http://slydiman.me/ffmpeg/b-frames.hevc (14MB)
http://slydiman.me/ffmpeg/hevc_ffmpeg_copy_invalid.mp4 (14MB)
http://slydiman.me/ffmpeg/hevc_ffmpeg_reencode_valid.mp4 (1.1MB)
http://slydiman.me/ffmpeg/hevc_mp4box_copy_valid.mp4 (14MB)

HEVC 4K 23.976fps, rext L5.0, yuv422p10le, recorded by SONY 7M4
http://slydiman.me/ffmpeg/b-frames-yuv422p10le.hevc (13MB)
http://slydiman.me/ffmpeg/hevc_ffmpeg_copy_yuv422p10le_ok.mp4 (13MB)

Change History (22)

comment:1 by slydiman, 13 months ago

Description: modified (diff)

comment:2 by Balling, 13 months ago

Status: newopen

You must playback the invalid MP4 in Google Chrome

Yes, Chrome Canary is horrific playback. Still, you were not using last version: Lavf58.51.101 is old. Latest version is a little different:

ffmpeg -i http://slydiman.me/ffmpeg/invalid.mp4 -vf vfrdet -f null -

[null @ 0000016fe18d5880] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 99 >= 98
[out#0/null @ 0000016fdfa58480] video:47kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=  100 fps= 26 q=-0.0 Lsize=N/A time=00:00:01.96 bitrate=N/A speed=0.51x
[Parsed_vfrdet_0 @ 0000016fe185e400] VFR:0.030303 (3/96) min: -24000 max: 48000 avg: 16000

while newest version gives this:

ffmpeg -i invli.mp4 -vf vfrdet -f null -

[null @ 000001a403501a00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 98 >= 97
[out#0/null @ 000001a4035e83c0] video:46kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=   99 fps=0.0 q=-0.0 Lsize=N/A time=00:00:01.94 bitrate=N/A speed= 3.4x
[Parsed_vfrdet_0 @ 000001a407079640] VFR:0.020408 (2/96) min: -24000 max: 48000 avg: 0

Also:

ffmpeg -i http://slydiman.me/ffmpeg/b-frames.h264 -vf vfrdet -f null -

gives perfect VFR:0.000000 (0/100) but says:

[h264 @ 00000186dc07f8c0] non-existing SPS 0 referenced in buffering period

Extracting ffmpeg -i invli.mp4 -c copy nceaca.264 back gives bitperfect file with original Annex B.

There are many bugs, e.g. just now: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20230505113123.28404-1-jeebjp@gmail.com/

Last edited 13 months ago by Balling (previous) (diff)

comment:3 by slydiman, 13 months ago

I have remuxed b-frames.h264 to mp4 using today's ffmpeg with Lavf60.5.100 - nothing changed.
The problem is invalid PTS of B-frames, not DTS.
BTW,

%ffmpeg -i b-frames.h264 -c copy invalid.mp4

[mp4 @ 000001933f6ed800] Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
[mp4 @ 000001933f6ed800] pts has no value
    Last message repeated 100 times

Note this problem is related to most modern cameras SONY including SONY FX3.

It seems ffmpeg can calculate PTS for B-frames if they have been encoded with ffmpeg itself.

The sample http://slydiman.me/ffmpeg/b-frames.h264 has been recorded by the camera SONY 7SM3 and GOP is
IIII-BBBB-BBBB-PPPP-BBBB-BBBB-PPPP-... (7 blocks)

I see the following GOP for H265 recorded by SONY FX3
IIIIIIII-BBBBBBBB-BBBBBBBB-PPPPPPPP-BBBBBBBB-BBBBBBBB-PPPPPPPP-... (7 blocks)

Something is unexpected for ffmpeg in such GOPs and ffmpeg just does not calculate PTS at all.

Last edited 13 months ago by slydiman (previous) (diff)

comment:4 by Gyan, 13 months ago

Have you tried the dts2pts bsf?

comment:5 by slydiman, 13 months ago

Replying to Gyan:

Have you tried the dts2pts bsf?

I have no idea what is dts2pts. Can you provide some details/instructions?

I tried the latest mp4box and it can remux b-frames.h264 to mp4 just fine (the resulting file can be playback in Chrome without any problems).

Last edited 13 months ago by slydiman (previous) (diff)

comment:6 by pdr0, 13 months ago

works ok for chrome playback
ffmpeg -f h264 -r 50 -i b-frames.h264 out.mp4

In the old days, you always had to input format and framerate for CFR elementary streams

in reply to:  6 comment:7 by slydiman, 13 months ago

Replying to pdr0:

works ok for chrome playback
ffmpeg -f h264 -r 50 -i b-frames.h264 out.mp4

No, this means libx264 encoding by default. It is the equivalent of
%ffmpeg -i b-frames.h264 -c h264 valid.mp4

The problem is that remuxing without reencoding produces an invalid file
%ffmpeg -i b-frames.h264 -c copy invalid.mp4

comment:8 by Balling, 13 months ago

I have remuxed b-frames.h264 to mp4 using today's ffmpeg with Lavf60.5.100 - nothing changed.

Some timestamps did.

I see the following GOP for H265 recorded by SONY FX3
IIIIIIII-BBBBBBBB-BBBBBBBB-PPPPPPPP-BBBBBBBB-BBBBBBBB-PPPPPPPP-... (7 blocks)

Where is that hevc sample??

BTW, I also used JM 18 reference encoder, it produces the same file (crazy.yuv) with md5 87DFBE705624E76A746F90871B78CEFC as ffmpeg itself.

ldecod.exe -p InputFile=b-frames.h264 -p OutputFile=crazy.yuv

Last edited 13 months ago by Balling (previous) (diff)

in reply to:  4 comment:9 by Balling, 13 months ago

Replying to Gyan:

Have you tried the dts2pts bsf?

ffmpeg -i http://slydiman.me/ffmpeg/b-frames.h264 -c:v copy -bsf:v dts2pts dts2ptsstrange.mp4

makes it a little better, but the end is still bad, also, that still does not fix the

[h264 @ 00000186dc07f8c0] non-existing SPS 0 referenced in buffering period

and still gives VFR file (mediainfo app):

Frame rate mode             : Variable
Frame rate                  : 52.604 FPS
Minimum frame rate          : 50.000 FPS
Maximum frame rate          : 1200 000.000 FPS
Original frame rate         : 50.000 FPS

Last edited 13 months ago by Balling (previous) (diff)

comment:10 by slydiman, 13 months ago

Description: modified (diff)

in reply to:  8 comment:11 by slydiman, 13 months ago

Replying to Balling:

Where is that hevc sample?

I have updated the description and added
http://slydiman.me/ffmpeg/b-frames.hevc

comment:12 by Balling, 13 months ago

Yep, same thing with hevc. http://slydiman.me/ffmpeg/hevc_ffmpeg_copy_invalid.mp4 has horrible playback with Chrome Canary, while http://slydiman.me/ffmpeg/hevc_mp4box_copy_valid.mp4 is valid.

Though Chrome still takes time to load https://slydiman.me/ffmpeg/hevc_mp4box_copy_valid.mp4 maybe because the file produces these warnings if you -c copy it (or just ffplay it), file after -c copy opens super fast in chrome, so GPAC/mp4box still has some bug there apparently:

[mov,mp4,m4a,3gp,3g2,mj2 @ 000001db95c67ec0] st: 0 edit list: 1 Missing key frame while searching for timestamp: 10010
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001db95c67ec0] st: 0 edit list 1 Cannot find an index entry before timestamp: 10010.
Last edited 13 months ago by Balling (previous) (diff)

comment:13 by slydiman, 13 months ago

So, the message like

[mp4 @ 000001933f6ed800] Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
[mp4 @ 000001933f6ed800] pts has no value

during copying is the sign of the problem.
Probably it must be fixed somewhere in https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/demux.c

I don't see any problem around the warning non-existing SPS 0 referenced in buffering period. MP4 is playable anyway because SPS and PPS will be inside avcC atom which is read before mdat.

comment:14 by slydiman, 13 months ago

Description: modified (diff)

comment:15 by slydiman, 13 months ago

Note ffmpeg muxes an invalid file from HEVC B-frames yuv420p10le. But everything seems ok in case of HEVC B-frames yuv422p10le with the same GOP. Google Chrome cannot playback the file hevc_ffmpeg_copy_yuv422p10le_ok.mp4, so I tested this file in Adobe Premiere Pro 2022. I have updated the description.

comment:16 by Balling, 12 months ago

Owner: set to Sasi Inguva

Since you touched this code...

I also opened an issue with https://github.com/gpac/gpac/issues/2462

Since its mp4box hevc file is not actually perfect apparently, it warns on it and says

0, 50, 50, 1, 24883200, 6293ededaf5a936d9af4b648a6174c82
0, 51, 51, 1, 24883200, 7786f45387f7e2072f7c25597224bd87
0, 53, 53, 1, 24883200, 44f16bcbaed6644d40680fd9db2f25f3

comment:17 by Balling, 12 months ago

Apparently hevc 420 10 bit sample has what is called stream access point (SAP) of very rare type SAP 2. See patent: https://patentimages.storage.googleapis.com/3c/b6/be/2e367218e89e47/WO2014056435A1.pdf

We do support many of RAPs like CRA (clean random access) but not others like BLA, see ab77b878f1205225c6de1370fb0e998dbcc8bc69 (quote from there: "We should probably add BLA frames as well, but I don't have any sample so I prefered to leave that for later"). But that is again different with SAP 2.

Last edited 12 months ago by Balling (previous) (diff)

comment:18 by Balling, 12 months ago

Okay. Got a fix in https://github.com/gpac/gpac/commit/fe02e73f7e1e6caf74a826bdd64b49eff1e5bdc5
that fixes SAP 2 in both hevc samples, only for gpac.exe file.hevc inspect:deep

Last edited 12 months ago by Balling (previous) (diff)

comment:19 by Balling, 12 months ago

BTW, do you have original mp4 files or whatever Sony uses? I need to look at how they signal editlist and sdtp box.

in reply to:  19 comment:20 by slydiman, 12 months ago

Replying to Balling:

Okay. Got a fix in https://github.com/gpac/gpac/commit/fe02e73f7e1e6caf74a826bdd64b49eff1e5bdc5 that fixes SAP 2 in both hevc samples, only for gpac.exe file.hevc inspect:deep

Can it help somehow to fix ffmpeg? Usually I don't use gpac. Sometimes mp4box crashes on huge files in the middle of process. But I expected to get ffmpeg fixed.

Replying to Balling:

BTW, do you have original mp4 files or whatever Sony uses? I need to look at how they signal editlist and sdtp box.

Sure
http://slydiman.me/ffmpeg/sony_hevc_yuv422p10le_4k.mp4

I don't see any sdtp. Note SONY uses MXF-like metadata and fragmented old kkad metadata inside with SPS and many PPS. Note I can decode these metadata. I have not seen specs of these metadata in the public domain.

comment:21 by Balling, 12 months ago

Please also attach yuv420p10le hevc.mp4

Last edited 12 months ago by Balling (previous) (diff)

in reply to:  21 comment:22 by slydiman, 12 months ago

Replying to Balling:

Please also attach yuv420p10le hevc.mp4

It is huge 5.6GB. I can dump any part if necessary.
http://slydiman.me/ffmpeg/sony_hevc_yuv420p10le_4k.txt

Note: See TracTickets for help on using tickets.