Non Intra H.264 files are not muxed correctly in mxf

[Francesco Bucciantini]

Summary of the bug:

when muxing H.264 Files with Long GOP or when muxing any kind of arbitrary H.264 file which makes use of motion compensation, the final muxed mxf file seems not to be alright as decoders report some errors at decoding time.

here is an example:

x264.exe "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\AVS Script.avs" --preset medium --profile high422 --level 5.2 --ref 4 min-keyint 1 --keyint 40 --bitrate 500000 --vbv-maxrate 500000 --vbv-bufsize 500000 --deblock -4:-4 --overscan show --colormatrix bt709 --range tv --log-level info --thread-input --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd cbr --aud --output-csp i422 --output-depth 10 --output "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\raw_video.h264"

ffmpeg.exe -i "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\AVS Script.avs" -vn -sn -c:a pcm_s24le -ar 48000 -af loudnorm=I=-24:LRA=1:tp=-2 -f wav  "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\audio.wav"

ffmpeg.exe -i "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\raw_video.h264" -i "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\audio.wav" -c:v copy -c:a copy -f mxf -y "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\output.mxf"

pause

The code above uses Avisynth to index a sample file and encode it as 500 Mbit/s with 1 keyframe every 40. Then, such a file, called raw_file.h264 is muxed in mxf using FFMpeg, however the resulting file - although it completes the muxing - has something wrong 'cause once it's played some players don't decode all frames: some are decoded, some other are not and the players report "requested frame not found".

It's possible to verify this by trying to play the encoded and muxed video in VirtualDub FilterMod and it will report the very same error.
Muxing the very same file into another container like .ts via TSMuxer works and shows no playback/decoding issues whatsoever. Same goes for .mp4 using MP4Box, so it's just mxf that seems not to be ok.

Remuxing the files with BBC BMX Transwrap like:

ffmpeg.exe -i "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\raw_video.h264" -i "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\audio.wav" -c:v copy -c:a copy -f mxf -y "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\pre-final_output.mxf"

bmxtranswrap.exe -p -y 10:00:00:00 -t op1a -o "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\final_output.mxf" "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\pre-final_output.mxf"

improves things and produces a better file with less decoding errors and - if the GOP is shorter and I have exactly 1 intra every second (so keyint = framerate), the BBC BMX Transwrap remuxed file works without any decoding issue.

I made 4 samples (colorbars, nothing exciting).

Test 1:

Test1_H264_30p_keyint30_BMX.mxf
Test1_H264_30p_keyint30_FFMpeg.mxf

Test 2:

Test2_H264_30p_keyint40_BMX.mxf
Test2_H264_30p_keyint40_FFMpeg.mxf

About Test1, the framerate is actually 29,970 so 30000/1001, but anyway, it has been encoded with --min-keyint 1 --keyint 30, so after the raw_video.h264 has been muxed with BMX it works just fine, whereas the FFMpeg muxed version has decoding issues.

Of the 100 frames encoded, this is the BMX muxed file being correctly decoded as:

Frame 0, Intra
Frame 1, B
Frame 2, B
Frame 3, B
Frame 4, P
Frame 5, B
Frame 6, B
Frame 7, B
Frame 8, P
Frame 9, B
Frame 10, B
Frame 11, B
Frame 12, P
Frame 13, B
Frame 14, B
Frame 15, B
Frame 16, P
Frame 17, B
Frame 18, B
Frame 19, B
Frame 20, P
Frame 21, B
Frame 22, B
Frame 23, B
Frame 24, P
Frame 25, B
Frame 26, B
Frame 27, B
Frame 28, P
Frame 29, P
Frame 30, I
Frame 31, B
Frame 32, B
Frame 33, B
Frame 34, P
Frame 35, B
Frame 36, B
Frame 37, B
Frame 38, P
Frame 39, B
Frame 40, B
Frame 41, B
Frame 42, P
Frame 43, B
Frame 44, B
Frame 45, B
Frame 46, P
Frame 47, B
Frame 48, B
Frame 49, B
Frame 50, P
Frame 51, B
Frame 52, B
Frame 53, B
Frame 54, P
Frame 55, B
Frame 56, B
Frame 57, B
Frame 58, P
Frame 59, P
Frame 60, I
Frame 61, B
Frame 62, B
Frame 63, B
Frame 64, P
Frame 65, B
Frame 66, B
Frame 67, B
Frame 68, P
Frame 69, B
Frame 70, B
Frame 71, B
Frame 72, P
Frame 73, B
Frame 74, B
Frame 75, B
Frame 76, P
Frame 77, B
Frame 78, B
Frame 79, B
Frame 80, P
Frame 81, B
Frame 82, B
Frame 83, B
Frame 84, P
Frame 85, B
Frame 86, B
Frame 87, B
Frame 88, P
Frame 89, P
Frame 90, I
Frame 91, B
Frame 92, B
Frame 93, B
Frame 94, P
Frame 95, B
Frame 96, B
Frame 97, B
Frame 98, P
Frame 99, B
Frame 100, P

while this is the FFMpeg muxed file (so same raw_video.h264 but muxed with FFMpeg) and it has a clear recurring pattern error while being decoded:

Frame 0, Intra
Frame 1, requested frame not found
Frame 2, requested frame not found
Frame 3, B
Frame 4, B
Frame 5, requested frame not found
Frame 6, requested frame not found
Frame 7, B
Frame 8, B
Frame 9, requested frame not found
Frame 10, requested frame not found
Frame 11, B
Frame 12, B
Frame 13, requested frame not found
Frame 14, requested frame not found
Frame 15, B
Frame 16, B
Frame 17, requested frame not found
Frame 18, requested frame not found
Frame 19, B
Frame 20, B
Frame 21, requested frame not found
Frame 22, requested frame not found
Frame 23, B
Frame 24, B
Frame 25, requested frame not found
Frame 26, requested frame not found
Frame 27, B
Frame 28, B
Frame 29, P
Frame 30, I
Frame 31, requested frame not found
Frame 32, requested frame not found
Frame 33, B
Frame 34, B
Frame 35, requested frame not found
Frame 36, requested frame not found
Frame 37, B
Frame 38, B
Frame 39, requested frame not found
Frame 40, requested frame not found
Frame 41, B
Frame 42, B
Frame 43, requested frame not found
Frame 44, requested frame not found
Frame 45, B
Frame 46, B
Frame 47, requested frame not found
Frame 48, requested frame not found
Frame 49, B
Frame 50, B
Frame 51, requested frame not found
Frame 52, requested frame not found
Frame 53, B
Frame 54, B
Frame 55, requested frame not found
Frame 56, requested frame not found
Frame 57, B
Frame 58, B
Frame 59, P
Frame 60, I
Frame 61, requested frame not found
Frame 62, requested frame not found
Frame 63, B
Frame 64, B
Frame 65, requested frame not found
Frame 66, requested frame not found
Frame 67, B
Frame 68, B
Frame 69, requested frame not found
Frame 70, requested frame not found
Frame 71, B
Frame 72, B
Frame 73, requested frame not found
Frame 74, requested frame not found
Frame 75, B
Frame 76, B
Frame 77, requested frame not found
Frame 78, requested frame not found
Frame 79, B
Frame 80, B
Frame 81, requested frame not found
Frame 82, requested frame not found
Frame 83, B
Frame 84, B
Frame 85, requested frame not found
Frame 86, requested frame not found
Frame 87, B
Frame 88, B
Frame 89, P
Frame 90, I
Frame 91, requested frame not found
Frame 92, requested frame not found
Frame 93, B
Frame 94, B
Frame 95, requested frame not found
Frame 96, requested frame not found
Frame 97, B
Frame 98, B
Frame 99, requested frame not found
Frame 100, requested frame not found

So essentially the pattern of the GOP is: IBBBPBBBPBBBPBBBPBBBPBBBPBBBPP and then it goes back to I and starts over and apparently the decoding issue is consistent in time.

Now, as to test 2, the file is again a 29,970p, so 30000/10001 but this time it has been encoded with --min-keyint 1 --keyint 40, so the GOP is longer than 1 second (keyint != framerate) and in this case both BMX and FFMpeg muxed files have decoding issues.

Let's start with BMX:

Frame 0, Intra
Frame 1, B
Frame 2, B
Frame 3, B
Frame 4, P
Frame 5, B
Frame 6, B
Frame 7, B
Frame 8, P
Frame 9, B
Frame 10, B
Frame 11, B
Frame 12, P
Frame 13, B
Frame 14, B
Frame 15, B
Frame 16, P
Frame 17, B
Frame 18, B
Frame 19, B
Frame 20, P
Frame 21, B
Frame 22, B
Frame 23, B
Frame 24, P
Frame 25, B
Frame 26, B
Frame 27, B
Frame 28, P
Frame 29, B
Frame 30, B
Frame 31, B
Frame 32, P
Frame 33, requested frame not found
Frame 34, B
Frame 35, B
Frame 36, B
Frame 37, B
Frame 38, B
Frame 39, P
Frame 40, I
Frame 41, B
Frame 42, B
Frame 43, B
Frame 44, P
Frame 45, B
Frame 46, B
Frame 47, B
Frame 48, P
Frame 49, B
Frame 50, B
Frame 51, B
Frame 52, P
Frame 53, B
Frame 54, B
Frame 55, B
Frame 56, P
Frame 57, B
Frame 58, B
Frame 59, B
Frame 60, P
Frame 61, B
Frame 62, B
Frame 63, B
Frame 64, P
Frame 65, B
Frame 66, B
Frame 67, B
Frame 68, P
Frame 69, B
Frame 70, B
Frame 71, B
Frame 72, P
Frame 73, requested frame not found
Frame 74, B
Frame 75, B
Frame 76, B
Frame 77, B
Frame 78, B
Frame 79, P
Frame 80, I
Frame 81, B
Frame 82, B
Frame 83, B
Frame 84, P
Frame 85, B
Frame 86, B
Frame 87, B
Frame 88, P
Frame 89, B
Frame 90, B
Frame 91, B
Frame 92, P
Frame 93, B
Frame 94, B
Frame 95, B
Frame 96, P
Frame 97, B
Frame 98, B
Frame 99, B
Frame 100, P

Now the FFMpeg muxed one:

Frame 0, Intra
Frame 1, requested frame not found
Frame 2, requested frame not found
Frame 3, B
Frame 4, P
Frame 5, requested frame not found
Frame 6, requested frame not found
Frame 7, B
Frame 8, P
Frame 9, requested frame not found
Frame 10, requested frame not found
Frame 11, B
Frame 12, P
Frame 13, requested frame not found
Frame 14, requested frame not found
Frame 15, B
Frame 16, P
Frame 17, requested frame not found
Frame 18, requested frame not found
Frame 19, B
Frame 20, P
Frame 21, requested frame not found
Frame 22, requested frame not found
Frame 23, B
Frame 24, P
Frame 25, requested frame not found
Frame 26, requested frame not found
Frame 27, B
Frame 28, P
Frame 29, requested frame not found
Frame 30, requested frame not found
Frame 31, B
Frame 32, P
Frame 33, requested frame not found
Frame 34, requested frame not found
Frame 35, B
Frame 36, B
Frame 37, requested frame not found
Frame 38, B
Frame 39, P
Frame 40, I
Frame 41, requested frame not found
Frame 42, requested frame not found
Frame 43, B
Frame 44, P
Frame 45, requested frame not found
Frame 46, requested frame not found
Frame 47, B
Frame 48, P
Frame 49, requested frame not found
Frame 50, requested frame not found
Frame 51, B
Frame 52, P
Frame 53, requested frame not found
Frame 54, requested frame not found
Frame 55, B
Frame 56, P
Frame 57, requested frame not found
Frame 58, requested frame not found
Frame 59, B
Frame 60, P
Frame 61, requested frame not found
Frame 62, requested frame not found
Frame 63, B
Frame 64, P
Frame 65, requested frame not found
Frame 66, requested frame not found
Frame 67, B
Frame 68, P
Frame 69, requested frame not found
Frame 70, requested frame not found
Frame 71, B
Frame 72, P
Frame 73, requested frame not found
Frame 74, requested frame not found
Frame 75, B
Frame 76, B
Frame 77, requested frame not found
Frame 78, B
Frame 79, P
Frame 80, I
Frame 81, requested frame not found
Frame 82, requested frame not found
Frame 83, B
Frame 84, P
Frame 85, requested frame not found
Frame 86, requested frame not found
Frame 87, B
Frame 88, P
Frame 89, requested frame not found
Frame 90, requested frame not found
Frame 91, B
Frame 92, P
Frame 93, requested frame not found
Frame 94, requested frame not found
Frame 95, B
Frame 96, P
Frame 97, requested frame not found
Frame 98, requested frame not found
Frame 99, B
Frame 100, P

Command line for Test 1:

x264.exe "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\AVS Script.avs" --preset medium --profile high422 --level 5.2 --ref 4 min-keyint 1 --keyint 30 --bitrate 500000 --vbv-maxrate 500000 --vbv-bufsize 500000 --deblock -4:-4 --overscan show --colormatrix bt709 --range tv --log-level info --thread-input --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd cbr --aud --output-csp i422 --output-depth 10 --output "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\raw_video.h264"

Command line for Test 2:

x264.exe "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\AVS Script.avs" --preset medium --profile high422 --level 5.2 --ref 4 min-keyint 1 --keyint 40 --bitrate 500000 --vbv-maxrate 500000 --vbv-bufsize 500000 --deblock -4:-4 --overscan show --colormatrix bt709 --range tv --log-level info --thread-input --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd cbr --aud --output-csp i422 --output-depth 10 --output "\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\raw_video.h264"

Test download: ​https://we.tl/t-cNrs0AByEk
(link valid for 7 days)

In attachment you're gonna find the container check report, highlighting the errors for each file.

[Francesco Bucciantini]

On top of this, ffmpeg doesn't indicate frame reordering in the index, while BMX does:

[Balling]

So mxf always uses ANNEX B style of h.264 and h.265 bitsreams. The convertion to it is not perfect in ffmpeg. For example, ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210910040948.14186-1-haihao.xiang@intel.com/ and

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/abe4571410fa4ef3947119d5064dea3e@mx.toyon.corp/

[Francesco Bucciantini]

Fixed by Philip de Nier in the BBC muxer (BBC BMX Transwrap): ​https://sourceforge.net/p/bmxlib/bmx/ci/8285326eee1762b982b7adadd554318d6fa05c8b/

Now only FFMpeg is left with the issue... :(

[ariley]

In mxf_parse_mpeg2_frame(), there's a bit of code that reads:

          } else if (c == 0x100) { // pic                                                                            
              int pict_type = (pkt->data[i+2]>>3) & 0x07;                                                            
              e->temporal_ref = (pkt->data[i+1]<<2) | (pkt->data[i+2]>>6);                                           
              if (pict_type == 2) { // P-frame                                                                       
                  e->flags |= 0x22;                                                                                  
                  sc->closed_gop = 0; // reset closed GOP, don't matter anymore                                      
              } else if (pict_type == 3) { // B-frame                                                                
                  if (sc->closed_gop)                                                                                
                      e->flags |= 0x13; // only backward prediction                                                  
                  else                                                                                               
                      e->flags |= 0x33;                                                                              
                  sc->temporal_reordering = -1;                                                                      

The corresponding code setting e->temporal_ref and sc->temporal_reordering in mxf_parse_h264_frame() is missing:

          case H264_NAL_SLICE:
              init_get_bits8(&gb, buf, buf_end - buf);
              get_ue_golomb_long(&gb); // skip first_mb_in_slice
              slice_type = get_ue_golomb_31(&gb);
              switch (slice_type % 5) {
              case 0:
                  e->flags |= 0x20; // P Picture
                  e->flags |= 0x06; // P Picture
                  break;
              case 1:
                  e->flags |= 0x30; // B Picture
                  e->flags |= 0x03; // non-referenced B Picture
                  break;
              }

Unfortunately, while the temporal order is carried in the mpeg2 picture header, it's not quite so trivial with H264.

[syehoonkim]

Replying to Francesco Bucciantini:

Fixed by Philip de Nier in the BBC muxer (BBC BMX Transwrap): ​https://sourceforge.net/p/bmxlib/bmx/ci/8285326eee1762b982b7adadd554318d6fa05c8b/

Now only FFMpeg is left with the issue... :(

On the recent PR of bmx(​https://github.com/bbc/bmx/pull/94), H.264 Long GOP Separate Fields videos are supported.
I did a short test and it seems works well.

[Francesco Bucciantini]

Yep, Philip nailed it. This was the last piece of the puzzle to have a proper AVC Long GOP support in BMX. It's been a while since September 2022, but the puzzle is now complete and I'm sure a lot of people are gonna be happy about this. :)
I will leave this ticket open in the FFMpeg bug tracker, however, 'cause as of today the FFMpeg mxf muxer still hasn't solved the issue. I just tested on the latest master and the issue is still there.