MXF muxer using long gop with x264 essence creates corrupt output

[steipal]

Summary of the bug:

The file is not playable in Premiere, Vegas, VirtualDub or VLC.
It seems to be an issue that x264 with reordered frames is not indexed correctly in MXF.

How to reproduce:

>ffmpeg -f lavfi -i mandelbrot -t 10 -c:v libx264 -pix_fmt yuv420p output.mxf

ffmpeg version N-104973-g26dcb05b97 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11.2.0 (Rev5, Built by MSYS2 project)
  configuration:  --disable-static --enable-shared --pkg-config=pkgconf --cc='ccache gcc' --cxx='ccache g++' --ld='ccache g++' --disable-autodetect --enable-amf --enable-bzlib --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --enable-sdl2 --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --enable-libaom --disable-debug --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libopenmpt --enable-version3 --enable-librav1e --enable-libsrt --enable-libgsm --enable-libvmaf --enable-libsvtav1 --enable-librubberband --enable-mbedtls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++ --extra-cflags=-DLIBXML_STATIC --extra-libs=-liconv --disable-w32threads --shlibdir=/local64/bin-video
  libavutil      57. 12.100 / 57. 12.100
  libavcodec     59. 14.100 / 59. 14.100
  libavformat    59. 10.100 / 59. 10.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 20.100 /  8. 20.100
  libswscale      6.  1.101 /  6.  1.101
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
Input #0, lavfi, from 'mandelbrot':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: rawvideo (RGB[0] / 0x424752), rgb0, 640x480 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 00000227b5e00d00] using SAR=1/1
[libx264 @ 00000227b5e00d00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000227b5e00d00] profile High, level 3.0, 4:2:0, 8-bit
Output #0, mxf, to 'output.mxf':
  Metadata:
    encoder         : Lavf59.10.100
  Stream #0:0: Video: h264, yuv420p(tv, progressive), 640x480 [SAR 1:1 DAR 4:3], q=2-31, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc59.14.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  250 fps=123 q=-1.0 Lsize=    2968kB time=00:00:09.92 bitrate=2450.6kbits/s speed=4.88x
video:2764kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 7.369122%
[libx264 @ 00000227b5e00d00] frame I:1     Avg QP:20.70  size: 24904
[libx264 @ 00000227b5e00d00] frame P:63    Avg QP:23.47  size: 17056
[libx264 @ 00000227b5e00d00] frame B:186   Avg QP:29.95  size:  9305
[libx264 @ 00000227b5e00d00] consecutive B-frames:  0.8%  0.0%  0.0% 99.2%
[libx264 @ 00000227b5e00d00] mb I  I16..4: 37.8% 45.2% 16.9%
[libx264 @ 00000227b5e00d00] mb P  I16..4:  0.6%  0.1%  0.0%  P16..4: 44.5% 14.5% 10.8%  0.0%  0.0%    skip:29.5%
[libx264 @ 00000227b5e00d00] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8: 30.5%  5.8%  2.7%  direct: 4.5%  skip:56.5%  L0:45.4% L1:45.1% BI: 9.5%
[libx264 @ 00000227b5e00d00] 8x8 transform intra:35.8% inter:16.5%
[libx264 @ 00000227b5e00d00] coded y,uvDC,uvAC intra: 23.2% 68.6% 26.7% inter: 12.5% 24.4% 19.3%
[libx264 @ 00000227b5e00d00] i16 v,h,dc,p:  9%  7%  2% 81%
[libx264 @ 00000227b5e00d00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 10% 22% 11% 13%  6% 12%  6%  4%
[libx264 @ 00000227b5e00d00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 16% 12% 10% 11%  9% 10%  9%  8%
[libx264 @ 00000227b5e00d00] i8c dc,h,v,p: 27%  9% 12% 52%
[libx264 @ 00000227b5e00d00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 00000227b5e00d00] ref P L0: 56.2%  5.4% 26.8% 11.7%
[libx264 @ 00000227b5e00d00] ref B L0: 71.6% 21.4%  6.9%
[libx264 @ 00000227b5e00d00] ref B L1: 87.9% 12.1%
[libx264 @ 00000227b5e00d00] kb/s:2264.17

It plays well in MPV and ffprobe.

Changing muxer to MOV works as expected.

[Balling]

Looks related... #9647

[Tomas Härdin]

Plays fine in VLC 3.0.17.4 with current master (00b03331a018cf59740153a20c762b714253dd10)