mxfenc assertion failure

[Jacob Lifshay]

Summary of the bug:
mxf muxer has an assertion failure when trying to use the h264 codec.
How to reproduce (debug output too long, so I attached as a file):

% wget "https://www.digitizationguidelines.gov/guidelines/MXF_sampleFiles/RDD48-sample11-gf-unc-ntsc-4.2.zip"
% sha256sum -b RDD48-sample11-gf-unc-ntsc-4.2.zip
ad3afa838c32d6ec231f7d0cc2b578de4e4d6f5d1053bc75e291e41a0138457a *RDD48-sample11-gf-unc-ntsc-4.2.zip
% unzip RDD48-sample11-gf-unc-ntsc-4.2.zip
% gdb -q --batch -ex run -ex bt --args ./ffmpeg-git -y -v 9 -loglevel 99 -i RDD48-sample11-gf-unc-ntsc-4.2.mxf -c:v h264 /tmp/out.mxf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
ffmpeg version N-120228-g3334bdc6e7 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14+deb12u1)
  configuration: --disable-stripping --assert-level=2 --enable-ftrapv --disable-shared --enable-static --optflags=-O0 --progs-suffix=-git --enable-sdl2 --extra-cflags=-g --enable-libx264 --enable-gpl --enable-libklvanc
  libavutil      60.  4.101 / 60.  4.101
  libavcodec     62.  6.100 / 62.  6.100
  libavformat    62.  1.102 / 62.  1.102
  libavdevice    62.  0.100 / 62.  0.100
  libavfilter    11.  1.100 / 11.  1.100
  libswscale      9.  0.100 /  9.  0.100
  libswresample   6.  0.100 /  6.  0.100
Splitting the commandline.
<snip>
Thread 72 "mux0:mxf" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffaeac16c0 (LWP 878133)]
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff76a9f4f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007ffff765afb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff7645472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x0000555555c92054 in avio_w8 (s=0x5555590d5300, b=-129) at libavformat/aviobuf.c:181
#5  0x0000555555decc0a in mxf_write_index_table_segment (s=0x5555590ccf40) at libavformat/mxfenc.c:2106
#6  0x0000555555df1def in mxf_write_packet (s=0x5555590ccf40, pkt=0x7fffa8000900) at libavformat/mxfenc.c:3373
#7  0x0000555555dd48b3 in write_packet (s=0x5555590ccf40, pkt=0x7fffa8000900) at libavformat/mux.c:746
#8  0x0000555555dd59b2 in interleaved_write_packet (s=0x5555590ccf40, pkt=0x7fffa8000900, flush=0, has_packet=0) at libavformat/mux.c:1087
#9  0x0000555555dd5b45 in write_packet_common (s=0x5555590ccf40, st=0x5555590cf240, pkt=0x7fffa8000900, interleaved=1) at libavformat/mux.c:1112
#10 0x0000555555dd5e42 in write_packets_common (s=0x5555590ccf40, pkt=0x7fffa8000900, interleaved=1) at libavformat/mux.c:1172
#11 0x0000555555dd6017 in av_interleaved_write_frame (s=0x5555590ccf40, pkt=0x7fffa8000900) at libavformat/mux.c:1228
#12 0x000055555569bcec in write_packet (mux=0x5555590ccb80, ost=0x5555590cd700, pkt=0x7fffa8000900) at fftools/ffmpeg_mux.c:236
#13 0x000055555569beb0 in sync_queue_process (mux=0x5555590ccb80, ms=0x5555590cd700, pkt=0x7fffa8000900, stream_eof=0x7fffaeac0dcc) at fftools/ffmpeg_mux.c:280
#14 0x000055555569c236 in mux_packet_filter (mux=0x5555590ccb80, mt=0x7fffaeac0dd0, ost=0x5555590cd700, pkt=0x7fffa8000900, stream_eof=0x7fffaeac0dcc) at fftools/ffmpeg_mux.c:356
#15 0x000055555569c550 in muxer_thread (arg=0x5555590ccb80) at fftools/ffmpeg_mux.c:438
#16 0x00005555556b7253 in task_wrapper (arg=0x5555590bd5e0) at fftools/ffmpeg_sched.c:2534
#17 0x00007ffff76a81f5 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#18 0x00007ffff772889c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[Jacob Lifshay]

ffmpeg in gdb output

[Zhao Zhili <zhilizhao@tencent.com>]

In 631bb6c/ffmpeg:

avformat/mxfenc: Ensure frame offset in valid range

Fix assert failure.
Fix #11666.

Signed-off-by: Zhao Zhili <​zhilizhao@tencent.com>