Crash when transcoding from H264 to HEVC with variable length SEI

[harlancc]

Summary of the bug:
How to reproduce:

./ffmpeg_g -re -i test_sei.flv  -vcodec libx265 -b:v 1700k -acodec libfdk_aac -bf 3 -force_key_frames source -f flv -loglevel level+info  -vf scale='720:-2' -f hevc test.h265

ffmpeg version: release 5.1 

lastest commit: 5746987bad4dd3880cd3a321ef3d970663cd8085

x265 branch: stable
lastest commit: ce882936d5f62ea94c13972fa42cab6192864a7d


    I add some test codes for libx265.c, and when the SEI length becomes longer, then crash will happen when transcoding is finished or I input Ctrl+C to force finishing it.

    When the SEI length is constant， or becomes shorter, the crash cannot happen.

Call Stack:

*** Error in `./ffmpeg_g': corrupted double-linked list: 0x00000000054f6eb0 ***

(gdb) bt
#0  0x00007ff4ae882387 in raise () from /usr/lib64/libc.so.6
#1  0x00007ff4ae883a78 in abort () from /usr/lib64/libc.so.6
#2  0x00007ff4ae8c4f67 in __libc_message () from /usr/lib64/libc.so.6
#3  0x00007ff4ae8cb474 in malloc_printerr () from /usr/lib64/libc.so.6
#4  0x00007ff4ae8cd5f2 in _int_free () from /usr/lib64/libc.so.6
#5  0x0000000001bbe078 in av_free (ptr=0x54f6f40) at src/libavutil/mem.c:251
#6  0x0000000001bbe0b7 in av_freep (arg=0x58bb670) at src/libavutil/mem.c:261
#7  0x0000000001bb21e7 in av_frame_free (frame=0x58bb670) at src/libavutil/frame.c:117
#8  0x0000000000d8afda in h264_free_pic (h=0x580ac00, pic=0x58bb670) at src/libavcodec/h264dec.c:335
#9  0x0000000000d8b057 in h264_decode_end (avctx=0x54d8e00) at src/libavcodec/h264dec.c:348
#10 0x0000000001036fd4 in ff_frame_thread_free (avctx=0x53c2200, thread_count=13) at src/libavcodec/pthread_frame.c:747
#11 0x000000000103512c in ff_thread_free (avctx=0x53c2200) at src/libavcodec/pthread.c:89
#12 0x0000000000bac2e8 in avcodec_close (avctx=0x53c2200) at src/libavcodec/avcodec.c:455
#13 0x000000000043ce8e in transcode () at src/fftools/ffmpeg.c:4433
#14 0x000000000043d395 in main (argc=31, argv=0x7ffeab5b2068) at src/fftools/ffmpeg.c:4560

[Carl Eugen Hoyos]

Please provide the input file.

[mkver]

Looks like you ran into the second issue ("A few lines below the same error happens again with the payload arrays.") described in ​this comment to ticket #9666. You should report this to x265 instead.