Matroska container doesn't write h264 extdata with a pcm_s16le input

[Reboot]

Summary of the bug: ffmpeg does not write the pixel format (extdata) of a h264 encoded video stream to an mkv file, depending on the length and codec of the audio source (in this particular instance pcm_s16le). This is when combining two inputs (still image video, and audio).

This bug probably also applies when encoding other pcm-included streams to different codecs.

How to reproduce:

% ffmpeg -f lavfi -i testsrc2,trim=end_frame=1 -f lavfi -i sine=d=120 -c:v libx264 -c:a pcm_s16le output.mkv
ffmpeg version is N-107013-g93b31dae1d-20220527
built on 2022-05-27 12:30 - built with gcc 11.2.0 (crosstool-NG 1.24.0.533_681aaef)

(this is using the latest gpl build from BtbN at time of writing)

After encoding, reading the output file with either ffmpeg or another application results in not finding the pixel format codec parameter for the video stream. This can be problematic for web apps.

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.
Thanks to #ffmpeg IRC members JEEB and durandai_1707 for helping me troubleshoot and reproduce this issue.

[Balling]

Same for ts container, mp4 not supported:

  Stream #0:0[0x100]: Video: h264 ([27][0][0][0] / 0x001B), none, 90k tbr, 90k tbn

May be a bug in still images avc?

There is also [matroska @ 00000203a8853fc0] Starting new cluster due to timestamp

[mkver]

The Matroska muxer correctly writes the extradata. But somehow the video packet is only ever sent to the muxer after all the audio packets, so that it is written after many audio packets (near audio packets with timestamp 1:50); it is therefore not found by avformat_find_stream_info, leading to the warning about missing pixel format. Said pixel format is only inferred by libavformat when a packet is decoded, although it could be inferred from the extradata alone (if one is willing to presume that the extradata will not be overwritten by new in-band extradata lateron).

So the issue is somewhere else, either in the ffmpeg cli program or in libavfilter I presume.

[Elon Musk]

Very strange to blame libavfilter when timestamp of video packet is 0, and it is received immediately. Nobody seems interested to do the bisect.