swscale rejects files with reserved colorspace tags after filter reinit

[Francesco Bucciantini]

Summary of the bug:

When a stream has different frame properties (i.e matrix/transfer/primaries changing between frames) then FFMpeg crashes.

In particular, I've uploaded a sample called XDCAM_50_With_Changing_Frame_Properties.mxf at ​https://streams.videolan.org/upload/ (backup link ​https://we.tl/t-qfePLdYbcM). It's an FULL HD 1920x1080 MPEG-2 High 4:2:2 Profile, Level High 50 Mbit/s yv16 25i TFF BT709 SDR file with the following characteristics:

frame 1 to 13 (matrix, transfer, primaries as BT709)
frame 14 to 910 (matrix, transfer, primaries are missing)
frame 911 (matrix BT709, transfer, primaries are missing)
frame 912 to 922 (matrix, transfer, primaries as BT709)

If we index it in Avisynth with LWLibavVideoSource() we can in fact see the frame properties changing as we move through the stream:

When opening the AVS Script.avs in FFMpeg, the frame properties from the first frame are set and everything works normally, however, if we try to open it with FFMpeg with a simple:

ffmpeg.exe -benchmark -i "XDCAM_50_With_Changing_Frame_Properties.mxf" -map 0:0? -f null out.null

it crashes with:

[vf#0:0 @ 0000020c8cb36d40] Reconfiguring filter graph because video parameters changed to yuv422p(tv, unknown), 1920x1080
[swscaler @ 0000020c8e93d140] Unsupported input (Error number -129 occurred): fmt:yuv422p csp:unknown prim:reserved trc:reserved -> fmt:yuv422p csp:bt709 prim:reserved trc:reserved
[vf#0:0 @ 0000020c8cb36d40] Error while filtering: Error number -129 occurred
[vf#0:0 @ 0000020c8cb36d40] Task finished with error code: -129 (Error number -129 occurred)
[vf#0:0 @ 0000020c8cb36d40] Terminating thread with return code -129 (Error number -129 occurred)

How to reproduce:

% ffmpeg -benchmark -i "XDCAM_50_With_Changing_Frame_Properties.mxf" -map 0:0? -f null out.null
ffmpeg version git-master
built on January 28th 2025

[Francesco Bucciantini]

I also wanna add that decoding completes fine in FFMpeg 6.1.1 Stable instead, so the issue has been introduced in the 7.x branch leading to the current master.

[Francesco Bucciantini]

To put it down even further, the problem with 7.x and master is when the matrix, primaries and transfer are changing, in fact if the file is trimmed to isolate the parts where those stay the same, then FFMpeg has no problem with them.

Obviously in the section in which they're all populated it's gonna say:
Stream #0:0: Video: wrapped_avframe, yuv422p(tv, bt709, top coded first (swapped)), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn

while in the one in which they're not populated it's gonna say:
Stream #0:0: Video: mpeg2video (4:2:2), yuv422p(tv, unknown/reserved/reserved, top first), 1920x1080 [SAR 1:1 DAR 16:9], 50000 kb/s, 25 fps, 25 tbr, 25 tbn

but it won't crash, however if we don't trim and we let it analyze a file with a transition, then we get the aforementioned crash:

vf#0:0 @ 0000020c8cb36d40] Reconfiguring filter graph because video parameters changed to yuv422p(tv, unknown), 1920x1080
[swscaler @ 0000020c8e93d140] Unsupported input (Error number -129 occurred): fmt:yuv422p csp:unknown prim:reserved trc:reserved -> fmt:yuv422p csp:bt709 prim:reserved trc:reserved
[vf#0:0 @ 0000020c8cb36d40] Error while filtering: Error number -129 occurred
[vf#0:0 @ 0000020c8cb36d40] Task finished with error code: -129 (Error number -129 occurred)
[vf#0:0 @ 0000020c8cb36d40] Terminating thread with return code -129 (Error number -129 occurred)

[Timo R.]

This is not a crash, it just exits cleanly saying what you're doing is not supported.
swscale does not support converting between those parameters, and since the output parameters cannot dynamically change, it would have to convert, but simply can't. So it exits.

[emcodem]

Conversion works when disabling filter reinit:

ffmpeg -reinit_filter:v 0 -i Interview.mxf -f null -

The mpeg2 stream changes from transfer 2 (709) to 0 (unknown/forbidden).
Might not be trivial to catch this special case of changing to unknown transfer characteristics in which case it would be best to just keep on working with the current transfer char or assume 709 as default.
Also, question could be why sws complains about "Unsupported input ... csp:unknown" only in the case when it's triggered from reinit_filter but it works with the affected portion of the video when run standalone.

My personal opinion is that reinit_filter should be opt-in instead of default because it such a breaking change. Not all mid-stream changes must be handled, especially aspect ratio and the 3 color descriptions

[Francesco Bucciantini]

Yes, it is indeed a breaking change compared to the 6.x branch, so I also agree with emcodem on the fact that it should be opt-in rather than opt-out.

Using the suggested method by emcodem works, namely:

ffmpeg -reinit_filter:v 0 -i Interview.mxf -f null -

however as soon as we insert -pix_fmt to convert from yv16 to yv12 (i.e from 4:2:2 8bit planar to 4:2:0 8bit planar):

ffmpeg -reinit_filter:v 0 -i Interview.mxf -pix_fmt yuv420p -f null -

it fails with the following error: Changing video frame properties on the fly is not supported by all filters.

One could work around the issue by setting the info in the filter graph at the very beginning, like, in this case, that would be:

setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709

however that's more of a workaround than a solution. The best thing would be to get the same behavior as the FFMpeg 6.x branch to avoid making this a breaking change which would hinder compatibility with a lot of scripts and BAT files unless rewritten ad hoc.

[MasterQuestionable]

͏    However alike input stream self probably wouldn't make much sense.
͏    Refer also:
͏    ​https://github.com/libjxl/libjxl/issues/4056#issuecomment-2607405171
͏    ​https://trac.ffmpeg.org/ticket/11142#comment:3

[ "-reinit_filter[:${Stream}]"
͏    bool, input, per-stream
͏    Determines if the filtergraph the stream feeds to shall reinitialize, when the input properties change mid-stream.
͏    Default: true

͏    Default as most video and all audio filters, cannot handle deviations of input properties.
͏    Upon reinitialization, existing filter state is lost.
͏    (e.g. the frame count "n" reference available in some filters)
͏    Any frame buffered at time of reinitialization is lost.

͏    The property changes would trigger reinitialization are:
͏    |*| Video: frame resolution, pixel format.
͏    |*| Audio: sample format, sample rate, channel count, channel layout. ]
.
͏    This is overall sensible, and would contribute to performance benefits.
͏    With low likelihood of breakage.

[Niklas Haas]

I think there are two issue here:

One is the filter reinit somehow triggers swscale, while running on the isolated portion alone does not.

The other is that swscale now rejects trc/prim=reserved. I'm not sure why it only triggers on filter reinit, but it's possibly to do with some other condition for when swscale

In either case, to be clear, this is an illegal/broken file, as it contains coded values which are marked as reserved by the spec. That swscale now correctly rejects invalid colorspace tags instead of implicitly accepting them in the past is indeed a regression in functionality, but a fix in terms of correctness.

As a possible resolution to the second error, what if we tie this behavior to e.g. "-err_detect compliant" (as an opt out), or "-err_detect ignore_err" (as an opt in)?

[MasterQuestionable]

͏    Best-effort playback is generally more preferable, than outright failing.
͏    Best effort should be made to present whatsoever presentable.
͏    ​https://trac.ffmpeg.org/ticket/11020#comment:3
͏    .
͏    For the color primaries shenanigans, random handling seems sensible enough nevertheless.
͏    Less intrusive than failing.

͏    Also your updated summary probably doesn't fit this issue:
͏    Would "swscale" properly handle varying color primaries after all?
͏    Does the title suggest that "reserved colorspace tags" might be supported before filter reinit?