ff_color_frame null pointer exception

[Andrzej Szombierski]

I'm investigating a crash in a complex application, so my apologies for not providing a small repro case.

I'm decoding a H264 NAL stream with ffmpeg without any HW acceleration. The pixel format is set to NV12. When attempting to decode the first frame, I get a null pointer exception in ff_color_frame, specifically in memset. ff_copy_frame is attempting to access frame->data[2], which is null for NV12.

It seems that the function assumes that each color component corresponds to one plane, which is clearly not the case for NV12.

[Balling]

The pixel format is set to NV12

Why use that if you are not HW decoding?

We ony support HEAD of master here, what version are you on? Try head of master.

[Andrzej Szombierski]

Replying to Balling:

The pixel format is set to NV12

Why use that if you are not HW decoding?

Ok, perhaps my summary was too short. I am using hardware decoding in most scenarios and everything works fine. I've added an option to use the SW decoder for troubleshooting and as a fallback option - and this is where the bug has appeared. NV12 isn't the best pixel format for SW decoding, but it shouldn't outright crash, right?

We ony support HEAD of master here, what version are you on? Try head of master.

I will try to make a small example reproducing the issue on master. The sources for ff_color_frame haven't changed much, and on HEAD it still contains a loop indexing the frame->data[] array by color component (which is wrong).

[Andrzej Szombierski]

Ok, never mind. With latest master, the NV12 format is rejected outright before it would cause any trouble. Older versions didn't object.

The code in ff_color_frame can still cause trouble with interleaved pixel formats, but right now it's only used in h264 which doesn't support them, so it's safe.