Null pointer when nvenc gets aborted

[Daniel Stankewitz]

Summary of the bug:

libavcodec crashes if nvenc gets aborted with i.e. "No capable devices found" or any other error.

How to reproduce:

With any NVIDIA GPU that is not AV1 capable (because that'll create the error "No capable devices found") with this example.

% ffmpeg.exe -v trace -f lavfi -i smptebars=size=1920x1080 -t 5 -c:v av1_nvenc -b:v 500k -c:a none -pix_fmt yuv420p test.mp4

or

const AVCodec* codec = avcodec_find_encoder_by_name("av1_nvenc");
AVCodecContext* codecContext = avcodec_alloc_context3(codec);
codecContext->width = 320;
codecContext->height = 240;
codecContext->time_base = { 1, 25 };
codecContext->pix_fmt = AV_PIX_FMT_YUV420P;
int res = avcodec_open2(codecContext, codec, NULL);
avcodec_free_context(&codecContext);

ffmpeg version: a02e45a1f3cc6f07c7437c1e225dea2683f843cf

In ff_nvenc_encode_init the initialization order is:

if ((ret = nvenc_load_libraries(avctx)) < 0)
    return ret;

if ((ret = nvenc_setup_device(avctx)) < 0)
    return ret;

if ((ret = nvenc_setup_encoder(avctx)) < 0)
    return ret;

if ((ret = nvenc_setup_surfaces(avctx)) < 0)
    return ret;

... but in ff_nvenc_encode_close() it tries to work with fifo buffers that are still null because they'll get initialized in nvenc_setup_surfaces() later (after nvenc_setup_device()).

[quinkblack]

Should be fixed by ​http://ffmpeg.org/pipermail/ffmpeg-devel/2023-February/307035.html

[Balling]

Fixed in 21101d9eb1f26c1f1a58454919a9ef5993ae9900.