hwaccel cuvid breaks B-Frames with h264_nvenc

[Andreas Håkon]

Summary of the bug:

When you transcode using h264_cuvid for deconding and h264_nvenc for encoding, if you enable "hwaccel cuvid" for a full GPU processing, then you can't enable B-frames.

This works (decode h264_cuvid / encode h264_nvenc + 1 B-Frames):

% ffmpeg -c:v h264_cuvid -i <inputfile>  \
-c:v h264_nvenc -bf:v 1 -c:a copy <outputfile>

This doesn't work (decode h264_cuvid / encode h264_nvenc + 1 B-Frames / hwaccel cuvid enabled):

% ffmpeg -hwaccel cuvid -c:v h264_cuvid -i <inputfile>  \
-c:v h264_nvenc -bf:v 1 -c:a copy <outputfile>

However, this works (decode h264_cuvid / encode h264_nvenc + No B-Frames / hwaccel cuvid enabled):

% ffmpeg -hwaccel cuvid -c:v h264_cuvid -i <inputfile>  \
-c:v h264_nvenc -bf:v 0 -c:a copy <outputfile>

The error when it fails is

[h264_nvenc @ 0x2f43120] EncodePicture failed!: no encode device (1)

So, when you use "-hwaccel cuvid" is imposible to use B-Frames.

[Timo R.]

Yes, this is known.
We are pretty sure that the bug in question is on Nvidias end though.
If you feed cuda frames to nvenc by using hwupload_cuda, bframes work as expected, so this can't be a bug in the ffmpeg CUDA frame input code in nvenc, leaving only a weird driver bug.

[Andreas Håkon]

Hi,

Sorry to hear that it's a known bug!

I need to do a full GPU processing, and this breaks completly the chain: as without B-Frames no good quality.
Any workaround?

[jgirot]

Log from first bad commit

[jgirot]

I was about to file a bug because I have encountered this same problem (almost I'm using mpeg2_cuvid). This is a regression in ffmpeg. I was able to do full hw-transcoding with ffmpeg v3.2.4, but not in latest git.

I performed a git-bisect and the first bad commit is: af1761f7b5b1b72197dc40934953b775c2d951cc

This is the commandline I used for the git-bisect:

FFREPORT=file=$logfile.log:level=99 ./ffmpeg -report -v 9 -hwaccel cuvid -c:v mpeg2_cuvid \
        -i $INFILE -c:v h264_nvenc -rc-lookahead 24 -bf 3 \
        $OUTFILE 2>&1 | tee $logfile.cli.log

With this I get an error about not having enough surfaces, but if I change the cli, then I can get the error about not having an encoding device.

I don't know the ffmpeg source very well, but based on the comment from the commit there was a change in how input files are initially probed. I believe that changed how cuvid is initialized which leads to there being no free surfaces.

I have attached the log from the suspect commit. I can provide other logs from my git-bisect if requested.

[jgirot]

Log from last good commit

[Timo R.]

43c417ac1adb89cd2c0fa5e5b6a6d955c792b42d