NVENC H.264 encoding causes aliasing / stair-stepping colors, worse on interlaced than progressive

[alexpigment]

Summary of the bug:

When encoding to H.264 with the NVENC encoder, I'm seeing a pronounced aliasing or stair-stepping effect that happens on the colors (chroma-only?) during motion. This occurs on progressive encodes, but is worse on interlaced encodes. I have verified this behavior on both an Nvidia 1060 and an Nvidia 750 while using the newest static Win64 build from Zeranoe as well as a build from January.

To rule out a problem with the GPU, driver, or NVENC itself, I used Staxrip x264 1.4.0.0 to encode with NVENC H.264 on 2 samples and saw no issues. Therefore, this issue appears to be specific to the FFMPEG implementation of NVENC. FFMPEG x264 does not suffer from this issue either.

How to reproduce:

% ffmpeg -i interlacedsample.mp4 -c:v h264_nvenc -b:v 20000000 -flags +ilme+ildct interlacedoutput.mp4

View the file in a player that supports deinterlacing (I'm using WMP 12 for consistency). Notice that the colors of the rings have massive stair-stepping around the curves, particularly in the lower right where the red and green intersect.

As mentioned above, this also occurs on progressive, albeit to a lesser but still very problematic degree.

% ffmpeg -i progressivesample.mp4 -c:v h264_nvenc -b:v 20000000 progressiveoutput.mp4

The same problem occurs here, most notably at hte top of the blue right and at the intersection of the green and red rings at the bottom left.

ffmpeg version N-84348-gdb7a05d
built on 2017-03-21

[alexpigment]

This is the input file as used in the first part of the reproduction steps.

[alexpigment]

This is the input file as used in the second part of the reproduction steps.

[alexpigment]

This is the problematic output file that results from the first part of the reproduction steps (bitrate lowered for file size considerations).

[alexpigment]

This is the problematic output file that results from the second part of the reproduction steps (bitrate lowered for file size considerations).

[dpavlekovic]

Hi, I'm experiencing the same problem, I originally thought it's the nvenc issue so thanks for reporting. Today I found that this is related to padding of input surface width/height to be divisible by 32 in libavcodec/nvenc.c:nvenc_alloc_surface:

        allocSurf.width = (avctx->width + 31) & ~31;
        allocSurf.height = (avctx->height + 31) & ~31;

If I remove this adjustment then the output looks OK (1920x1080 source, would be padded to 1920x1088). I've no experience though with NVEnc APIs, so not sure if this is a right thing to do for all cases.

Alternatively, keeping the padding and setting the input picture width/height to padded input surface dimensions rather than original frame dims in ff_nvenc_encode_frame seems to work as well:

@@ -1737,8 +1790,8 @@ int ff_nvenc_encode_frame(AVCodecContext *avctx, AVPacket *pkt,

         pic_params.inputBuffer = inSurf->input_surface;
         pic_params.bufferFmt = inSurf->format;
-        pic_params.inputWidth = avctx->width;
-        pic_params.inputHeight = avctx->height;
+        pic_params.inputWidth = inSurf->width;
+        pic_params.inputHeight = inSurf->height;
         pic_params.inputPitch = inSurf->pitch;
         pic_params.outputBitstream = inSurf->output_surface;

[Timo R.]

Fixed in a1652aca7e892ea4899df894feaa87ba4170e06c

Also backported to all releases since 3.1.

[alexpigment]

This doesn't appear to be addressed on my end. I've tested the latest Zeranoe build, but I also compiled a nightly Win32 build with the relevant patch applied. See "stairstepping.png" attachment.

Did either dpavlekovic or oromit test the reproduction steps to make sure the fix worked in that scenario?

[alexpigment]

Screenshot of "interlacedoutput.mp4" created in build after 'fix' was committed; still seeing the same issue. See bottom right corner where green and red rings meet. Same is true for yellow and blue rings.

[tbucher]

I ran into the same issue with a build from a couple of weeks ago, haven't updated since to verify whether the mentioned fix has changed anything.

At that time I found out that the aliasing artifacts manifest themselves while using nvenc with yuv420p colorspace, enforcing it to nv12 fixed it.

Can you confirm that nv12 works properly for you as well? If yes, it might be a lead for developers to follow.

[alexpigment]

Yes, adding -pix_fmt nv12 does indeed fix the problem! Thanks for that tip, tbucher.

This does make me wonder what to do with this issue. Do we consider this an issue with yuv420p that needs to be addressed, or should nv12 become the default selection when nothing is explicitly set? I know that with h264_qsv, yuv420p is not even an option and it will fall back to nv12 even if you specify yuv420p explictly.