Opened 8 months ago

Last modified 5 months ago

#6260 reopened defect

NVENC H.264 encoding causes aliasing / stair-stepping colors, worse on interlaced than progressive

Reported by: alexpigment Owned by:
Priority: normal Component: avcodec
Version: git-master Keywords: NVENC
Cc: Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: yes

Description

Summary of the bug:

When encoding to H.264 with the NVENC encoder, I'm seeing a pronounced aliasing or stair-stepping effect that happens on the colors (chroma-only?) during motion. This occurs on progressive encodes, but is worse on interlaced encodes. I have verified this behavior on both an Nvidia 1060 and an Nvidia 750 while using the newest static Win64 build from Zeranoe as well as a build from January.

To rule out a problem with the GPU, driver, or NVENC itself, I used Staxrip x264 1.4.0.0 to encode with NVENC H.264 on 2 samples and saw no issues. Therefore, this issue appears to be specific to the FFMPEG implementation of NVENC. FFMPEG x264 does not suffer from this issue either.

How to reproduce:

% ffmpeg -i interlacedsample.mp4 -c:v h264_nvenc -b:v 20000000 -flags +ilme+ildct interlacedoutput.mp4

View the file in a player that supports deinterlacing (I'm using WMP 12 for consistency). Notice that the colors of the rings have massive stair-stepping around the curves, particularly in the lower right where the red and green intersect.

As mentioned above, this also occurs on progressive, albeit to a lesser but still very problematic degree.

% ffmpeg -i progressivesample.mp4 -c:v h264_nvenc -b:v 20000000 progressiveoutput.mp4

The same problem occurs here, most notably at hte top of the blue right and at the intersection of the green and red rings at the bottom left.

ffmpeg version N-84348-gdb7a05d
built on 2017-03-21

Attachments (5)

interlacedsample.mp4 (2.1 MB) - added by alexpigment 8 months ago.
This is the input file as used in the first part of the reproduction steps.
progressivesample.mp4 (1.5 MB) - added by alexpigment 8 months ago.
This is the input file as used in the second part of the reproduction steps.
interlacedoutput.mp4 (2.4 MB) - added by alexpigment 8 months ago.
This is the problematic output file that results from the first part of the reproduction steps (bitrate lowered for file size considerations).
progressiveoutput.mp4 (2.1 MB) - added by alexpigment 8 months ago.
This is the problematic output file that results from the second part of the reproduction steps (bitrate lowered for file size considerations).
stairstepping.png (1.9 MB) - added by alexpigment 5 months ago.
Screenshot of "interlacedoutput.mp4" created in build after 'fix' was committed; still seeing the same issue. See bottom right corner where green and red rings meet. Same is true for yellow and blue rings.

Change History (12)

Changed 8 months ago by alexpigment

This is the input file as used in the first part of the reproduction steps.

Changed 8 months ago by alexpigment

This is the input file as used in the second part of the reproduction steps.

Changed 8 months ago by alexpigment

This is the problematic output file that results from the first part of the reproduction steps (bitrate lowered for file size considerations).

Changed 8 months ago by alexpigment

This is the problematic output file that results from the second part of the reproduction steps (bitrate lowered for file size considerations).

comment:1 Changed 8 months ago by cehoyos

  • Component changed from ffmpeg to undetermined
  • Keywords aliasing interlaced progressive chroma removed
  • Priority changed from important to normal

comment:2 Changed 6 months ago by dpavlekovic

Hi, I'm experiencing the same problem, I originally thought it's the nvenc issue so thanks for reporting. Today I found that this is related to padding of input surface width/height to be divisible by 32 in libavcodec/nvenc.c:nvenc_alloc_surface:

        allocSurf.width = (avctx->width + 31) & ~31;
        allocSurf.height = (avctx->height + 31) & ~31;

If I remove this adjustment then the output looks OK (1920x1080 source, would be padded to 1920x1088). I've no experience though with NVEnc APIs, so not sure if this is a right thing to do for all cases.

Alternatively, keeping the padding and setting the input picture width/height to padded input surface dimensions rather than original frame dims in ff_nvenc_encode_frame seems to work as well:

@@ -1737,8 +1790,8 @@ int ff_nvenc_encode_frame(AVCodecContext *avctx, AVPacket *pkt,

         pic_params.inputBuffer = inSurf->input_surface;
         pic_params.bufferFmt = inSurf->format;
-        pic_params.inputWidth = avctx->width;
-        pic_params.inputHeight = avctx->height;
+        pic_params.inputWidth = inSurf->width;
+        pic_params.inputHeight = inSurf->height;
         pic_params.inputPitch = inSurf->pitch;
         pic_params.outputBitstream = inSurf->output_surface;
Last edited 6 months ago by dpavlekovic (previous) (diff)

comment:3 Changed 6 months ago by oromit

  • Analyzed by developer set
  • Component changed from undetermined to avcodec
  • Reproduced by developer set
  • Resolution set to fixed
  • Status changed from new to closed
  • Version changed from unspecified to git-master

Fixed in a1652aca7e892ea4899df894feaa87ba4170e06c

Also backported to all releases since 3.1.

Last edited 6 months ago by oromit (previous) (diff)

comment:4 Changed 5 months ago by alexpigment

This doesn't appear to be addressed on my end. I've tested the latest Zeranoe build, but I also compiled a nightly Win32 build with the relevant patch applied. See "stairstepping.png" attachment.

Did either dpavlekovic or oromit test the reproduction steps to make sure the fix worked in that scenario?

Changed 5 months ago by alexpigment

Screenshot of "interlacedoutput.mp4" created in build after 'fix' was committed; still seeing the same issue. See bottom right corner where green and red rings meet. Same is true for yellow and blue rings.

comment:5 Changed 5 months ago by alexpigment

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:6 Changed 5 months ago by tbucher

I ran into the same issue with a build from a couple of weeks ago, haven't updated since to verify whether the mentioned fix has changed anything.

At that time I found out that the aliasing artifacts manifest themselves while using nvenc with yuv420p colorspace, enforcing it to nv12 fixed it.

Can you confirm that nv12 works properly for you as well? If yes, it might be a lead for developers to follow.

comment:7 Changed 5 months ago by alexpigment

Yes, adding -pix_fmt nv12 does indeed fix the problem! Thanks for that tip, tbucher.

This does make me wonder what to do with this issue. Do we consider this an issue with yuv420p that needs to be addressed, or should nv12 become the default selection when nothing is explicitly set? I know that with h264_qsv, yuv420p is not even an option and it will fall back to nv12 even if you specify yuv420p explictly.

Note: See TracTickets for help on using tickets.