Opened 13 months ago

Last modified 2 months ago

#8916 open defect

Incorrect field order indication when encoding interlace top field first using h264_nvenc

Reported by: Ptaah Owned by:
Priority: important Component: avcodec
Version: git-master Keywords: nvenc regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
When encoding top field first interlaced video (or making one using tinterlace filter), the resulting h264 bitstream is marked as bottom field first. This behaviour was not present prior version 4.0.
How to reproduce:

% ffmpeg -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -an output.mp4
% ffprobe -i output.mp4 -show_entries "frame=interlaced_frame,top_field_first" -pretty

[FRAME]
interlaced_frame=1
top_field_first=0
[/FRAME]

Change History (24)

comment:1 by Carl Eugen Hoyos, 13 months ago

Keywords: regression added; avcodec field tff bff removed

Which commit introduced the regression?
Please provide the command line you tested together with the complete, uncut console output to make this a valid ticket.

comment:2 by Ptaah, 13 months ago

I don't know, which commit introduced it, I would've tested it more thoroughly, but Zeranoe is apparently out of business and I don't have many ffmpeg windows build. What I can say is, it worked correctly with ffmpeg 3.3.2 and it doesn't work with ffmpeg 4.1.1 all the way up to current build.

.\ffmpeg.exe -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -an output.mp4
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20190212
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'interlaced.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.73.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 1857 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 1856 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
Output #0, mp4, to 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: h264 (h264_nvenc) (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.35.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
frame=  500 fps=354 q=35.0 Lsize=    4532kB time=00:00:19.96 bitrate=1860.2kbits/s speed=14.1x
video:4530kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.062222%

comment:3 by hydra3333, 11 months ago

Last edited 11 months ago by hydra3333 (previous) (diff)

comment:4 by hydra3333, 11 months ago

perhaps roll out own windows ffmpeg.exe build using MABS under Windows: https://github.com/m-ab-s/media-autobuild_suite

or, create a vmware ubuntu 20.04 VM and cross-compile ffmpeg.exe for windows using rdp's cross-compilation script: https://github.com/rdp/ffmpeg-windows-build-helpers

only if you are really really terribly desperate, create a vmware ubuntu 20.04 VM and cross-compile ffmpeg.exe for windows using the buggy often-not-working thing: https://github.com/hydra3333/h3333_python_cross_compile_script_v100

comment:5 by Carl Eugen Hoyos, 11 months ago

Far easier is of course to use wsl.

comment:6 by mablip, 9 months ago

Strange thing is if you add a B frame with -bf 1 you will have a correct flag.

% ffmpeg -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -bf 1 -an output.mp4
% ffprobe -i output.mp4 -show_entries "frame=interlaced_frame,top_field_first" -pretty

gives you:

[FRAME]
interlaced_frame=1
top_field_first=1
FRAME

Tested with latest nighlty build and nvidia drivers.

Last edited 9 months ago by mablip (previous) (diff)

comment:7 by mablip, 9 months ago

ffmpeg -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -pix_fmt yuv420p -an output.mp4

up to 3.3.4 looks ok:
[FRAME]
interlaced_frame=1
top_field_first=1

3.3.5 Introduces this problem

[FRAME]
interlaced_frame=1
top_field_first=0
FRAME

Last edited 9 months ago by mablip (previous) (diff)

comment:8 by Balling, 9 months ago

Status: newopen

You should use https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

comment:9 by Ptaah, 9 months ago

Great work, thanks. I will check what has changed in h264 nvenc implementation between those versions.

comment:10 by Ptaah, 9 months ago

I didn't find any relevant changes between 3.3.4 and 3.3.5 version. Where did you get 3.3.4 and 3.3.5 binaries? I would like to check what git commit are they based on.

comment:11 by mablip, 9 months ago

I just realized. Might have been 3.4 but labeled as 3.3.5. I googled old release to avoid the hassle of compiling.

Double checked and 3.4.0 gives you the error so its definitely introduced between 3.3.4 and 3.4

Here you can find a lot of the older versions complied for win
https://www.videohelp.com/software/ffmpeg/old-versions

https://github.com/FutaAlice/ffmpeg-static-libs

Last edited 9 months ago by mablip (previous) (diff)

in reply to:  8 comment:12 by hydra3333, 9 months ago

Replying to Balling:

You should use https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

Thank you Balling.
(I prefer ffmpeg built with OpenCL, fdk-aac, h264-8/10/12bit, h265-multibit, av1, nvenc, so I'll have a look at that link !)

in reply to:  8 ; comment:13 by hydra3333, 9 months ago

Replying to Balling:

You should use https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

Thank you Balling.
(I prefer ffmpeg built with OpenCL, fdk-aac, h264-8/10/12bit, h265-multibit, av1, nvenc, so I'll have a look at that link !)

edit: ah, I preferred not to install docker. Oh well.

Last edited 9 months ago by hydra3333 (previous) (diff)

in reply to:  10 ; comment:14 by mablip, 9 months ago

Replying to Ptaah:

I didn't find any relevant changes between 3.3.4 and 3.3.5 version. Where did you get 3.3.4 and 3.3.5 binaries? I would like to check what git commit are they based on.

Found anything useful between the versions?

in reply to:  13 comment:15 by Balling, 9 months ago

Replying to hydra3333:

Replying to Balling:

You should use https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

Thank you Balling.
(I prefer ffmpeg built with OpenCL, fdk-aac, h264-8/10/12bit, h265-multibit, av1, nvenc, so I'll have a look at that link !)

edit: ah, I preferred not to install docker. Oh well.

You do not need to use docker. https://github.com/BtbN/FFmpeg-Builds/releases those do have OpenCL, h264 is 12 bit by default, there is no such stuff anymore, there is no such library as av1 and nvenc is obviously on, as it is nvidia dev's repo.

in reply to:  14 comment:16 by Ptaah, 9 months ago

Replying to mablip:

Replying to Ptaah:

I didn't find any relevant changes between 3.3.4 and 3.3.5 version. Where did you get 3.3.4 and 3.3.5 binaries? I would like to check what git commit are they based on.

Found anything useful between the versions?

Unfortunately I've been otherwise occupied for the past couple of weeks, but I have it in my todo list.

comment:17 by Ptaah, 8 months ago

I've made some progress today. First I identified the range of versions in between which it got broken:
It works with avcodec 57.102.100 from 1.7.2017
It doesn't work with avcodec 57.105.100 from 1.9.2017

However I didn't find any significant changes in nvenc source files between these dates.

Then I went all in and began to parse the bitstream myself. What I noticed is, the good version generates bitstream with no SEI messages (simply SPS, PPS, IDR, P, ...), BUT the bad version generates two SEI messages after SPS and PPS. Second SEI has within itself pic_timing payload and it has pic_struct with value of 1 (H264_SEI_PIC_STRUCT_TOP_FIELD). After keyframe (which itself is top field) there is another SEI message with pic struct with value of 2 (H264_SEI_PIC_STRUCT_BOTTOM_FIELD).

Now let's look at the avcodec h264 decoder, specifically how it's determining field order in h264 bitstream.

h264_slice.c, line:1198

1198    if (cur->field_poc[0] != cur->field_poc[1]) {
1199        /* Derive top_field_first from field pocs. */
1200        out->top_field_first = cur->field_poc[0] < cur->field_poc[1];
1201    } else {
1202        if (sps->pic_struct_present_flag && h->sei.picture_timing.present) {
1203            /* Use picture timing SEI information. Even if it is a
1204             * information of a past frame, better than nothing. */
1205            if (h->sei.picture_timing.pic_struct == H264_SEI_PIC_STRUCT_TOP_BOTTOM ||
1206                h->sei.picture_timing.pic_struct == H264_SEI_PIC_STRUCT_TOP_BOTTOM_TOP)
1207                out->top_field_first = 1;
1208            else
1209                out->top_field_first = 0;
1210        } else if (out->interlaced_frame) {
1211            /* Default to top field first when pic_struct_present_flag
1212             * is not set but interlaced frame detected */
1213            out->top_field_first = 1;
1214        } else {
1215            /* Most likely progressive */
1216            out->top_field_first = 0;
1217        }
1218    }

When parsing bitstream generated by older version (without SEI) it either goes to line 1200, or 1213 (I'm not sure). However, when parsing bitstream generated by newer version (with SEI), it DEFINITELY goes to line 1209. Now it becomes clear. It's not a bug in nvenc implementation, it's seems to be a bug in avcodec h264 decoder, because it is not properly handling h264 interlaced with separated fields and SEI messages present. The reason might be simply because MBAFF is wildly more used than separated fields.

This hypothesis is supported by the fact, that MediaInfo correctly detects top field first with those specific files.

So the quick, even if not optimal, solution, is to strip output bitstream of all SEI. It will still work and the field order will be correctly detected by libavcodec.

One question however remains. Why nvenc started to output SEI nalu in h264 bitstream? I have no answer.

Last edited 8 months ago by Ptaah (previous) (diff)

comment:18 by Balling, 8 months ago

That is the point, in hevc, for example, fields can be NOT interlaced, but progressive, see #5514. You must look into interlaced flag as said in #7008 in AVC in .MTS.

Also see #5818. The same issue in mxf and even soft telecine flags are not supported, #2602, LOL.

Last edited 8 months ago by Balling (previous) (diff)

in reply to:  19 comment:20 by Ptaah, 3 months ago

I don't think so, the bug is present in avcodec h264 decoder.

Replying to Balling:

Is this fixing this? https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210721172916.40739-1-jeebjp@gmail.com/

comment:21 by Balling, 2 months ago

Does setting -vf setfield=tff fixes the issue? Also what about -bf -1 which is the default now?

That will mean this is duplicate of #9320 and #9339.

Last edited 2 months ago by Balling (previous) (diff)

in reply to:  21 ; comment:22 by Ptaah, 2 months ago

Replying to Balling:

Does setting -vf setfield=tff fixes the issue? Also what about -bf -1 which is the default now?

That will mean this is duplicate of #9320 and #9339.

Just checked FFmpeg 4.4 and the issue seems to be resolved.

in reply to:  22 comment:23 by Balling, 2 months ago

Replying to Ptaah:

Replying to Balling:

Does setting -vf setfield=tff fixes the issue? Also what about -bf -1 which is the default now?

That will mean this is duplicate of #9320 and #9339.

Just checked FFmpeg 4.4 and the issue seems to be resolved.

Is that because of 4c694093be68d401c60819e5171817c62afef8b2? Please test with -bf 0.

Last edited 2 months ago by Balling (previous) (diff)

comment:24 by Balling, 2 months ago

Hi, again. Can you test with -bf 0? I only have 2080 Ti, which does not support interlaced already, so cannot test.

Note: See TracTickets for help on using tickets.