Opened 4 years ago

Last modified 6 months ago

#8916 open defect

Incorrect field order indication when encoding interlace top field first using h264_nvenc

Reported by: Ptaah Owned by:
Priority: important Component: avcodec
Version: git-master Keywords: nvenc regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
When encoding top field first interlaced video (or making one using tinterlace filter), the resulting h264 bitstream is marked as bottom field first. This behaviour was not present prior version 4.0.
How to reproduce:

% ffmpeg -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -an output.mp4
% ffprobe -i output.mp4 -show_entries "frame=interlaced_frame,top_field_first" -pretty

[FRAME]
interlaced_frame=1
top_field_first=0
[/FRAME]

Change History (27)

comment:1 by Carl Eugen Hoyos, 4 years ago

Keywords: regression added; avcodec field tff bff removed

Which commit introduced the regression?
Please provide the command line you tested together with the complete, uncut console output to make this a valid ticket.

comment:2 by Ptaah, 4 years ago

I don't know, which commit introduced it, I would've tested it more thoroughly, but Zeranoe is apparently out of business and I don't have many ffmpeg windows build. What I can say is, it worked correctly with ffmpeg 3.3.2 and it doesn't work with ffmpeg 4.1.1 all the way up to current build.

.\ffmpeg.exe -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -an output.mp4
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20190212
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'interlaced.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.73.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 1857 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 1856 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
Output #0, mp4, to 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: h264 (h264_nvenc) (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.35.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
frame=  500 fps=354 q=35.0 Lsize=    4532kB time=00:00:19.96 bitrate=1860.2kbits/s speed=14.1x
video:4530kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.062222%

comment:3 by hydra3333, 3 years ago

Last edited 3 years ago by hydra3333 (previous) (diff)

comment:4 by hydra3333, 3 years ago

perhaps roll out own windows ffmpeg.exe build using MABS under Windows: ​https://github.com/m-ab-s/media-autobuild_suite

or, create a vmware ubuntu 20.04 VM and cross-compile ffmpeg.exe for windows using rdp's cross-compilation script: ​https://github.com/rdp/ffmpeg-windows-build-helpers

only if you are really really terribly desperate, create a vmware ubuntu 20.04 VM and cross-compile ffmpeg.exe for windows using the buggy often-not-working thing: ​https://github.com/hydra3333/h3333_python_cross_compile_script_v100

comment:5 by Carl Eugen Hoyos, 3 years ago

Far easier is of course to use wsl.

comment:6 by mablip, 3 years ago

Strange thing is if you add a B frame with -bf 1 you will have a correct flag.

% ffmpeg -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -bf 1 -an output.mp4
% ffprobe -i output.mp4 -show_entries "frame=interlaced_frame,top_field_first" -pretty

gives you:

[FRAME]
interlaced_frame=1
top_field_first=1
FRAME

Tested with latest nighlty build and nvidia drivers.

Last edited 3 years ago by mablip (previous) (diff)

comment:7 by mablip, 3 years ago

ffmpeg -i interlaced.mp4 -codec:v h264_nvenc -flags +ildct+ilme -pix_fmt yuv420p -an output.mp4

up to 3.3.4 looks ok:
[FRAME]
interlaced_frame=1
top_field_first=1

3.3.5 Introduces this problem

[FRAME]
interlaced_frame=1
top_field_first=0
FRAME

Last edited 3 years ago by mablip (previous) (diff)

comment:8 by Balling, 3 years ago

Status: new → open

You should use ​https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

comment:9 by Ptaah, 3 years ago

Great work, thanks. I will check what has changed in h264 nvenc implementation between those versions.

comment:10 by Ptaah, 3 years ago

I didn't find any relevant changes between 3.3.4 and 3.3.5 version. Where did you get 3.3.4 and 3.3.5 binaries? I would like to check what git commit are they based on.

comment:11 by mablip, 3 years ago

I just realized. Might have been 3.4 but labeled as 3.3.5. I googled old release to avoid the hassle of compiling.

Double checked and 3.4.0 gives you the error so its definitely introduced between 3.3.4 and 3.4

Here you can find a lot of the older versions complied for win
​https://www.videohelp.com/software/ffmpeg/old-versions

​https://github.com/FutaAlice/ffmpeg-static-libs

Last edited 3 years ago by mablip (previous) (diff)

in reply to:  8 comment:12 by hydra3333, 3 years ago

Replying to Balling:

You should use ​https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

Thank you Balling.
(I prefer ffmpeg built with OpenCL, fdk-aac, h264-8/10/12bit, h265-multibit, av1, nvenc, so I'll have a look at that link !)

in reply to:  8 ; comment:13 by hydra3333, 3 years ago

Replying to Balling:

You should use ​https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

Thank you Balling.
(I prefer ffmpeg built with OpenCL, fdk-aac, h264-8/10/12bit, h265-multibit, av1, nvenc, so I'll have a look at that link !)

edit: ah, I preferred not to install docker. Oh well.

Last edited 3 years ago by hydra3333 (previous) (diff)

in reply to:  10 ; comment:14 by mablip, 3 years ago

Replying to Ptaah:

I didn't find any relevant changes between 3.3.4 and 3.3.5 version. Where did you get 3.3.4 and 3.3.5 binaries? I would like to check what git commit are they based on.

Found anything useful between the versions?

in reply to:  13 comment:15 by Balling, 3 years ago

Replying to hydra3333:

Replying to Balling:

You should use ​https://github.com/BtbN/FFmpeg-Builds/ now, hydra3333

Thank you Balling.
(I prefer ffmpeg built with OpenCL, fdk-aac, h264-8/10/12bit, h265-multibit, av1, nvenc, so I'll have a look at that link !)

edit: ah, I preferred not to install docker. Oh well.

You do not need to use docker. ​https://github.com/BtbN/FFmpeg-Builds/releases those do have OpenCL, h264 is 12 bit by default, there is no such stuff anymore, there is no such library as av1 and nvenc is obviously on, as it is nvidia dev's repo.

in reply to:  14 comment:16 by Ptaah, 3 years ago

Replying to mablip:

Replying to Ptaah:

I didn't find any relevant changes between 3.3.4 and 3.3.5 version. Where did you get 3.3.4 and 3.3.5 binaries? I would like to check what git commit are they based on.

Found anything useful between the versions?

Unfortunately I've been otherwise occupied for the past couple of weeks, but I have it in my todo list.

comment:17 by Ptaah, 3 years ago

I've made some progress today. First I identified the range of versions in between which it got broken:
It works with avcodec 57.102.100 from 1.7.2017
It doesn't work with avcodec 57.105.100 from 1.9.2017

However I didn't find any significant changes in nvenc source files between these dates.

Then I went all in and began to parse the bitstream myself. What I noticed is, the good version generates bitstream with no SEI messages (simply SPS, PPS, IDR, P, ...), BUT the bad version generates two SEI messages after SPS and PPS. Second SEI has within itself pic_timing payload and it has pic_struct with value of 1 (H264_SEI_PIC_STRUCT_TOP_FIELD). After keyframe (which itself is top field) there is another SEI message with pic struct with value of 2 (H264_SEI_PIC_STRUCT_BOTTOM_FIELD).

Now let's look at the avcodec h264 decoder, specifically how it's determining field order in h264 bitstream.

h264_slice.c, line:1198 

1198    if (cur->field_poc[0] != cur->field_poc[1]) {
1199        /* Derive top_field_first from field pocs. */
1200        out->top_field_first = cur->field_poc[0] < cur->field_poc[1];
1201    } else {
1202        if (sps->pic_struct_present_flag && h->sei.picture_timing.present) {
1203            /* Use picture timing SEI information. Even if it is a
1204             * information of a past frame, better than nothing. */
1205            if (h->sei.picture_timing.pic_struct == H264_SEI_PIC_STRUCT_TOP_BOTTOM ||
1206                h->sei.picture_timing.pic_struct == H264_SEI_PIC_STRUCT_TOP_BOTTOM_TOP)
1207                out->top_field_first = 1;
1208            else
1209                out->top_field_first = 0;
1210        } else if (out->interlaced_frame) {
1211            /* Default to top field first when pic_struct_present_flag
1212             * is not set but interlaced frame detected */
1213            out->top_field_first = 1;
1214        } else {
1215            /* Most likely progressive */
1216            out->top_field_first = 0;
1217        }
1218    }

When parsing bitstream generated by older version (without SEI) it either goes to line 1200, or 1213 (I'm not sure). However, when parsing bitstream generated by newer version (with SEI), it DEFINITELY goes to line 1209. Now it becomes clear. It's not a bug in nvenc implementation, it's seems to be a bug in avcodec h264 decoder, because it is not properly handling h264 interlaced with separated fields and SEI messages present. The reason might be simply because MBAFF is wildly more used than separated fields.

This hypothesis is supported by the fact, that MediaInfo correctly detects top field first with those specific files.

So the quick, even if not optimal, solution, is to strip output bitstream of all SEI. It will still work and the field order will be correctly detected by libavcodec.

One question however remains. Why nvenc started to output SEI nalu in h264 bitstream? I have no answer.

Last edited 3 years ago by Ptaah (previous) (diff)

comment:18 by Balling, 3 years ago

That is the point, in hevc, for example, fields can be NOT interlaced, but progressive, see #5514. You must look into interlaced flag as said in #7008 in AVC in .MTS.

Also see #5818. The same issue in mxf and even soft telecine flags are not supported, #2602, LOL.

Last edited 3 years ago by Balling (previous) (diff)

in reply to:  19 comment:20 by Ptaah, 3 years ago

I don't think so, the bug is present in avcodec h264 decoder.

Replying to Balling:

Is this fixing this? ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210721172916.40739-1-jeebjp@gmail.com/

comment:21 by Balling, 3 years ago

Does setting -vf setfield=tff fixes the issue? Also what about -bf -1 which is the default now?

That will mean this is duplicate of #9320 and #9339.

Last edited 3 years ago by Balling (previous) (diff)

in reply to:  21 ; comment:22 by Ptaah, 3 years ago

Replying to Balling:

Does setting -vf setfield=tff fixes the issue? Also what about -bf -1 which is the default now?

That will mean this is duplicate of #9320 and #9339.

Just checked FFmpeg 4.4 and the issue seems to be resolved.

in reply to:  22 comment:23 by Balling, 3 years ago

Replying to Ptaah:

Replying to Balling:

Does setting -vf setfield=tff fixes the issue? Also what about -bf -1 which is the default now?

That will mean this is duplicate of #9320 and #9339.

Just checked FFmpeg 4.4 and the issue seems to be resolved.

Is that because of 4c694093be68d401c60819e5171817c62afef8b2? Please test with -bf 0.

Last edited 3 years ago by Balling (previous) (diff)

comment:24 by Balling, 3 years ago

Hi, again. Can you test with -bf 0? I only have 2080 Ti, which does not support interlaced already, so cannot test.

comment:25 by Balling, 11 months ago

Can you retest?

in reply to:  25 ; comment:26 by Ptaah, 11 months ago

Replying to Balling:

Can you retest?

Seems like it got worse somehow:

[FRAME]
interlaced_frame=0
top_field_first=1
[/FRAME]

ffmpeg version is 2023-05-25-git-944243477b, nvidia driver 525.

in reply to:  26 comment:27 by mablip, 6 months ago

Replying to Ptaah:

Replying to Balling:

Can you retest?

Seems like it got worse somehow:

[FRAME]
interlaced_frame=0
top_field_first=1
[/FRAME]

ffmpeg version is 2023-05-25-git-944243477b, nvidia driver 525.

Seems impossible after Nvidia driver ->520 with cuda 12 and forward to actually get an interlaced_frame. Everything fails for me with drivers above 520, impossible to get the correct flag.

Note: See TracTickets for help on using tickets.