Enhancement request for the detelecine filter

[mkver]

Hello,

the detelecine filter currently does not use full frames even when they are available in the source but combines two fields from different frames. This results in suboptimal output, in particular for progressively encoded input files with hard telecining (I know that encoding such stuff progressively without IVTC is stupid; but it's nevertheless done). My attachment contains such an example: A five frame source (one complete 2:3 pulldown period (top field first)) with hard telecining. The underlying four frame sequence (on which the pulldown has been applied) has a scenecut between the third and the fourth frame so that the fourth frame of the telecined video is a mixture of two fields from totally different frames; plus errors introduced in the encoding which entails that the bottom fields of frames four and five do not coincide and the bottom field of frame four and the top field of frame five do not match very well. In reconstructing the fourth frame of the untelecined video one should therefore use the top and bottom fields of the fifth frame of the telecined video; but that is not what the detelecine filter does.

[mkver]

This is one five frame 2:3 pulldown period (top field first).

[Carl Eugen Hoyos]

To make this a valid ticket, please provide the command line that you tested including the complete, uncut console output.
But please also explain if you believe there is a bug in the current detelecine filter: If not, I don't think there is anything to enhance: The filter is simply meant to undo the effects of the telecine filter so ideally your report should contain a progressive source and a command line with the telecine filter. If the effect can't be undone with the detelecine filter, you have found a bug.
And please do not attach output files, this always leads to confusion.

[Carl Eugen Hoyos]

After reading your report a few times, I believe I understand the issue: Are you sure that it is never a disadvantage to use the field from the complete frames instead of from the mixed one?

[mkver]

The fourth image of a 2:3 telecined video.

[mkver]

The fifth image of a 2:3 telecine period.

[mkver]

This is the fourth picture of the output of the detelecine filter.

[mkver]

Sorry for not including the command line and the console output. I read the bug report guidelines, but I did not think of this as a bug, but merely as a suggestion for an enhancement (therefore I also did not set the type of this ticket to "defect"). Anyway, here is it (when I initially did this I did of course not use the "-v 9 -loglevel 99 -report" parameters):

ffmpeg started on 2016-06-24 at 03:08:00
Report written to "ffmpeg-20160624-030800.log"
Command line:
ffmpeg -v 9 -loglevel 99 -report -i Hardtelecine.mkv -vf detelecine -c:v libx264 -qp 0 Hardtelecine.detelecined.mkv
ffmpeg version N-80256-g0a9e781 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --disable-w32threads --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmfx --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 24.100 / 55. 24.100
  libavcodec     57. 45.100 / 57. 45.100
  libavformat    57. 37.101 / 57. 37.101
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 46.101 /  6. 46.101
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Reading option '-i' ... matched as input file with argument 'Hardtelecine.mkv'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'detelecine'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'libx264'.
Reading option '-qp' ... matched as AVOption 'qp' with argument '0'.
Reading option 'Hardtelecine.detelecined.mkv' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Applying option loglevel (set logging level) with argument 99.
Applying option report (generate a report) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input file Hardtelecine.mkv.
Successfully parsed a group of options.
Opening an input file: Hardtelecine.mkv.
[file @ 00000000004b69e0] Setting default whitelist 'file,crypto'
[matroska,webm @ 00000000004b5300] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 50000 from timebase
[matroska,webm @ 00000000004b5300] Before avformat_find_stream_info() pos: 233 bytes read:32768 seeks:0
[h264 @ 00000000004c8640] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 00000000004c8640] nal_unit_type: 8, nal_ref_idc: 3
[h264 @ 00000000004c8640] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 00000000004c8640] nal_unit_type: 8, nal_ref_idc: 3
[h264 @ 00000000004c8640] user data:"x264 - core 148 r2694 3b70645 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=16 deblock=1:0:0 analyse=0x1:0x131 me=umh subme=9 psy=0 mixed_ref=1 me_range=24 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0"
[h264 @ 00000000004c8640] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 00000000004c8640] nal_unit_type: 5, nal_ref_idc: 3
[h264 @ 00000000004c8640] user data:"x264 - core 148 r2694 3b70645 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=16 deblock=1:0:0 analyse=0x1:0x131 me=umh subme=9 psy=0 mixed_ref=1 me_range=24 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0"
[h264 @ 00000000004c8640] nal_unit_type: 1, nal_ref_idc: 2
[h264 @ 00000000004c8640] nal_unit_type: 1, nal_ref_idc: 2
[h264 @ 00000000004c8640] nal_unit_type: 1, nal_ref_idc: 2
[h264 @ 00000000004c8640] nal_unit_type: 1, nal_ref_idc: 2
[matroska,webm @ 00000000004b5300] decoding for stream 0 failed
[matroska,webm @ 00000000004b5300] After avformat_find_stream_info() pos: 396347 bytes read:396347 seeks:0 frames:5
Input #0, matroska,webm, from 'Hardtelecine.mkv':
  Metadata:
    encoder         : Haali Matroska Writer b0
  Duration: 00:00:00.17, bitrate: 19005 kb/s
    Stream #0:0(eng), 5, 1/20000: Video: h264 (High 4:4:4 Predictive), 1 reference frame, yuv420p(left), 718x480 (0x0) [SAR 8:9 DAR 359:270], 0/1, 29.97 fps, 29.97 tbr, 20k tbn, 59.94 tbc (default)
Successfully opened the file.
Parsing a group of options: output file Hardtelecine.detelecined.mkv.
Applying option vf (set video filters) with argument detelecine.
Applying option c:v (codec name) with argument libx264.
Successfully parsed a group of options.
Opening an output file: Hardtelecine.detelecined.mkv.
[file @ 0000000002243b40] Setting default whitelist 'file,crypto'
Successfully opened the file.
detected 4 logical cores
[Parsed_detelecine_0 @ 00000000004bc2a0] Detelecine pattern 23 removes up to 2 frames per frame, pts advance factor: 5/4
[graph 0 input from stream 0:0 @ 00000000004bcc20] Setting 'video_size' to value '718x480'
[graph 0 input from stream 0:0 @ 00000000004bcc20] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 00000000004bcc20] Setting 'time_base' to value '1/20000'
[graph 0 input from stream 0:0 @ 00000000004bcc20] Setting 'pixel_aspect' to value '8/9'
[graph 0 input from stream 0:0 @ 00000000004bcc20] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 00000000004bcc20] Setting 'frame_rate' to value '30000/1001'
[graph 0 input from stream 0:0 @ 00000000004bcc20] w:718 h:480 pixfmt:yuv420p tb:1/20000 fr:30000/1001 sar:8/9 sws_param:flags=2
[format @ 00000000004bd0a0] compat: called with args=[yuv420p|yuvj420p|yuv422p|yuvj422p|yuv444p|yuvj444p|nv12|nv16|nv21]
[format @ 00000000004bd0a0] Setting 'pix_fmts' to value 'yuv420p|yuvj420p|yuv422p|yuvj422p|yuv444p|yuvj444p|nv12|nv16|nv21'
[AVFilterGraph @ 00000000022058a0] query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
[Parsed_detelecine_0 @ 00000000004bc2a0] FPS: 30000/1001 -> 24000/1001
[Parsed_detelecine_0 @ 00000000004bc2a0] TB: 1/20000 -> 1/16000
[h264 @ 00000000004bdbc0] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 00000000004bdbc0] nal_unit_type: 8, nal_ref_idc: 3
[libx264 @ 0000000002205300] using mv_range_thread = 40
[libx264 @ 0000000002205300] using SAR=8/9
[libx264 @ 0000000002205300] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0000000002205300] profile High 4:4:4 Predictive, level 3.0, 4:2:0 8-bit
[libx264 @ 0000000002205300] 264 - core 148 r2694 3b70645 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=0 mixed_ref=1 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=23 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0
[matroska @ 0000000002242be0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Output #0, matroska, to 'Hardtelecine.detelecined.mkv':
  Metadata:
    encoder         : Lavf57.37.101
    Stream #0:0(eng), 0, 1/1000: Video: h264 (libx264), 1 reference frame (H264 / 0x34363248), yuv420p(left), 718x480 (0x0) [SAR 8:9 DAR 359:270], 0/1, q=-1--1, 23.98 fps, 1k tbn, 23.98 tbc (default)
    Metadata:
      encoder         : Lavc57.45.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Press [q] to stop, [?] for help
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 00000000004bdbc0] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 00000000004bdbc0] nal_unit_type: 5, nal_ref_idc: 3
[h264 @ 00000000004bdbc0] user data:"x264 - core 148 r2694 3b70645 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=16 deblock=1:0:0 analyse=0x1:0x131 me=umh subme=9 psy=0 mixed_ref=1 me_range=24 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc=cqp mbtree=0 qp=0"
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 0000000002534000] nal_unit_type: 1, nal_ref_idc: 2
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 00000000025344a0] nal_unit_type: 1, nal_ref_idc: 2
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[h264 @ 000000000051efe0] nal_unit_type: 1, nal_ref_idc: 2
[h264 @ 000000000051f480] nal_unit_type: 1, nal_ref_idc: 2
Clipping frame in rate conversion by 0.000008
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
Clipping frame in rate conversion by 0.000481
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[output stream 0:0 @ 00000000004bcea0] EOF on sink link output stream 0:0:default.
No more output streams to write to, finishing.
[libx264 @ 0000000002205300] scene cut at 3 Icost:247409 Pcost:241812 ratio:0.0226 bias:0.0250 gop:3 (imb:1130 pmb:74)
[libx264 @ 0000000002205300] frame=   0 QP=0.00 NAL=3 Slice:I Poc:0   I:1350 P:0    SKIP:0    size=120527 bytes
[matroska @ 0000000002242be0] Writing block at offset 725, size 121056, pts 0, dts 0, duration 42, keyframe 1
[libx264 @ 0000000002205300] frame=   1 QP=0.00 NAL=2 Slice:P Poc:2   I:128  P:1044 SKIP:178  size=34483 bytes
[matroska @ 0000000002242be0] Writing block at offset 121790, size 34483, pts 42, dts 42, duration 42, keyframe 0
[libx264 @ 0000000002205300] frame=   2 QP=0.00 NAL=2 Slice:P Poc:4   I:1047 P:303  SKIP:0    size=122464 bytes
[matroska @ 0000000002242be0] Writing block at offset 156281, size 122464, pts 83, dts 83, duration 42, keyframe 0
[libx264 @ 0000000002205300] frame=   3 QP=0.00 NAL=2 Slice:I Poc:6   I:1350 P:0    SKIP:0    size=107946 bytes
[matroska @ 0000000002242be0] Writing block at offset 278753, size 107946, pts 125, dts 125, duration 42, keyframe 0
[matroska @ 0000000002242be0] end duration = 167
[matroska @ 0000000002242be0] stream 0 end duration = 167
frame=    4 fps=0.0 q=-1.0 Lsize=     378kB time=00:00:00.12 bitrate=24554.7kbits/s speed=0.377x    
video:377kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.203913%
Input file #0 (Hardtelecine.mkv):
  Input stream #0:0 (video): 5 packets read (396068 bytes); 5 frames decoded; 
  Total: 5 packets (396068 bytes) demuxed
Output file #0 (Hardtelecine.detelecined.mkv):
  Output stream #0:0 (video): 4 frames encoded; 4 packets muxed (385949 bytes); 
  Total: 4 packets (385949 bytes) muxed
5 frames successfully decoded, 0 decoding errors
[AVIOContext @ 00000000022057a0] Statistics: 25 seeks, 33 writeouts
[libx264 @ 0000000002205300] frame I:2     Avg QP: 0.00  size:114236
[libx264 @ 0000000002205300] frame P:2     Avg QP: 0.00  size: 78474
[libx264 @ 0000000002205300] mb I  I16..4: 62.0%  0.0% 38.0%
[libx264 @ 0000000002205300] mb P  I16..4: 35.6%  0.0%  7.9%  P16..4: 41.3%  5.0%  3.6%  0.0%  0.0%    skip: 6.6%
[libx264 @ 0000000002205300] 8x8 transform intra:0.0% inter:41.3%
[libx264 @ 0000000002205300] coded y,uvDC,uvAC intra: 99.6% 99.8% 99.8% inter: 65.1% 68.2% 66.9%
[libx264 @ 0000000002205300] i16 v,h,dc,p: 12% 88%  0%  0%
[libx264 @ 0000000002205300] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 67%  1%  2%  3%  2%  2%  2%  1%
[libx264 @ 0000000002205300] i8c dc,h,v,p:  1% 86% 13%  0%
[libx264 @ 0000000002205300] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0000000002205300] ref P L0: 92.0%  3.1%  5.0%
[libx264 @ 0000000002205300] kb/s:18481.68
[AVIOContext @ 00000000004beba0] Statistics: 396347 bytes read, 0 seeks

Because my initial description seems to be harder to understand than I imagined, I'll try it again: and are the last two frames of a 2:3 telecined video. The bottom fields of both frames should coincide, but in practice they don't. The detelecine filter now does not use the latter of these two frames as reconstruction of the fourth frame of the original source from which the telecine has been made. Instead it uses the bottom field of the former and the top field of the latter giving this result:
I choose this example with a scenecut intentionally, but actually the situation arises (albeit not so severly) in non-scenecuts. Half of all the frames output by the detelecine filter are mixtures of two different frames from the telecined source (assuming a usual 2:3 pulldown). It could be as low as one quarter.
I don't see a reason why it should be a disadvantage to use the complete frames instead of the mixed ones. The only videos for which this would be worse are probably videos made for this very purpose: One can of course create take a progressive clip, telecine it and disturb the bottom field of the last frame. But I don't know of any reason why such a sitation should happen in practice.
There is by the way another complication: The detelecine filter accepts input with numbers bigger than 3 so that it is possible that for a frame of the original non-telecine source there are two or more full frame copies of this frame from which one has to choose one. The first complete frame sounds like a sensible strategy in such cases (which are probably rare; I have never ever heard of a pattern with a number >3).