severe filter artifacts (yadif) when applied to 10bit 422 video

[Pavel Koshevoy]

Summary of the bug:
When yadif filter is applied there appear severe noise+interlacing artifacts on the right half of each output frame, and some noise artifacts on the left half of each output frame.

How to reproduce:

$ /Developer/x86_64/bin/ffmpeg -i ~/Movies/VideoBad/QT708.mov -vf yadif=0:0:1 -c:v libx264 -profile:v high422 -x264opts crf=23:keyint=40:ref=3 -c:a copy -y ~/Movies/JobOutput/artifacts.nut
ffmpeg version N-59395-gc09bb23 Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 27 2013 13:01:57 with gcc 4.2.1 (GCC) (Apple Inc. build 5666) (dot 3)
  configuration: --prefix=/Developer/x86_64 --disable-debug --disable-shared --enable-vda --enable-swscale --enable-avfilter --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libspeex --enable-pthreads --enable-gpl --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-postproc --enable-libx264 --enable-libxvid --enable-yasm --enable-runtime-cpudetect --extra-cflags=-I/opt/local/include --extra-ldflags='-headerpad_max_install_names -L/opt/local/lib'
  libavutil      52. 59.100 / 52. 59.100
  libavcodec     55. 46.100 / 55. 46.100
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.103 /  4.  0.103
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x10200e800] multiple edit list entries, a/v desync might occur, patch welcome
    Last message repeated 2 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x10200e800] Could not find codec parameters for stream 2 (Subtitle: none (c708 / 0x38303763), 720x486, 21 kb/s): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/pkoshevoy/Movies/VideoBad/QT708.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2011-02-22 17:06:15
  Duration: 00:00:28.28, start: 0.000000, bitrate: 40290 kb/s
    Stream #0:0(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2011-02-22 17:06:15
      handler_name    : Apple Alias Data Handler
    Stream #0:1(eng): Video: prores (apch / 0x68637061), yuv422p10le, 720x486, 32795 kb/s, SAR 10:11 DAR 400:297, 29.97 fps, 29.97 tbr, 2997 tbn, 2997 tbc (default)
    Metadata:
      creation_time   : 2011-02-22 17:06:15
      handler_name    : Apple Alias Data Handler
    Stream #0:2(eng): Subtitle: none (c708 / 0x38303763), 720x486, 21 kb/s (default)
    Metadata:
      creation_time   : 2011-02-22 17:06:15
      handler_name    : Apple Alias Data Handler
No pixel format specified, yuv422p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x102064400] using SAR=10/11
[libx264 @ 0x102064400] using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64 SlowShuffle
[libx264 @ 0x102064400] profile High 4:2:2, level 3.1, 4:2:2 8-bit
[libx264 @ 0x102064400] 264 - core 136 - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=40 keyint_min=4 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, nut, to '/Users/pkoshevoy/Movies/JobOutput/artifacts.nut':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    encoder         : Lavf55.22.100
    Stream #0:0: Video: h264 (libx264) (H264 / 0x34363248), yuv422p, 720x486 [SAR 10:11 DAR 400:297], q=-1--1, 95904 tbn, 29.97 tbc (default)
    Metadata:
      creation_time   : 2011-02-22 17:06:15
      X-Language      : eng
      handler_name    : Apple Alias Data Handler
    Stream #0:1: Audio: pcm_s16le (PSD[16] / 0x10445350), 48000 Hz, stereo, 1536 kb/s (default)
    Metadata:
      creation_time   : 2011-02-22 17:06:15
      X-Language      : eng
      handler_name    : Apple Alias Data Handler
Stream mapping:
  Stream #0:1 -> #0:0 (prores -> libx264)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  994 fps= 28 q=-1.0 Lsize=   28256kB time=00:00:33.16 bitrate=6979.2kbits/s    
video:22001kB audio:6219kB subtitle:0 global headers:0kB muxing overhead 0.128596%
[libx264 @ 0x102064400] frame I:33    Avg QP:24.89  size: 60085
[libx264 @ 0x102064400] frame P:340   Avg QP:26.24  size: 33669
[libx264 @ 0x102064400] frame B:621   Avg QP:25.55  size: 14651
[libx264 @ 0x102064400] consecutive B-frames: 12.5% 10.7%  6.0% 70.8%
[libx264 @ 0x102064400] mb I  I16..4: 65.9%  8.2% 25.9%
[libx264 @ 0x102064400] mb P  I16..4:  8.6%  2.1%  6.0%  P16..4: 14.0% 10.8%  7.3%  0.0%  0.0%    skip:51.3%
[libx264 @ 0x102064400] mb B  I16..4:  0.2%  0.1%  0.2%  B16..8:  7.0%  1.0%  0.5%  direct:15.3%  skip:75.8%  L0:40.0% L1:46.2% BI:13.8%
[libx264 @ 0x102064400] 8x8 transform intra:11.0% inter:42.1%
[libx264 @ 0x102064400] coded y,uvDC,uvAC intra: 41.0% 76.7% 61.6% inter: 4.1% 24.3% 22.8%
[libx264 @ 0x102064400] i16 v,h,dc,p: 21% 69%  4%  6%
[libx264 @ 0x102064400] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 26% 35%  3%  4%  3%  5%  3%  6%
[libx264 @ 0x102064400] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 52% 18%  4%  3%  3%  4%  2%  4%
[libx264 @ 0x102064400] i8c dc,h,v,p: 44% 44%  7%  5%
[libx264 @ 0x102064400] Weighted P-Frames: Y:1.2% UV:0.6%
[libx264 @ 0x102064400] ref P L0: 43.2% 14.7% 25.0% 17.0%  0.1%
[libx264 @ 0x102064400] ref B L0: 74.3% 22.2%  3.5%
[libx264 @ 0x102064400] ref B L1: 91.2%  8.8%
[libx264 @ 0x102064400] kb/s:5434.09

[Pavel Koshevoy]

A screenshot of the artifacts in the output file

[Pavel Koshevoy]

The source file (136MB) can be accessed here --
​http://pavelbucket.s3.amazonaws.com/ffmpeg-ticket-3250/QT708.mov

[Carl Eugen Hoyos]

Regression since 975110a8

[Carl Eugen Hoyos]

$ ffmpeg -i QT708_cut.mov -vf yadif -qscale 2 out.avi
ffmpeg version N-59396-g4156df5 Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 28 2013 02:28:25 with gcc 4.3 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      52. 59.100 / 52. 59.100
  libavcodec     55. 46.100 / 55. 46.100
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.103 /  4.  0.103
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'QT708_cut.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 512
    compatible_brands: qt
    encoder         : Lavf55.22.100
  Duration: 00:00:00.33, start: 0.018352, bitrate: 63885 kb/s
    Stream #0:0(eng): Video: prores (apch / 0x68637061), yuv422p10le, 720x486, 63927 kb/s, SAR 10:11 DAR 400:297, 29.97 fps, 29.97 tbr, 11988 tbn, 11988 tbc (default)
    Metadata:
      handler_name    : DataHandler
Please use -q:a or -q:v, -qscale is ambiguous
Output #0, avi, to 'out.avi':
  Metadata:
    major_brand     : qt
    minor_version   : 512
    compatible_brands: qt
    ISFT            : Lavf55.22.100
    Stream #0:0(eng): Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 720x486 [SAR 10:11 DAR 400:297], q=2-31, 200 kb/s, 29.97 tbn, 29.97 tbc (default)
    Metadata:
      handler_name    : DataHandler
Stream mapping:
  Stream #0:0 -> #0:0 (prores -> mpeg4)
Press [q] to stop, [?] for help
frame=   10 fps=0.0 q=2.0 Lsize=     789kB time=00:00:00.33 bitrate=19378.6kbits/s
video:783kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.747895%

[Pavel Koshevoy]

Now that you've attached a cut down version of the source file may I remove the original source file? I assume you don't need it anymore...

[Carl Eugen Hoyos]

Can you explain what "c708" is? Does the sample contain subtitles?

[Pavel Koshevoy]

Yes, it has CEA-708 data packets, stored in 'ccdp' atoms. I don't think this structure is documented in QuickTime docs. So, the way I parse it (not necessarily correctly) is like this:

Skip from the head:
4 bytes atom size
4 bytes 'ccdp'
5 bytes 96 69 52 4f 67
2 bytes rolling index
2 bytes 72 f4

Skip from the tail:
3 bytes rolling index
10 bytes 73 91 e1 00 00 00 c1 3f ff 74

The remaining payload appear to be a bunch of cc_data_pkt (3 bytes each) per CEA-708 spec.

If you enable subtitles/captions in QuickTime player you'll be able to see the text (it's not exciting).

[Pavel Koshevoy]

Replying to pkoshevoy:

Now that you've attached a cut down version of the source file may I remove the original source file? I assume you don't need it anymore...

I've removed the original source file. Let me know if you decide you need it again.

[Michael Niedermayer]

robert krüger, informed me that mans agreed, iam waiting for robert to post a patch reverting the revert

[Pavel Koshevoy]

Seems to be fixed now

[Carl Eugen Hoyos]

Fixed by Robert Krüger / Måns in 975110a8