H.261 encoding: -cmp rd or sse ruins image quality

[Maik Merten]

Summary of the bug:
How to reproduce:

ffmpeg -i /tmp/test.y4m -f h261 -vb 256k -cmp rd /tmp/test.h261

ffmpeg version N-58246-g7d0e927 Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 18 2013 17:15:01 with gcc 4.8 (Ubuntu/Linaro 4.8.1-10ubuntu8)

Test sequence used is ​http://media.xiph.org/video/derf/y4m/sign_irene_cif.y4m

Bitrate of the file is very much inflated by specifying the -cmp rd option, yet picture quality is severely affected by blockyness. Apparently the encoder is forced into aggressive quantization.

Attached picture shows one representative frame of video output.

[Carl Eugen Hoyos]

For future tickets: Please always provide your failing command line together with complete, uncut console output. If a problem is reproducible with -f lavfi -i testsrc please consider using it since it it may make reproducing the whole ticket easier.

Image quality starts to degrade after 20 seconds.

$ ffmpeg -f lavfi -i testsrc=s=352x288 -vb 256k -cmp rd -t 60 out.h261
ffmpeg version N-58249-g2974050 Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 18 2013 18:18:50 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      52. 53.100 / 52. 53.100
  libavcodec     55. 43.101 / 55. 43.101
  libavformat    55. 21.100 / 55. 21.100
  libavdevice    55.  5.100 / 55.  5.100
  libavfilter     3. 91.100 /  3. 91.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, lavfi, from 'testsrc=s=352x288':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 352x288 [SAR 1:1 DAR 11:9], 25 tbr, 25 tbn, 25 tbc
Output #0, h261, to 'out.h261':
  Metadata:
    encoder         : Lavf55.21.100
    Stream #0:0: Video: h261, yuv420p, 352x288 [SAR 1:1 DAR 11:9], q=2-31, 256 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> h261)
Press [q] to stop, [?] for help
[h261 @ 0x306e800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 16 times
[h261 @ 0x306e800] warning, clipping 2 dct coefficients to -127..127
    Last message repeated 1 times
[h261 @ 0x306e800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 4 times
[h261 @ 0x306e800] warning, clipping 2 dct coefficients to -127..127
    Last message repeated 1 times
[h261 @ 0x306e800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 5 times
[h261 @ 0x306e800] warning, clipping 2 dct coefficients to -127..127
[h261 @ 0x306e800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 12 times
[h261 @ 0x306e800] warning, clipping 2 dct coefficients to -127..127
[h261 @ 0x306e800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 76 times
[h261 @ 0x306e800] warning, clipping 1 dct coefficients to -127..127411.3kbits/s
    Last message repeated 47 times
frame= 1500 fps=606 q=31.0 Lsize=    3090kB time=00:01:00.00 bitrate= 422.0kbits/s
video:3090kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.000000%

[Michael Niedermayer]

seems the -mbd rd fix, fixed this too

[Carl Eugen Hoyos]

I still see the exact same artefacts (starting at ~20sec) with ddaf543:

$ ffmpeg -f lavfi -i testsrc=s=352x288 -vb 256k -cmp rd -t 60 out.h261
ffmpeg version N-58302-gddaf543 Copyright (c) 2000-2013 the FFmpeg developers
  built on Nov 20 2013 17:10:53 with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      52. 53.100 / 52. 53.100
  libavcodec     55. 43.101 / 55. 43.101
  libavformat    55. 21.100 / 55. 21.100
  libavdevice    55.  5.100 / 55.  5.100
  libavfilter     3. 91.100 /  3. 91.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, lavfi, from 'testsrc=s=352x288':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 352x288 [SAR 1:1 DAR 11:9], 25 tbr, 25 tbn, 25 tbc
Output #0, h261, to 'out.h261':
  Metadata:
    encoder         : Lavf55.21.100
    Stream #0:0: Video: h261, yuv420p, 352x288 [SAR 1:1 DAR 11:9], q=2-31, 256 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> h261)
Press [q] to stop, [?] for help
[h261 @ 0x3798800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 16 times
[h261 @ 0x3798800] warning, clipping 2 dct coefficients to -127..127
    Last message repeated 1 times
[h261 @ 0x3798800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 4 times
[h261 @ 0x3798800] warning, clipping 2 dct coefficients to -127..127
    Last message repeated 1 times
[h261 @ 0x3798800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 5 times
[h261 @ 0x3798800] warning, clipping 2 dct coefficients to -127..127
[h261 @ 0x3798800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 12 times
[h261 @ 0x3798800] warning, clipping 2 dct coefficients to -127..127
[h261 @ 0x3798800] warning, clipping 1 dct coefficients to -127..127
    Last message repeated 76 times
[h261 @ 0x3798800] warning, clipping 1 dct coefficients to -127..127409.5kbits/s
    Last message repeated 47 times
frame= 1500 fps=611 q=31.0 Lsize=    3090kB time=00:01:00.00 bitrate= 422.0kbits/s
video:3090kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.000000%

[Michael Niedermayer]

are you sure this is not an artifact of low bitrate h261 ?
do you see fewer artifacts with -cmp sse ?

[Carl Eugen Hoyos]

I tested the following:

$ ffmpeg -f lavfi -i testsrc=s=352x288 -vb 256k -t 60 -cmp sad outsad.h261

This encodes at 302.5kbits/s and the output file looks visually not great but acceptable (some artefacts are visible after 20 seconds, I originally thought that they were related to the codec and the bitrate.
I then tried:

$ ffmpeg -f lavfi -i testsrc=s=352x288 -vb 256k -t 60 -cmp rd outrd.h261

The output file outrd,h261 is encoded with 422.0kbits/s and looks visually significantly worse than outsad.h261 (starting from 20 seconds on, outsad.h261 also gets worse after 20 seconds, but still looks much better than outrd.h261).
-cmp sse produces an even larger file with similar artefacts as outrd.h261 but they start earlier.

If I encode only ten seconds, file size is similar for all three cmp modes (sad: 424.5kbits/s, rd: 414.8kbits/s, sse; 426.2kbits/s) and all three output files do not show obvious artefacts. If I should guess, I'd say there is a bug (that worsens h261 quality significantly after some encoding time) and some cmp modes make this bug much more visible than others (and the default sad).

The issue that -cmp 1 produces larger files with significantly more artefacts than -cmp 0 is not a regression, I can reproduce it with 1c3990db

[Michael Niedermayer]

Fixed in b80106169ab16c31e27ea464ca62b805b205d61d