undefined behavior at libavcodec/jpeg2000dwt.c:84:18

[Johannes Kauffmann]

UBSan complains about undefined behaviour when running make check

How to reproduce:

mkdir build && cd build
../configure --toolchain=gcc-usan --enable-debug=2 --disable-stripping --disable-doc
UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 make -j8 check

... which prints:

Test j2k-dwt failed. Look at tests/data/fate/j2k-dwt.err for details.
make: *** [/home/user/ffmpeg/tests/Makefile:307: fate-j2k-dwt] Error 1
make: *** Waiting for unfinished jobs....

j2k-dwt.err contains:

src/libavcodec/jpeg2000dwt.c:84:18: runtime error: left shift of negative value -66
    #0 0x55863fe454bd in ff_dwt_encode.cold (/home/user/ffmpeg/build/libavcodec/tests/jpeg2000dwt+0x144bd)
    #1 0x55863fe5a046 in test_dwt src/libavcodec/tests/jpeg2000dwt.c:44
    #2 0x55863fe465f0 in main src/libavcodec/tests/jpeg2000dwt.c:134
    #3 0x7fd8ddf2fd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0x7fd8ddf2fe3f in __libc_start_main_impl ../csu/libc-start.c:392
    #5 0x55863fe48204 in _start (/home/user/ffmpeg/build/libavcodec/tests/jpeg2000dwt+0x17204)

Tested with:
ffmpeg git dd2ea014ef273157fe9a0e928e77841fbbee9b2f
GCC 11.2.0
Ubuntu 22.04.

[Carl Eugen Hoyos]

$ make libavcodec/tests/jpeg2000dwt && libavcodec/tests/jpeg2000dwt
5/3i, decomp:15 border 151 170 140 183 milli-err2:        0
9/7i, decomp:15 border 151 170 140 183 milli-err2:      544
9/7f, decomp:15 border 151 170 140 183 err2:               0.000
5/3i, decomp:21 border 173 201  81 189 milli-err2:        0
9/7i, decomp:21 border 173 201  81 189 milli-err2:      592
9/7f, decomp:21 border 173 201  81 189 err2:               0.000
src/libavcodec/jpeg2000dwt.c:84:18: runtime error: left shift of negative value -66
5/3i, decomp:22 border 213 227  76 245 milli-err2:        0
9/7i, decomp:22 border 213 227  76 245 milli-err2:      533
9/7f, decomp:22 border 213 227  76 245 err2:               0.000
5/3i, decomp:13 border 134 157 184 203 milli-err2:        0
9/7i, decomp:13 border 134 157 184 203 milli-err2:      535
9/7f, decomp:13 border 134 157 184 203 err2:               0.000
5/3i, decomp: 1 border 204 237   6 106 milli-err2:        0
9/7i, decomp: 1 border 204 237   6 106 milli-err2:      219
9/7f, decomp: 1 border 204 237   6 106 err2:               0.000
5/3i, decomp:28 border  76 211  13 210 milli-err2:        0
9/7i, decomp:28 border  76 211  13 210 milli-err2:      791
9/7f, decomp:28 border  76 211  13 210 err2:               0.000
5/3i, decomp:21 border  76  99  43 123 milli-err2:        0
9/7i, decomp:21 border  76  99  43 123 milli-err2:      686
9/7f, decomp:21 border  76  99  43 123 err2:               0.000
5/3i, decomp:15 border 192 243 174 204 milli-err2:        0
9/7i, decomp:15 border 192 243 174 204 milli-err2:      476
9/7f, decomp:15 border 192 243 174 204 err2:               0.000
5/3i, decomp:21 border  17  68  93 204 milli-err2:        0
9/7i, decomp:21 border  17  68  93 204 milli-err2:      633
9/7f, decomp:21 border  17  68  93 204 err2:               0.000
5/3i, decomp:11 border 142 168  82 174 milli-err2:        0
9/7i, decomp:11 border 142 168  82 174 milli-err2:      696
9/7f, decomp:11 border 142 168  82 174 err2:               0.000
5/3i, decomp:23 border 142 209 171 235 milli-err2:        0
9/7i, decomp:23 border 142 209 171 235 milli-err2:      626
9/7f, decomp:23 border 142 209 171 235 err2:               0.000
5/3i, decomp:30 border  37 185  79 245 milli-err2:        0
9/7i, decomp:30 border  37 185  79 245 milli-err2:      953
9/7f, decomp:30 border  37 185  79 245 err2:               0.000
5/3i, decomp: 5 border 129 236  30 243 milli-err2:        0
9/7i, decomp: 5 border 129 236  30 243 milli-err2:      620
9/7f, decomp: 5 border 129 236  30 243 err2:               0.000
5/3i, decomp:10 border   5 160 146 247 milli-err2:        0
9/7i, decomp:10 border   5 160 146 247 milli-err2:      797
9/7f, decomp:10 border   5 160 146 247 err2:               0.000
5/3i, decomp: 5 border 104 162   6  47 milli-err2:        0
9/7i, decomp: 5 border 104 162   6  47 milli-err2:      603
9/7f, decomp: 5 border 104 162   6  47 err2:               0.000
5/3i, decomp:24 border  78 250 102 218 milli-err2:        0
9/7i, decomp:24 border  78 250 102 218 milli-err2:      836
9/7f, decomp:24 border  78 250 102 218 err2:               0.000
5/3i, decomp:28 border  86  98  56  79 milli-err2:        0
9/7i, decomp:28 border  86  98  56  79 milli-err2:      597
9/7f, decomp:28 border  86  98  56  79 err2:               0.000
5/3i, decomp: 6 border  95 238 197 214 milli-err2:        0
9/7i, decomp: 6 border  95 238 197 214 milli-err2:      478
9/7f, decomp: 6 border  95 238 197 214 err2:               0.000
5/3i, decomp:17 border  77 169  93 165 milli-err2:        0
9/7i, decomp:17 border  77 169  93 165 milli-err2:      616
9/7f, decomp:17 border  77 169  93 165 err2:               0.000
5/3i, decomp:22 border 178 187   7 119 milli-err2:        0
9/7i, decomp:22 border 178 187   7 119 milli-err2:      392
9/7f, decomp:22 border 178 187   7 119 err2:               0.000

[Balling]

I will just live it here: ​https://groups.google.com/a/isocpp.org/g/std-proposals/c/UorPkl3Lx1U

This whole undefined in standard, but gcc does not agree for 2's complement is kinda crazy.

Do not we require 2's complement... But on the other hand those are always fixed by mkver:

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/GV1P250MB07378F41C030CC0726A137AE8F529@GV1P250MB0737.EURP250.PROD.OUTLOOK.COM/

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/GV1P250MB07377FF4AAC085EC65213D3B8F559@GV1P250MB0737.EURP250.PROD.OUTLOOK.COM/

[Balling]

Fixed in fff010591b35874b4c7a7e9fe00d7541f0b7c994.