TSAN reports heap-user-after-free in ff_h2645_extract_rbsp

[jsierant-at-vewd]

The problem has been detected in FFmpeg for the chromium project and is reproducable with following code:

std::vector<std::uint8_t> bitstream = {
    0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0xE0, 0x64, 0x9D, 0x78, 0x24, 0x72,
    0x72, 0x19, 0x9E, 0x40, 0xF5, 0x24, 0x0F, 0x81, 0xE4, 0x1B, 0xA8, 0x87,
    0xB3, 0xCF, 0x84, 0x10, 0x60, 0xFE, 0x1C, 0x5F, 0x80, 0x00, 0x08, 0xE8,
    0x5A, 0x00, 0x02, 0x7E, 0x61, 0x80, 0x20, 0x60, 0x67, 0x00, 0xCD, 0x80,
    0x5D, 0xC5, 0x59, 0x83, 0x44, 0x75, 0x51, 0xC1, 0xEB, 0x24, 0xFF, 0x64,
    0x2F, 0x0D, 0xF2, 0xC5, 0x60, 0x15, 0x21, 0x19};

static const int kIsNalff = 0;
static const int kNalLengthSize = 2;
static const int kSmallPadding = 1;
static const int kUseRef = 0;

::H2645Packet packet{};
::AVCodecContext context{};
::ff_h2645_packet_split(&packet, bitstream.data(), bitstream.size(), &context,
                        kIsNalff, kNalLengthSize, AV_CODEC_ID_HEVC,
                        kSmallPadding, kUseRef);

TSAN report:

WARNING: ThreadSanitizer: heap-use-after-free (pid=422689)
  Read of size 1 at 0x7b140000a958 by main thread:
    #0 ff_h2645_extract_rbsp third_party/ffmpeg/libavcodec/h2645_parse.c:57:17 (libffmpeg.so+0x2d12f7)
    #1 ff_h2645_packet_split third_party/ffmpeg/libavcodec/h2645_parse.c:485:20 (libffmpeg.so+0x2d21af)
    #2 media::FFmpegHevcTest_NalSplit_Test::TestBody() media/ffmpeg/ffmpeg_common_unittest.cc:396:3 (media_unittests+0x14a0039)
    #3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/src/googletest/src/gtest.cc (media_unittests+0x15225ed)
    #4 testing::Test::Run() third_party/googletest/src/googletest/src/gtest.cc:2689:5 (media_unittests+0x1522548)
    #5 testing::TestInfo::Run() third_party/googletest/src/googletest/src/gtest.cc:2866:11 (media_unittests+0x1522f9c)
    #6 testing::TestSuite::Run() third_party/googletest/src/googletest/src/gtest.cc:3020:28 (media_unittests+0x1523ac6)
    #7 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5730:44 (media_unittests+0x152bf1f)
    #8 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/googletest/src/gtest.cc (media_unittests+0x152bb79)
    #9 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:5313:10 (media_unittests+0x152ba64)
    #10 RUN_ALL_TESTS() third_party/googletest/src/googletest/include/gtest/gtest.h:2485:46 (media_unittests+0x1fdcb1f)
    #11 base::TestSuite::Run() base/test/test_suite.cc:480:16 (media_unittests+0x1fdc48b)
    #12 Invoke<int (base::TestSuite::*)(), TestSuiteNoAtExit *> base/bind_internal.h:498:12 (media_unittests+0x1483c14)
    #13 int base::internal::InvokeHelper<false, int>::MakeItSo<int (base::TestSuite::*)(), TestSuiteNoAtExit*>(int (base::TestSuite::*&&)(), TestSuiteNoAtExit*&&) base/bind_internal.h:637:12 (media_unittests+0x1483c14)
    #14 int base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, int ()>::RunImpl<int (base::TestSuite::*)(), std::tuple<base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, 0ul>(int (base::TestSuite::*&&)(), std::tuple<base::internal::UnretainedWrapper<TestSuiteNoAtExit> >&&, std::integer_sequence<unsigned long, 0ul>) base/bind_internal.h:710:12 (media_unittests+0x1483b9d)
    #15 base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, int ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:679:12 (media_unittests+0x1483b58)
    #16 base::OnceCallback<int ()>::Run() && base/callback.h:100:12 (media_unittests+0x1b96a63)
    #17 base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned long, int, unsigned long, bool, base::OnceCallback<void ()>) base/test/launcher/unit_test_launcher.cc:179:38 (media_unittests+0x200183d)
    #18 base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>, unsigned long) base/test/launcher/unit_test_launcher.cc:249:10 (media_unittests+0x20016d3)
    #19 main media/test/run_all_unittests.cc:55:10 (media_unittests+0x14839a6)

  Previous write of size 8 at 0x7b140000a958 by main thread:
    #0 operator delete(void*) /home/jenkins/shared_ws/opera_work_0/chromium/src/third_party/llvm/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 (media_unittests+0xe4edde)
    #1 ~basic_string tc_cache_workdir/x86_64-gcc5-4.22-2020-10-28-c3cf13530b5fd448252afe7097a212ec/lib/gcc/x86_64-linux-gnu/5.5.0/../../../../include/c++/5.5.0/bits/basic_string.h:559:9 (media_unittests+0x151e917)
    #2 testing::internal::UnitTestOptions::FilterMatchesTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) third_party/googletest/src/googletest/src/gtest.cc:727:1 (media_unittests+0x151e917)
    #3 testing::internal::UnitTestImpl::FilterTests(testing::internal::UnitTestImpl::ReactionToSharding) third_party/googletest/src/googletest/src/gtest.cc:5918:35 (media_unittests+0x152d9d6)
    #4 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5658:33 (media_unittests+0x152bc23)
    #5 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/googletest/src/gtest.cc (media_unittests+0x152bb79)
    #6 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:5313:10 (media_unittests+0x152ba64)
    #7 RUN_ALL_TESTS() third_party/googletest/src/googletest/include/gtest/gtest.h:2485:46 (media_unittests+0x1fdcb1f)
    #8 base::TestSuite::Run() base/test/test_suite.cc:480:16 (media_unittests+0x1fdc48b)
    #9 Invoke<int (base::TestSuite::*)(), TestSuiteNoAtExit *> base/bind_internal.h:498:12 (media_unittests+0x1483c14)
    #10 int base::internal::InvokeHelper<false, int>::MakeItSo<int (base::TestSuite::*)(), TestSuiteNoAtExit*>(int (base::TestSuite::*&&)(), TestSuiteNoAtExit*&&) base/bind_internal.h:637:12 (media_unittests+0x1483c14)
    #11 int base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, int ()>::RunImpl<int (base::TestSuite::*)(), std::tuple<base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, 0ul>(int (base::TestSuite::*&&)(), std::tuple<base::internal::UnretainedWrapper<TestSuiteNoAtExit> >&&, std::integer_sequence<unsigned long, 0ul>) base/bind_internal.h:710:12 (media_unittests+0x1483b9d)
    #12 base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, int ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:679:12 (media_unittests+0x1483b58)
    #13 base::OnceCallback<int ()>::Run() && base/callback.h:100:12 (media_unittests+0x1b96a63)
    #14 base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned long, int, unsigned long, bool, base::OnceCallback<void ()>) base/test/launcher/unit_test_launcher.cc:179:38 (media_unittests+0x200183d)
    #15 base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>, unsigned long) base/test/launcher/unit_test_launcher.cc:249:10 (media_unittests+0x20016d3)
    #16 main media/test/run_all_unittests.cc:55:10 (media_unittests+0x14839a6)

  Location is heap block of size 68 at 0x7b140000a910 allocated by main thread:
    #0 operator new(unsigned long) /home/jenkins/shared_ws/opera_work_0/chromium/src/third_party/llvm/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (media_unittests+0xe4e726)
    #1 allocate tc_cache_workdir/x86_64-gcc5-4.22-2020-10-28-c3cf13530b5fd448252afe7097a212ec/lib/gcc/x86_64-linux-gnu/5.5.0/../../../../include/c++/5.5.0/ext/new_allocator.h:104:27 (media_unittests+0x10234a2)
    #2 allocate tc_cache_workdir/x86_64-gcc5-4.22-2020-10-28-c3cf13530b5fd448252afe7097a212ec/lib/gcc/x86_64-linux-gnu/5.5.0/../../../../include/c++/5.5.0/bits/alloc_traits.h:491:20 (media_unittests+0x10234a2)
    #3 std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) tc_cache_workdir/x86_64-gcc5-4.22-2020-10-28-c3cf13530b5fd448252afe7097a212ec/lib/gcc/x86_64-linux-gnu/5.5.0/../../../../include/c++/5.5.0/bits/stl_vector.h:170:20 (media_unittests+0x10234a2)
    #4 void std::vector<unsigned char, std::allocator<unsigned char> >::_M_range_initialize<unsigned char const*>(unsigned char const*, unsigned char const*, std::forward_iterator_tag) tc_cache_workdir/x86_64-gcc5-4.22-2020-10-28-c3cf13530b5fd448252afe7097a212ec/lib/gcc/x86_64-linux-gnu/5.5.0/../../../../include/c++/5.5.0/bits/stl_vector.h:1287:35 (media_unittests+0x10c2e8b)
    #5 vector tc_cache_workdir/x86_64-gcc5-4.22-2020-10-28-c3cf13530b5fd448252afe7097a212ec/lib/gcc/x86_64-linux-gnu/5.5.0/../../../../include/c++/5.5.0/bits/stl_vector.h:377:2 (media_unittests+0x149ffeb)
    #6 media::FFmpegHevcTest_NalSplit_Test::TestBody() media/ffmpeg/ffmpeg_common_unittest.cc:381:41 (media_unittests+0x149ffeb)
    #7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/src/googletest/src/gtest.cc (media_unittests+0x15225ed)
    #8 testing::Test::Run() third_party/googletest/src/googletest/src/gtest.cc:2689:5 (media_unittests+0x1522548)
    #9 testing::TestInfo::Run() third_party/googletest/src/googletest/src/gtest.cc:2866:11 (media_unittests+0x1522f9c)
    #10 testing::TestSuite::Run() third_party/googletest/src/googletest/src/gtest.cc:3020:28 (media_unittests+0x1523ac6)
    #11 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/googletest/src/gtest.cc:5730:44 (media_unittests+0x152bf1f)
    #12 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/googletest/src/gtest.cc (media_unittests+0x152bb79)
    #13 testing::UnitTest::Run() third_party/googletest/src/googletest/src/gtest.cc:5313:10 (media_unittests+0x152ba64)
    #14 RUN_ALL_TESTS() third_party/googletest/src/googletest/include/gtest/gtest.h:2485:46 (media_unittests+0x1fdcb1f)
    #15 base::TestSuite::Run() base/test/test_suite.cc:480:16 (media_unittests+0x1fdc48b)
    #16 Invoke<int (base::TestSuite::*)(), TestSuiteNoAtExit *> base/bind_internal.h:498:12 (media_unittests+0x1483c14)
    #17 int base::internal::InvokeHelper<false, int>::MakeItSo<int (base::TestSuite::*)(), TestSuiteNoAtExit*>(int (base::TestSuite::*&&)(), TestSuiteNoAtExit*&&) base/bind_internal.h:637:12 (media_unittests+0x1483c14)
    #18 int base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, int ()>::RunImpl<int (base::TestSuite::*)(), std::tuple<base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, 0ul>(int (base::TestSuite::*&&)(), std::tuple<base::internal::UnretainedWrapper<TestSuiteNoAtExit> >&&, std::integer_sequence<unsigned long, 0ul>) base/bind_internal.h:710:12 (media_unittests+0x1483b9d)
    #19 base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<TestSuiteNoAtExit> >, int ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:679:12 (media_unittests+0x1483b58)
    #20 base::OnceCallback<int ()>::Run() && base/callback.h:100:12 (media_unittests+0x1b96a63)
    #21 base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned long, int, unsigned long, bool, base::OnceCallback<void ()>) base/test/launcher/unit_test_launcher.cc:179:38 (media_unittests+0x200183d)
    #22 base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>, unsigned long) base/test/launcher/unit_test_launcher.cc:249:10 (media_unittests+0x20016d3)
    #23 main media/test/run_all_unittests.cc:55:10 (media_unittests+0x14839a6)

SUMMARY: ThreadSanitizer: heap-use-after-free third_party/ffmpeg/libavcodec/h2645_parse.c:57:17 in ff_h2645_extract_rbsp

The problem originates in ff_h2645_extract_rbsp (line 57 has been marked):

#if HAVE_FAST_64BIT
    for (i = 0; i + 1 < length; i += 9) {
        if (!((~AV_RN64(src + i) & // line 57
               (AV_RN64(src + i) - 0x0100010001000101ULL)) &
              0x8000800080008080ULL))
            continue;
        FIND_FIRST_ZERO;
        STARTCODE_TEST;
        i -= 7;
    }
#else

It seems that when the optimizations are enabled (HAVE_FAST_UNALIGNED and HAVE_FAST_64BIT are set) the size of the provided data src is exceeded (despite that length = 68, byte 72 is read (calculated as 0x7b140000a958 - 0x7b140000a910)).
The same problem will probably be visible when HAVE_FAST_64BIT is not set as the algorithm is the same for 32 bits machines.

Accessing memory outside of allocated space may lead to segmentation fault and an abnormal termination of the process.

The FFmpeg version that has been used is:
Upstream Git: ​git://source.ffmpeg.org/ffmpeg.git
Last Upstream Merge: 71ec3e4583def61fbb32a4815376773ef7c80dee, Aug 27 2020

The code seems to be same as in latest FFmpeg release (4.3.1) so the problem should also be visible there.

[mkver]

Your data is missing its padding - AV_INPUT_BUFFER_PADDING_SIZE of padding is required (to allow to use algorithms that overread slightly).