Crash with recent libx265

[haruka]

Summary of the bug:

On OS X, When linked against recent libx265 (starts from changeset 10903:13b2356290d6), ffmpeg would crash on launch.

Not tested on other platforms.

How to reproduce:

Any attempt to launch ffmpeg would crash it.

Backtrace:

$ lldb ffmpeg_g
(lldb) target create "ffmpeg_g"
rCurrent executable set to 'ffmpeg_g' (x86_64).
(lldb) r
Process 10720 launched: '/Users/MrX/src/ffmpeg/ffmpeg_g' (x86_64)
Process 10720 stopped
* thread #1: tid = 0x42e75, 0x00007fff5fc048d9 dyld`dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*)) + 479, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x7fff5f3fffe8)
    frame #0: 0x00007fff5fc048d9 dyld`dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*)) + 479
dyld`dyld::notifyBatchPartial:
->  0x7fff5fc048d9 <+479>: callq  *0x68(%rax)
    0x7fff5fc048dc <+482>: leaq   (%r12,%r12,2), %rbx
    0x7fff5fc048e0 <+486>: movq   -0x40(%rbp), %r15
    0x7fff5fc048e4 <+490>: movq   %rax, (%r15,%rbx,8)
(lldb) bt
* thread #1: tid = 0x42e75, 0x00007fff5fc048d9 dyld`dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*)) + 479, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x7fff5f3fffe8)
  * frame #0: 0x00007fff5fc048d9 dyld`dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*)) + 479
    frame #1: 0x00007fff5fc0f972 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 84
    frame #2: 0x00007fff5fc05063 dyld`dyld::runInitializers(ImageLoader*) + 89
    frame #3: 0x00007fff5fc0c1d1 dyld`dlopen + 578
    frame #4: 0x00000001065ee857 libdyld.dylib`dlopen + 59
    frame #5: 0x000000010319e55d libx265.71.dylib`x265_api_get_71 + 125
    frame #6: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #7: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #8: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #9: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #10: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
...
    frame #174466: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #174467: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #174468: 0x000000010319e582 libx265.71.dylib`x265_api_get_71 + 162
    frame #174469: 0x00000001006197be ffmpeg_g`libx265_encode_init_csp + 14
    frame #174470: 0x0000000100334380 ffmpeg_g`avcodec_register_all + 101104
    frame #174471: 0x0000000100017cec ffmpeg_g`main + 796
    frame #174472: 0x00000001067d25c9 libdyld.dylib`start + 1

[Carl Eugen Hoyos]

Why do you think this is a bug that can be fixed within FFmpeg?
In which situation would dlopen() crash?

Did you already test with a small test program that calls x265_api_get()?

[haruka]

I think you are right that the problem isn't in the ffmpeg part.

After checking the specified changeset, x265 doc and ffmpeg source, it seems that the x265_api_get() now has adapted to multilib build, and it appears to me that the problem is indeed in that function which just recursively calls itself if the library is not built as multilib.

Without more insights, I thought that it's ffmpeg which is not updating to the new API format, but apparently I was wrong. x265 has just fixed that problem in the tip. Maybe I just chose a wrong time to follow the tip...

This issue can be closed now, sorry for the confusion.

[Carl Eugen Hoyos]

Thank you for the clarification!