DNxHD error during encoding: too many threads

[ralambda]

Summary:
When encoding DNxHD mxf file the encoder crashes with the error "too many threads". I've reproduced this error with various input files of various resolutions and codecs including h264 (yuv420, high, 5.1), AVC Intra, and MPEG-2.

Error:

[dnxhd @ 0x3e92c00] too many threads
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

How to reproduce:

/usr/local/bin/ffmpeg -v 9 -loglevel 99 -y -r 23.976 -i input_file -c:v dnxhd -b:v 36M -pix_fmt:v yuv422p -s:v 1920x1080 -an output_file.mxf

ffmpeg version

ffmpeg version N-86980-g62b7553 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11)
configuration: --prefix=/home/alchemist/ffmpeg_build --extra-cflags=-I/home/alchemist/ffmpeg_build/include --extra-ldflags='-L/home/alchemist/ffmpeg_build/lib -ldl' --bindir=/home/alchemist/bin --pkg-config-flags=--static --enable-gpl --enable-libfdk_aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree
libavutil      55. 73.100 / 55. 73.100
libavcodec     57.102.100 / 57.102.100
libavformat    57. 76.100 / 57. 76.100
libavdevice    57.  7.100 / 57.  7.100
libavfilter     6. 98.100 /  6. 98.100
libswscale      4.  7.102 /  4.  7.102
libswresample   2.  8.100 /  2.  8.100
libpostproc    54.  6.100 / 54.  6.100

Built on: CentOS 7 from source (Git master)

The command above worked successfully on the last version I had built. Unfortunately I don't have the build info for that version. I compiled from the git today (August 16, 2017) and now getting the error.

The issue might be related to this update: ​https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/9835ee60da31184dd6f6700328e75c78fa485efe

The terminal output is over 26,000 lines when using -v 9 -loglevel 99.
Here's the output once encoding starts:

[dnxhd @ 0x5ac2e80] cid 1253
[dnxhd @ 0x6018820] cid 1253
[dnxhd @ 0x61489c0] cid 1253
[dnxhd @ 0x6278d60] cid 1253
[dnxhd @ 0x63a8f00] cid 1253
[dnxhd @ 0x64d9140] cid 1253
[dnxhd @ 0x66093c0] cid 1253
[dnxhd @ 0x6739760] cid 1253
[dnxhd @ 0x6869900] cid 1253
[dnxhd @ 0x6999ca0] cid 1253
[dnxhd @ 0x6ac9e40] cid 1253
[dnxhd @ 0x6bfa1e0] cid 1253
[dnxhd @ 0x6d2a380] cid 1253
[dnxhd @ 0x6e5a720] cid 1253
[dnxhd @ 0x6f8a8c0] cid 1253
[dnxhd @ 0x70bac60] cid 1253
[dnxhd @ 0x71eae00] cid 1253
[dnxhd @ 0x731b1a0] cid 1253
[dnxhd @ 0x744b340] cid 1253
[dnxhd @ 0x757b6e0] cid 1253
[dnxhd @ 0x76ab880] cid 1253
[dnxhd @ 0x77dbc20] cid 1253
[dnxhd @ 0x790bdc0] cid 1253
[dnxhd @ 0x7a3c160] cid 1253
[dnxhd @ 0x7b6c300] cid 1253
[dnxhd @ 0x7c9c6a0] cid 1253
[dnxhd @ 0x7dcc840] cid 1253
[dnxhd @ 0x7efcbe0] cid 1253
[dnxhd @ 0x802cd80] cid 1253
[dnxhd @ 0x815d120] cid 1253
[dnxhd @ 0x828d2c0] cid 1253
[dnxhd @ 0x83bd660] cid 1253
[dnxhd @ 0x84ed800] cid 1253
[dnxhd @ 0x861dba0] cid 1253
[dnxhd @ 0x874dd40] cid 1253
[dnxhd @ 0x887e0e0] cid 1253
[dnxhd @ 0x89ae280] cid 1253
[dnxhd @ 0x8ade620] cid 1253
[dnxhd @ 0x8c0e7c0] cid 1253
[dnxhd @ 0x8d3eb60] cid 1253
[dnxhd @ 0x3e92c00] cid 1253
[dnxhd @ 0x3e92c00] too many threads
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[AVIOContext @ 0x3e96d80] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 0x3e710c0] Statistics: 2790563 bytes read, 1 seeks
Conversion failed!

[ralambda]

This was resolved by compiling version 3.3.3 Hilbert

built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11)
configuration: --prefix=/home/user/ffmpeg_build --extra-cflags=-I/home/user/ffmpeg_build/include --extra-ldflags='-L/home/user/ffmpeg_build/lib -ldl' --bindir=/home/user/bin --pkg-config-flags=--static --enable-gpl --enable-nonfree --enable-libfdk_aac --enable-libfreetype --enable-libx264 --enable-libx265
libavutil      55. 58.100 / 55. 58.100
libavcodec     57. 89.100 / 57. 89.100
libavformat    57. 71.100 / 57. 71.100
libavdevice    57.  6.100 / 57.  6.100
libavfilter     6. 82.100 /  6. 82.100
libswscale      4.  6.100 /  4.  6.100
libswresample   2.  7.100 /  2.  7.100
libpostproc    54.  5.100 / 54.  5.100

[Elon Musk]

Doesn't make sense, how many cores/CPUS do you have?
Also does this happens with DNxHD only?

[ralambda]

The system has 20 physical cores (40 logical). It seems that each core is printed in the encoding output above.

This only happens with DNxHD encoding.

Important note: the DNxHD encoder is working on 6 and 8 core systems.

[Elon Musk]

What happens if you explicitly set threads with '-threads' to some number less than 64?

[ralambda]

It works if -threads less than or equal to 32 on the 20 core system It fails if -threads is greater than 32.

It's the same error when it fails when the -threads option is added: [dnxhd @ 0x319ec80] too many threads

Works:

/ffmpeg -v 9 -loglevel 99 -y -r 23.976 -i input_file.mp4 -threads 32 -c:v dnxhd -b:v 36M -pix_fmt:v yuv422p -s:v 1920x1080 -an output_file.mxf

Fails:

/ffmpeg -v 9 -loglevel 99 -y -r 23.976 -i input_file.mp4 -threads 33 -c:v dnxhd -b:v 36M -pix_fmt:v yuv422p -s:v 1920x1080 -an output_file.mxf

[Elon Musk]

Fixed in 28e9ba951d1a0b0aca53b242aa64f484ca75e874.