DNxHR doesn't work with LB in Shutter Encoder

[Soto Soto]

Summary of the bug: I've just been forwarded by the creator of Shutter Encoder who uses FFmpeg as the backend of their software. I wanted to create a small 480x270 file at low bitrate and the render stops after a few minutes. The last line the console says is "Assertion s->buf_ptr < s->buf_end failed at src/libavcodec/put_bits.h:150".

How to reproduce:

1. Used a 1920x1080p, 60fps, h.264 10gb file.
2. Dropped it in Shutter Encoder, set to DNxHR > LB ("Low Bitrate" doesn't state the number of the bitrate itself). Change scale to 480x270p
3. Render
4. Error shown in the last line of the console (right click in progress bar and click "Console").

[mkver]

1. The component that you selected is ffmpeg. Here, this means the ffmpeg commandline tool (as opposed to Shutter Encoder using the libavcodec library on its own) and not the FFmpeg project. Are you sure that Shutter Encoder uses the ffmpeg tool instead of the libavcodec library?
2. If so, then you can somehow forward the actually used command line to us? (Is it more than just "-c:v dnxhd -profile:v dnxhr_lb"?)
3. Furthermore, we will also need the input file (or any file) to actually reproduce the problem. Given that DNXHD is an intra-only codec, it should be enough to send the part of the file that actually contains the packet that is decoded to the frame whose encoding triggers the assert.

[Soto Soto]

Replying to mkver:

1. The component that you selected is ffmpeg. Here, this means the ffmpeg commandline tool (as opposed to Shutter Encoder using the libavcodec library on its own) and not the FFmpeg project. Are you sure that Shutter Encoder uses the ffmpeg tool instead of the libavcodec library?

According to Paul Pacifico, the developer of Shutter Encoder, FFmpeg is the backend of their software. I contacted Paul through reddit and redirected me here. This is the link to my conversation with Paul: ​https://www.reddit.com/r/shutterencoder/comments/w6m85d/comment/ihh3obk

2. If so, then you can somehow forward the actually used command line to us? (Is it more than just "-c:v dnxhd -profile:v dnxhr_lb"?)

According to Shutter Encoder's console window, I saved a text from the last time I did the render and it seems that the command is (redacted file outputs and changed file names for this example):

-threads 0 -hwaccel none -i "[REDACTED]\clip.mp4" -c:v dnxhd -profile:v dnxhr_lb -pix_fmt yuv422p -filter_complex "[0:v]scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)*0.5:(oh-ih)*0.5" -map "[out]" -c:a pcm_s161e -ar 48000 -map a? -sws_flags bicubic -metadata creation_time="2022-07-23T22:16:59.922821300Z" -y "[REDACTED]\clip_Proxy.mov"

There's more to this but it may be too much to put in here so I'll link it here​https://i.imgur.com/XwdYzXW.png

3. Furthermore, we will also need the input file (or any file) to actually reproduce the problem. Given that DNXHD is an intra-only codec, it should be enough to send the part of the file that actually contains the packet that is decoded to the frame whose encoding triggers the assert.

I can provide another file I just recorded, tested with Shutter Encoder and still drops the error. NOTE: this link will expire in two days. ​https://we.tl/t-fxYATPx84B

[mkver]

Regression since 044c09c0a0b5e2d91d75619c1327e232a4914030. Before that commit, encoding simply failed with "picture could not fit ratecontrol constraints, increase qmax". Since that commit, one receives three "Internal error, put_bits buffer too small" error messages (these lead to asserts in case --assert-level is set to two) and runs twice into the assert "s->buf_ptr < s->buf_end" in flush_put_bits().
I will upload snippets from your provided file that exhibit the issue.

[Soto Soto]

Replying to mkver:
It is difficult for me to understand your response, I don't know anything about the language and code used here, since I use other software to avoid the coding part, and besides that my main language is not english, so some words used in this context seem complex to understand. Is there something you suggest me to do on my end (like contacting the Shutter Encoder dev) or will there be any update in the future?

[Soto Soto]

Replying to mkver:
It is difficult for me to understand your response, I don't know anything about the language and code used here, since I use other software to avoid the coding part, and besides that my main language is not english, so some words used in this context seem complex to understand. Is there something you suggest me to do on my end (like contacting the Shutter Encoder dev) or will there be any update in the future?

[mkver]

You (or the Shutter Encoder dev) are not expected to do anything any more (thanks for your report);I just created a file that allows to reproduce the issue in a very direct manner, without the need for a huge input file.

[pselker]

I'm not a developer or a contributor here, but I was getting this same error in a different context and learned something small I figured I'd share here just in case it'd be useful. I used the sample file and ffmpeg command and was able to replicate the error -- but scaling to any other resolution (even lower res like 180p) eliminated the error immediately. (i.e. -s 640x360, 1280x720, etc.) Perhaps already known, but figured I'd flag.

UPDATE: We are seeing this same error going to DNXHD 90, and we now think it seems to happen (at least for us) in frames that have a prominent grid pattern, like an LED wall. Rescaling the footage *can* eliminate the problem but doesn't always, and it can also just shift the problem to a nearby frame.

We have had preliminary success changing the DCT algorithm to "int" - it seems that the floating point ones are all having this issue much more frequently. We are seeing many fewer of these errors - but we still see some on gridlike frames.

[slenk]

I just experienced the same error with some of my own footage:

↳ ffprobe ../1_RAWS/2025-07-25\ 20-03-49.mkv 
ffprobe version 8.0 Copyright (c) 2007-2025 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
  configuration: --disable-decoder=amrnb --disable-gnutls --disable-liblensfun --disable-libopencv --disable-podpages --disable-sndio --disable-stripping --enable-avfilter --enable-chromaprint --enable-frei0r --enable-gcrypt --enable-gpl --enable-ladspa --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librabbitmq --enable-librist --enable-librsvg --enable-librubberband --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-nonfree --enable-openal --enable-opencl --enable-opengl --enable-openssl --enable-pthreads --enable-shared --enable-version3 --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-vaapi --enable-libvpl --enable-libxavs2 --enable-libdavs2 --enable-libvmaf --enable-librav1e --enable-libvvenc --enable-libilbc --enable-libklvanc --enable-libfdk-aac --enable-libkvazaar --enable-omx --cc=x86_64-linux-gnu-gcc --cxx=x86_64-linux-gnu-g++ --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      60.  8.100 / 60.  8.100
  libavcodec     62. 11.100 / 62. 11.100
  libavformat    62.  3.100 / 62.  3.100
  libavdevice    62.  1.100 / 62.  1.100
  libavfilter    11.  4.100 / 11.  4.100
  libswscale      9.  1.100 /  9.  1.100
  libswresample   6.  1.100 /  6.  1.100
Input #0, matroska,webm, from '../1_RAWS/2025-07-25 20-03-49.mkv':
  Metadata:
    ENCODER         : Lavf61.1.100
  Duration: 02:45:37.63, start: 0.000000, bitrate: 4007 kb/s
  Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1k tbn
    Metadata:
      DURATION        : 02:45:37.633000000
  Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp
    Metadata:
      title           : simple_aac_recording0
      DURATION        : 02:45:37.537000000

ffmpeg -hide_banner -nostdin \  
    -i "$input" \               
    -c:a copy \                                                                                                                                                                                                                                                      
    -c:v dnxhd \                                                                                                                                           
    -profile:v dnxhr_lb \                                                                                                                                  
    -vf scale=1280x720 \                                                                                                                                   
    "$output"                                                                                                                                              

...
Assertion s->buf_ptr < s->buf_end failed at libavcodec/put_bits.h:160
...

• the error is on line 160 instead of 150
• the input file is made with OBS
• I get the same error with -vf scale=-1:480 (fails at an earlier time)
• If I don't scale at all, it fails at a later time
• transcoding with -profile:v dnxhr_sq does not trigger the error
• transcoding with -pix_fmt yuv422p -b:v 60M does not trigger the error