using -hwaccel nvdec produces 'No decoder surfaces left' with interlaced input and 3 or more b-frames

[malakudi]

ffmpeg git and 4.1 with -hwaccel nvdec report "No decoder surfaces left" when trying to transcode an interlaced sd 50i input MBAFF type and using 3 or more b-frames for encoding. This is a regression because it works fine on 4.0.3. It also doesn't happen with -hwaccel cuvid. With progressive input also doesn't happen. With only 2 b-frames also doesn't happen.

How to reproduce:

./ffmpeg-git -hwaccel nvdec -hwaccel_output_format cuda -f mpegts -i input_sd_interlaced_50i_mbaff.ts -vcodec h264_nvenc -refs 4 -bf 3 -c:a copy -f mpegts -y /dev/null
ffmpeg version N-92510-gfa08345e88 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-runtime-cpudetect --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-debug --enable-libaom --enable-avfilter --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-libnpp --enable-cuda-sdk --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-static --disable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 46.100 /  7. 46.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, mpegts, from 'input_sd_interlaced_50i_mbaff.ts':
  Duration: 00:01:42.92, start: 1.400000, bitrate: 1302 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, smpte170m, top first), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mpegts, to '/dev/null':
  Metadata:
    encoder         : Lavf58.22.100
    Stream #0:0: Video: h264 (h264_nvenc) (Main), cuda, 720x576 [SAR 16:11 DAR 20:11], q=-1--1, 2000 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.39.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1(gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
[h264 @ 0x5557a875ab00] No decoder surfaces left
Error while decoding stream #0:0: Invalid data found when processing input
[h264 @ 0x5557a873e2c0] No decoder surfaces left
[h264_nvenc @ 0x5557a802f300] Error mapping an input resource: invalid param (8)
Video encoding failed
Conversion failed!

Same command with ffmpeg 4.0.3 succeeds

./ffmpeg-4.0.3 -hwaccel nvdec -hwaccel_output_format cuda -f mpegts -i input_sd_interlaced_50i_mbaff.ts -vcodec h264_nvenc -refs 4 -bf 3 -c:a copy -f mpegts -y /dev/null
ffmpeg version 4.0.3 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-runtime-cpudetect --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-debug --enable-libaom --enable-avfilter --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-libnpp --enable-cuda-sdk --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-static --disable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
Input #0, mpegts, from 'input_sd_interlaced_50i_mbaff.ts':
  Duration: 00:01:42.92, start: 1.400000, bitrate: 1302 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, smpte170m, top first), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mpegts, to '/dev/null':
  Metadata:
    encoder         : Lavf58.12.100
    Stream #0:0: Video: h264 (h264_nvenc) (Main), cuda, 720x576 [SAR 16:11 DAR 20:11], q=-1--1, 2000 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.18.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1(gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
frame= 2537 fps=544 q=16.0 Lsize=   29353kB time=00:01:42.84 bitrate=2338.2kbits/s speed=  22x    
video:24905kB audio:1995kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.119826%

Same command with -bf 2 succeeds

./ffmpeg-git -hwaccel nvdec -hwaccel_output_format cuda -f mpegts -i input_sd_interlaced_50i_mbaff.ts -vcodec h264_nvenc -refs 4 -bf 2 -c:a copy -f mpegts -y /dev/null
ffmpeg version N-92510-gfa08345e88 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-runtime-cpudetect --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-debug --enable-libaom --enable-avfilter --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-libnpp --enable-cuda-sdk --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-static --disable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 46.100 /  7. 46.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, mpegts, from 'input_sd_interlaced_50i_mbaff.ts':
  Duration: 00:01:42.92, start: 1.400000, bitrate: 1302 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, smpte170m, top first), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mpegts, to '/dev/null':
  Metadata:
    encoder         : Lavf58.22.100
    Stream #0:0: Video: h264 (h264_nvenc) (Main), cuda, 720x576 [SAR 16:11 DAR 20:11], q=-1--1, 2000 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.39.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1(gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
frame= 2537 fps=473 q=15.0 Lsize=   29374kB time=00:01:42.84 bitrate=2339.9kbits/s speed=19.2x    
video:24919kB audio:1995kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.144446%

Same command with -bf 3 but progressive input succeeds

./ffmpeg-git -hwaccel nvdec -hwaccel_output_format cuda -f mpegts -i input_hdready_progressive_ntsc.ts -vcodec h264_nvenc -refs 4 -bf 3 -c:a copy -f mpegts -y /dev/null
ffmpeg version N-92510-gfa08345e88 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-runtime-cpudetect --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-debug --enable-libaom --enable-avfilter --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-libnpp --enable-cuda-sdk --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-static --disable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 46.100 /  7. 46.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
[h264 @ 0x55dca7f9bfc0] mmco: unref short failure
[h264 @ 0x55dca7f9bfc0] number of reference frames (0+6) exceeds max (5; probably corrupt input), discarding one
[h264 @ 0x55dca7f9bfc0] mmco: unref short failure
Input #0, mpegts, from 'input_hdready_progressive_ntsc.ts':
  Duration: 00:01:31.61, start: 1.400000, bitrate: 4118 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc
    Stream #0:1[0x101]: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[h264 @ 0x55dca86f9cc0] mmco: unref short failure
[h264 @ 0x55dca86f9cc0] number of reference frames (0+6) exceeds max (5; probably corrupt input), discarding one
[h264 @ 0x55dca86ea000] mmco: unref short failure
Output #0, mpegts, to '/dev/null':
  Metadata:
    encoder         : Lavf58.22.100
    Stream #0:0: Video: h264 (h264_nvenc) (Main), cuda, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 59.94 fps, 90k tbn, 59.94 tbc
    Metadata:
      encoder         : Lavc58.39.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 128 kb/s
frame= 5410 fps=320 q=33.0 Lsize=   27270kB time=00:01:31.45 bitrate=2442.6kbits/s speed=5.41x    
video:23254kB audio:1415kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 10.543756%

same command with -bf 3, interlaced input BUT -hwaccel cuvid, succeeds

./ffmpeg-git -hwaccel cuvid -c:v h264_cuvid -f mpegts -i input_sd_interlaced_50i_mbaff.ts -vcodec h264_nvenc -refs 4 -bf 3 -c:a copy -f mpegts -y /dev/null
ffmpeg version N-92510-gfa08345e88 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-runtime-cpudetect --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-debug --enable-libaom --enable-avfilter --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-libnpp --enable-cuda-sdk --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-static --disable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 46.100 /  7. 46.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, mpegts, from 'input_sd_interlaced_50i_mbaff.ts':
  Duration: 00:01:42.92, start: 1.400000, bitrate: 1302 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, smpte170m, top first), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_cuvid) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mpegts, to '/dev/null':
  Metadata:
    encoder         : Lavf58.22.100
    Stream #0:0: Video: h264 (h264_nvenc) (Main), cuda, 720x576 [SAR 16:11 DAR 20:11], q=-1--1, 2000 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.39.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1(gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
frame= 2537 fps=410 q=16.0 Lsize=   29353kB time=00:01:43.08 bitrate=2332.8kbits/s speed=16.7x    
video:24905kB audio:1995kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.119826%

[Carl Eugen Hoyos]

Please use git bisect to find the change introducing the regression.

[malakudi]

Will try to do it as soon as possible. FYI, if there is a cuda filter in the command (scale_npp, yadif_cuda etc), then there is no issue again.

For example, this works:

./ffmpeg-git -hwaccel nvdec -hwaccel_output_format cuda -f mpegts -i input_sd_interlaced_50i_mbaff.ts -vf scale_npp=w=544 -vcodec h264_nvenc -refs 4 -bf 3 -c:a copy -f mpegts -y /dev/null
ffmpeg version N-92510-gfa08345e88 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-runtime-cpudetect --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-mips32r2 --disable-mips32r6 --disable-mips64r6 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-msa --disable-libopencv --disable-podpages --disable-sndio --disable-debug --enable-libaom --enable-avfilter --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx265 --enable-libxvid --enable-libzvbi --enable-libnpp --enable-cuda-sdk --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-static --disable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 23.101 / 56. 23.101
  libavcodec     58. 39.100 / 58. 39.100
  libavformat    58. 22.100 / 58. 22.100
  libavdevice    58.  6.100 / 58.  6.100
  libavfilter     7. 46.100 /  7. 46.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, mpegts, from 'input_sd_interlaced_50i_mbaff.ts':
  Duration: 00:01:42.92, start: 1.400000, bitrate: 1302 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, smpte170m, top first), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mpegts, to '/dev/null':
  Metadata:
    encoder         : Lavf58.22.100
    Stream #0:0: Video: h264 (h264_nvenc) (Main), cuda, 544x576 [SAR 360:187 DAR 20:11], q=-1--1, 2000 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.39.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: -1
    Stream #0:1(gre): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 160 kb/s
frame= 2537 fps=368 q=14.0 Lsize=   29123kB time=00:01:42.84 bitrate=2319.9kbits/s speed=14.9x    
video:24686kB audio:1995kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.155005%

[Timo R.]

There's not much to bisect here. This probably happens since the change from copying the frame to directly handing through the mapped cuvid surface, making the decoder dependend on the other parts of the chain releasing frames fast enough. The logic to calculate this does not seem to be flawless with interlaced content. I suspect it's a paff vs. mbaff issue as well.

You can always increase the buffer size by bumping up extra_hw_frames until the chain runs through.

[malakudi]

Replying to oromit:

There's not much to bisect here. This probably happens since the change from copying the frame to directly handing through the mapped cuvid surface, making the decoder dependend on the other parts of the chain releasing frames fast enough. The logic to calculate this does not seem to be flawless with interlaced content. I suspect it's a paff vs. mbaff issue as well.

You can always increase the buffer size by bumping up extra_hw_frames until the chain runs through.

Indeed, it works correctly on latest git by adding command line "-extra_hw_frames 2"

Maybe we could add an info message on error output like "No decoding surfaces left. Try to increase them with -extra_hw_frames option"

[oviano]

I've encountered this too under similar circumstances using the C API, but also with FFmpeg. In my case I had b frames and rc-lookahead in the encoder.

Since rc-lookahead in NVENC has a maximum of 32, and since that is the maximum number of decoder surfaces supported in NVDEC, it would seem logically impossible to do a a straight forward NVDEC -> NVENC transcode with rc-lookahead of 32.

This would not be solved by extra_hw_frames because 32 is the limit and NVDEC bails out if you try and add more.

The solution is hinted at by the fact if you include a scale_cuda or yadif_cuda then the issue goes away. I believe this is because these filters create their own hw_frames_ctx and generate frames from this as needed. This means the decoder surface is freed right away and the encoder just consumes the filter output frames as many as it likes.

I haven't tried, but it might be that cuda_scale specifying no change in width or height would also solve the problem. It shouldn't do, because if width/height is not supposed to change it should just pass the surface through but looking at vf_cuda_scale.c there is a passthrough variable defined, and set...but not used.

EDIT: just tried it, and the above does indeed work. So another workaround for this problem is to use a dummy cuda_scale. Although that would only last until such time as someone makes the passthrough actually passthrough....

[Balling]

So this could be because this dead code frames_ctx->initial_pool_size = dpb_size + 2; should be executed here ​https://github.com/FFmpeg/FFmpeg/blob/317b7b06fd97cd39feac7df57db22a30550351ff/libavcodec/nvdec.c#L621 see #6989. Well, it can be fixed with
-extra_hw_frames 2, OBVIOUSLY, no surprises here /facepalm.

And see comment, wow.

/*
     * We add two extra frames to the pool to account for deinterlacing filters
     * holding onto their frames.
     */

​https://github.com/Svechnikov/ffmpeg-cuda-deinterlace-problems this could be used, BTW (on the other hand for those files -extra_hw_frames 3 is required).

[hydra3333]

nice pickup, Balling. I'm just plain lucky I haven't encountered it.

[Balling]

You know what, I may be an idiot, but I CANNOT reproduce it anymore even on those 3 extra_hw_frames sample from above! At all. Please note that -refs option is now called -dpb_size.

On the other hand apparently deinterlace is so broken in nvxxx, it is insane.

​https://forums.developer.nvidia.com/t/nvdec-in-ffmpeg-cuvid-drops-frames-when-using-deinterlacing-deint-2/65837

Even 4 b frames work now...

[Balling]

It is very possible that this is fixed, but due to new default -bf -1 new bug is possible (1 of two bugs in #9351) even for not interlaced, apparently new auto logic is bad (P.S. I now believe that this is decoder bug and is just #8948). See: ​https://www.reddit.com/r/ffmpeg/comments/oxwskj/unable_to_make_cuda_ffmpeg_work/h7tbpud?utm_source=share&utm_medium=web2x&context=3