nvenc failure at libavcodec/encode.c:438

[ajaraptor]

Summary of the bug:

nvenc fails on snapshot build, always with the same error:

Assertion !avpkt->data || avpkt->buf failed at libavcodec/encode.c:438

the bug is NOT present in the last stable version, 4.2.1

the specific code differences in encode.c that seem to be the source of the problem are new lines 435-438:

435 if (!ret)
436 // Encoders must always return ref-counted buffers.
437 // Side-data only packets have no data and can be not ref-counted.
438 av_assert0(!avpkt->data || avpkt->buf);
439 return ret;

These lines are NOT present in the 4.2.1 source for libavcodec/encode.c

Log/report files showing two failures using the snapshot build and one success (truncated) using the 4.2.1 build are submitted via VideoLAN. Also submitted is a linux diff file showing the diffences between the two versions of encode.c.

[Carl Eugen Hoyos]

Apparently 73ee53f317418a5719f6169e6171b40f90d18321

[hydra3333]

Yes, I'm getting this error as well.

"C:\SOFTWARE\ffmpeg\0-homebuilt-x64\ffmpeg.exe" -nostats -v verbose -init_hw_device opencl=ocl:0.0 -filter_hw_device ocl -i "D:\TEMP\SCRATCH\documentary.2019-11-16.mpg" -map_metadata -1 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -filter_complex "[0:v]yadif=0:0:0,hwupload,unsharp_opencl=lx=3:ly=3:la=0.5:cx=3:cy=3:ca=0.5,hwdownload,format=pix_fmts=yuv420p,setdar=dar=16/9" -c:v h264_nvenc -pix_fmt nv12 -preset slow -bf 2 -g 50 -refs 3 -rc:v vbr_hq -rc-lookahead:v 32 -cq 22 -qmin 16 -qmax 25 -coder cabac -strict experimental -movflags +faststart+write_colr -profile:v high -level 5.1 -af loudnorm=I=-16:TP=0.0:LRA=11:measured_I=-23.18:measured_LRA=4.90:measured_TP=-5.04:measured_thresh=-33.42:offset=-0.01:linear=true:print_format=summary -c:a libfdk_aac -cutoff 18000 -ab 384k -ar 48000  -y "T:\HDTV\autoTVS-mpg\Converted\documentary.2019-11-16.mp4" 
ffmpeg version N-95734-gbfa8272f40-hydra3333/v02_courtesy_DeadSix27 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (GCC)
  configuration: --arch=x86_64 --target-os=mingw64 --cross-prefix=x86_64-w64-mingw32- --pkg-config=pkg-config --disable-w32threads --enable-pthreads --enable-cross-compile --enable-pic --disable-shared --enable-static --enable-libsoxr --enable-libass --enable-iconv --enable-libtwolame --enable-libzvbi --enable-libcaca --enable-libmodplug --enable-libmp3lame --enable-version3 --enable-zlib --enable-librtmp --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libgsm --enable-libopus --enable-bzlib --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-libvpx --enable-libilbc --enable-libwavpack --enable-libwebp --enable-dxva2 --disable-avisynth --enable-vapoursynth --enable-gray --enable-libmysofa --enable-libflite --enable-lzma --enable-libsnappy --enable-libzimg --enable-libx264 --enable-libx265 --enable-libaom --enable-libdav1d --enable-frei0r --enable-filter=frei0r --enable-librubberband --enable-libvidstab --enable-libxvid --enable-libgme --enable-runtime-cpudetect --enable-libfribidi --enable-gnutls --enable-gmp --enable-fontconfig --enable-libfontconfig --enable-libfreetype --enable-libbluray --enable-libcdio --disable-schannel --enable-ladspa --enable-libxml2 --enable-libdavs2 --enable-libopenmpt --enable-libxavs --enable-libxavs2 --enable-libsrt --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-cuda-nvcc --enable-opengl --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-gpl --extra-version=hydra3333/v02_courtesy_DeadSix27 --enable-avresample --pkg-config-flags=--static --extra-libs='-lpsapi -lintl -liconv -lssp' --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DLIBXML_STATIC --extra-cflags=-DGLIB_STATIC_COMPILATION --enable-nonfree --enable-libfdk-aac --enable-decklink --prefix=/home/u/Desktop/workdir/x86_64_products/ffmpeg_static_non_free_opencl.installed --enable-opencl
  libavutil      56. 35.101 / 56. 35.101
  libavcodec     58. 62.100 / 58. 62.100
  libavformat    58. 35.100 / 58. 35.100
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 66.100 /  7. 66.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Routing option strict to both codec and muxer layer
[AVHWDeviceContext @ 00000189b036d740] 0.0: NVIDIA CUDA / GeForce RTX 2060 SUPER
[AVHWDeviceContext @ 00000189b036d740] DXVA2 to OpenCL mapping function found (clCreateFromDX9MediaSurfaceKHR).
[AVHWDeviceContext @ 00000189b036d740] DXVA2 in OpenCL acquire function found (clEnqueueAcquireDX9MediaSurfacesKHR).
[AVHWDeviceContext @ 00000189b036d740] DXVA2 in OpenCL release function found (clEnqueueReleaseDX9MediaSurfacesKHR).
[AVHWDeviceContext @ 00000189b036d740] The cl_khr_d3d11_sharing extension is required for D3D11 to OpenCL mapping.
[AVHWDeviceContext @ 00000189b036d740] D3D11 to OpenCL mapping not usable.
[mpeg @ 00000189b0372a00] max_analyze_duration 5000000 reached at 5000000 microseconds st:0
Input #0, mpeg, from 'D:\TEMP\SCRATCH\documentary.2019-11-16.mpg':
  Duration: 01:24:46.57, start: 0.229756, bitrate: 3747 kb/s
    Stream #0:0[0x1e0]: Video: mpeg2video (Main), 1 reference frame, yuv420p(tv, top first, left), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x1c0]: Audio: mp2, 48000 Hz, stereo, s16p, 192 kb/s
Stream mapping:
  Stream #0:0 (mpeg2video) -> yadif (graph 0)
  setdar (graph 0) -> Stream #0:0 (h264_nvenc)
  Stream #0:1 -> #0:1 (mp2 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 00000189c1fabe40] w:720 h:576 pixfmt:yuv420p tb:1/90000 fr:25/1 sar:64/45 sws_param:flags=2
[auto_scaler_0 @ 00000189c1ea9a80] w:iw h:ih flags:'bilinear' interl:0
[format @ 00000189c25c2a80] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_setdar_5' and the filter 'format'
[Parsed_setdar_5 @ 00000189c2033200] w:720 h:576 dar:16/9 sar:64/45 -> dar:16/9 sar:64/45
[auto_scaler_0 @ 00000189c1ea9a80] w:720 h:576 fmt:yuv420p sar:64/45 -> w:720 h:576 fmt:nv12 sar:64/45 flags:0x2
[h264_nvenc @ 00000189b03d16c0] Loaded Nvenc version 9.1
[h264_nvenc @ 00000189b03d16c0] Nvenc initialized successfully
[h264_nvenc @ 00000189b03d16c0] 1 CUDA capable devices found
[h264_nvenc @ 00000189b03d16c0] [ GPU #0 - < GeForce RTX 2060 SUPER > has Compute SM 7.5 ]
[h264_nvenc @ 00000189b03d16c0] supports NVENC
[h264_nvenc @ 00000189b03d16c0] Lookahead enabled: depth 32, scenecut enabled, B-adapt enabled.
[graph_1_in_0_1 @ 00000189c1ea8b80] tb:1/48000 samplefmt:s16p samplerate:48000 chlayout:0x3
[Parsed_loudnorm_0 @ 00000189c1ea9680] auto-inserting filter 'auto_resampler_0' between the filter 'graph_1_in_0_1' and the filter 'Parsed_loudnorm_0'
[format_out_0_1 @ 00000189c1ea8c80] auto-inserting filter 'auto_resampler_1' between the filter 'Parsed_loudnorm_0' and the filter 'format_out_0_1'
[auto_resampler_0 @ 00000189c1ea9e80] ch:2 chl:stereo fmt:s16p r:48000Hz -> ch:2 chl:stereo fmt:dbl r:192000Hz
[auto_resampler_1 @ 00000189c1ea9280] ch:2 chl:stereo fmt:dbl r:192000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz
Output #0, mp4, to 'T:\HDTV\autoTVS-mpg\Converted\documentary.2019-11-16.mp4':
  Metadata:
    encoder         : Lavf58.35.100
    Stream #0:0: Video: h264 (h264_nvenc) (High), 1 reference frame (avc1 / 0x31637661), nv12(progressive, left), 720x576 [SAR 64:45 DAR 16:9], q=16-25, 2000 kb/s, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.62.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
    Stream #0:1: Audio: aac (libfdk_aac) (mp4a / 0x6134706D), 48000 Hz, stereo, s16, delay 2048, 384 kb/s
    Metadata:
      encoder         : Lavc58.62.100 libfdk_aac
Assertion !avpkt->data || avpkt->buf failed at libavcodec/encode.c:438

[Timo R.]

Fixed by 4fb6ce27f0880599f0504ce4234b4de4fa87ad74