OpenEncodeSessionEx failed: out of memory (10): (no details)

[javex]

Summary of the bug: I'm trying to use my Nvidia 1660 Super to transcode a video (see info below) using the latest code from the master branch (a1bfb5290e) and I'm getting the following error:

[h264_nvenc @ 0x55d14231ab00] OpenEncodeSessionEx failed: out of memory (10): (no details)
[h264_nvenc @ 0x55d14231ab00] Nvenc unloaded
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

I've attached a full debug log (stderr output) in case it helps.

Info about the system:

$ uname -a
Linux media-server.home.xevaj.eu 5.15.71-1-lts #1 SMP Wed, 28 Sep 2022 10:41:31 +0000 x86_64 GNU/Linux
$ nvidia-smi
Fri Sep 30 22:25:27 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   61C    P0    34W / 125W |      0MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Not sure what else I can supply, I don't think the video file is the culprit, but happy to debug with others if you have some good samples I could use to confirm.

How to reproduce:

% ./ffmpeg -v 9 -loglevel 99 -hwaccel cuda -hwaccel_output_format cuda  -i ffmpeg-samples/input.mkv -c:v h264_nvenc ffmpeg-samples/output.mkv
ffmpeg version
built on
./ffmpeg --version
ffmpeg version N-108445-ga1bfb5290e Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.2.0 (GCC)
  configuration: --samples=ffmpeg-samples --cc= --enable-cuda-llvm --enable-nvenc
  libavutil      57. 38.100 / 57. 38.100
  libavcodec     59. 48.100 / 59. 48.100
  libavformat    59. 33.100 / 59. 33.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 49.100 /  8. 49.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100

Video data:

./ffmpeg -i ffmpeg-samples/input.mkv
ffmpeg version N-108445-ga1bfb5290e Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.2.0 (GCC)
  configuration: --samples=ffmpeg-samples --cc= --enable-cuda-llvm --enable-nvenc
  libavutil      57. 38.100 / 57. 38.100
  libavcodec     59. 48.100 / 59. 48.100
  libavformat    59. 33.100 / 59. 33.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 49.100 /  8. 49.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100
Input #0, matroska,webm, from 'ffmpeg-samples/input.mkv':
  Metadata:
    encoder         : libebml v1.3.6 + libmatroska v1.4.9
    creation_time   : 2018-07-11T04:28:24.000000Z
  Duration: 00:42:20.71, start: 0.000000, bitrate: 9310 kb/s
  Stream #0:0: Video: hevc (Main), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS-eng         : 8668560
      DURATION-eng    : 00:42:20.705000000
      NUMBER_OF_FRAMES-eng: 60916
      NUMBER_OF_BYTES-eng: 2753031717
      _STATISTICS_WRITING_APP-eng: mkvmerge v24.0.0 ('Beyond The Pale') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2018-07-11 04:28:24
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:1(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 640 kb/s (default)
    Metadata:
      BPS-eng         : 640000
      DURATION-eng    : 00:42:20.672000000
      NUMBER_OF_FRAMES-eng: 79396
      NUMBER_OF_BYTES-eng: 203253760
      _STATISTICS_WRITING_APP-eng: mkvmerge v24.0.0 ('Beyond The Pale') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2018-07-11 04:28:24
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng): Subtitle: subrip
    Metadata:
      BPS-eng         : 99
      DURATION-eng    : 00:41:29.453000000
      NUMBER_OF_FRAMES-eng: 923
      NUMBER_OF_BYTES-eng: 30928
      _STATISTICS_WRITING_APP-eng: mkvmerge v24.0.0 ('Beyond The Pale') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2018-07-11 04:28:24
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[javex]

Stderr output from ffmpeg command

[javex]

Note the only similar ticket I could find was ​https://trac.ffmpeg.org/ticket/4424 which did not have a resolution.

I have nothing else running on the GPU (see nvidia-smi output) so this is the only command sent to the GPU so I don't think it's about the limit mentioned there, but in case it is, I have not found any instructions on how to deal with it to stay at the limit of 2.

[Balling]

Why P0? Should not it be P8 on 34W??

[javex]

Replying to Balling:

Why P0? Should not it be P8 on 34W??

I'm not sure. How can I check or debug this issue?