Opened 4 days ago

Closed 4 days ago

Last modified 3 days ago

#7674 closed defect (needs_more_info)

ffmpeg with cuvid transcoding after version 3.4.1 work unstable on heavy load CUDA card

Reported by: maxfs79 Owned by:
Priority: normal Component: undetermined
Version: unspecified Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Hi!

For the transoding of media streams I use Nvidia Quadro P5000 video cards and ffmpeg software version 3.4.1 (OS Linux Ubuntu 16.04.5, kernel 4.15).

On version 3.4.1 all work is fine on this video card it was possible to generate 79 H264 streams:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |

+----------------------+----------------------+

| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro P5000 Off | 00000000:01:00.0 Off | Off |
| 49% 76C P0 79W / 180W | 12792MiB / 16278MiB | 45% Default |
+-------------------------------+----------------------+----------------------+

# nvidia-smi -i 0 | grep ffmpeg | wc -l
79

---

Utilization

Gpu : 42 %
Memory : 16 %
Encoder : 42 %
Decoder : 92 %

When I try upgrade ffmpeg to version 3.4.2 or higer (4.X ot 4.X), ffmpeg was work unstable afte 40 streams.

Configuration ffmpeg:
ffmpeg -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -drop_second_field 1 -i udp://232.10.10.1:1234?fifo_size=300000 -b:v 2800k -b:a 192k -c:v h264_nvenc -profile:v high -preset hp -c:a aac -f flv rtmp://127.0.0.1:1935/live/001

What can be a issue ?

Change History (16)

comment:1 Changed 4 days ago by cehoyos

  • Component changed from ffmpeg to undetermined

Please test current FFmpeg git head and provide a simplified command line (if possible without network input or output) including the complete, uncut console output to make this a valid ticket. If you see a crash, please provide backtrace, disassembly and register dump as explained on https://ffmpeg.org/bugreports.html

comment:2 Changed 4 days ago by cehoyos

If you believe there is a regression, run git bisect to find the change introducing the issue.

comment:3 follow-up: Changed 4 days ago by oromit

Also, please be _a lot_ more specific than "ffmpeg was work unstable".

comment:4 Changed 4 days ago by maxfs79

  • Summary changed from ffmpeg with cuvid transcoding afte version 3.4.1 work with crash on heavy load CUDA card to ffmpeg with cuvid transcoding after version 3.4.1 work with crash on heavy load CUDA card

comment:5 in reply to: ↑ 3 Changed 4 days ago by maxfs79

Replying to oromit:

Also, please be _a lot_ more specific than "ffmpeg was work unstable".

Unstable work looks like this, if up to 40 ffmpeg is running at the same time, then everything is OK, if I add for example 55 streams, the nvidia-smi utility slowly shows statistics, and the ffmpeg processes randomly exit after some time.

comment:6 Changed 4 days ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from new to closed

Please reopen this ticket if you can confirm the issue is reproducible with current FFmpeg git head, if you can point us to the commit introducing the regression and if you can provide simplified command line including the complete, uncut console output.

comment:7 Changed 4 days ago by maxfs79

Hi!

I was check issue on git-master version, this is issue also have.

Than I was removed patchs (revert) from version 4.1, now my issue is resolve :)

I think this is patch need analyzing in case, when used cuvid transcoding:

932037c6bb6b41a24e75b031426844a2e6472a74
48e52e4edd12adbc36eee0eebe1b97ffe0255be3
32bc4e77f61a5483c83a360b9ccbfc2840daba1e
bbe1b21022e4872bc64066d46a4567dc1b655f7a

comment:8 Changed 4 days ago by maxfs79

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

comment:9 Changed 4 days ago by cehoyos

To make this a valid bug report please provide the command line you tested together with the complete, uncut console output and point us to the change (it is one commit, not four) that introduced the regression.

comment:10 Changed 4 days ago by maxfs79

My step-by-step:

git clone -b release/4.1 https://git.ffmpeg.org/ffmpeg.git ffmpeg41

dependent patches, for the revert "32bc4e77f61a5483c83a360b9ccbfc2840daba1e"
git revert 932037c6bb6b41a24e75b031426844a2e6472a74
git revert 48e52e4edd12adbc36eee0eebe1b97ffe0255be3

regression patch:
git revert 32bc4e77f61a5483c83a360b9ccbfc2840daba1e

test configuration

ffmpeg -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -drop_second_field 1 -i udp://232.10.10.1:1234?fifo_size=300000 -b:v 2800k -b:a 192k -c:v h264_nvenc -profile:v high -preset hp -c:a aac -f flv rtmp://127.0.0.1:1935/live/001
.
.
.
ffmpeg -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -drop_second_field 1 -i udp://232.10.10.60:1234?fifo_size=300000 -b:v 2800k -b:a 192k -c:v h264_nvenc -profile:v high -preset hp -c:a aac -f flv rtmp://127.0.0.1:1935/live/060

With issue patch, when 59 stream, starting next stream over 20 sec and on others stream freeze video.

comment:11 Changed 4 days ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from reopened to closed

comment:12 Changed 4 days ago by maxfs79

  • Summary changed from ffmpeg with cuvid transcoding after version 3.4.1 work with crash on heavy load CUDA card to ffmpeg with cuvid transcoding after version 3.4.1 work unstable on heavy load CUDA card

comment:13 Changed 3 days ago by oromit

The patch you claim introduced the regression fixes a leak and potential crash. It seems very unlikely to me that it would introduce performance issues.

comment:14 Changed 3 days ago by maxfs79

May be when using cuvid transcoding video frames are processed inside the GPU, it is possible that this was not provided when developing the patch.

comment:15 Changed 3 days ago by oromit

That patch is specifically only for the case of pure on-GPU transcoding. It sits in the path that registers a CUDA frame to nvenv.

comment:16 Changed 3 days ago by maxfs79

That's right, but when added two strings:

+ p_nvenc->nvEncUnregisterResource(ctx->nvencoder, ctx->registered_frames[tmpoutsurf->reg_idx].regptr);
+ ctx->registered_frames[tmpoutsurf->reg_idx].regptr = NULL;

GPU adapter begin to work slowly when load encoder to 80-90% :(

Note: See TracTickets for help on using tickets.