Opened 6 months ago

Last modified 7 weeks ago

#7030 open defect

qsv in ffmpeg

Reported by: palich2000 Owned by:
Priority: important Component: avcodec
Version: git-master Keywords: qsv regression
Cc: zhong.li@intel.com Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Testing was conducted on the same equipment on the same video stream.
configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-testinclude -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-testlib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-libmp3lame --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --enable-runtime-cpudetect --enable-decklink --enable-libmfx

ffmpeg version 3.2.10

ffmpeg -i rtmp://...
-c:v h264_qsv -b:v 1000k -g 50 -r 25
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

%CPU LOAD (top) 29,7
%GPU LOAD (metrics_monitor)
RENDER usage: 15.15, VIDEO usage: 6.06, VIDEO_E usage: 0.00 GT Freq: 350.00

This works well.

ffmpeg version 3.4.2
add -init_hw_device qsv:hw

ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

%CPU LOAD (top) 36.0
%GPU LOAD (metrics_monitor)
RENDER usage: 14.14, VIDEO usage: 5.05, VIDEO_E usage: 0.00 GT Freq: 350.00

This works well.

ffmpeg version 3.3.6

ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

Unrecognized option 'init_hw_device'.

Without this option, QSV is not initialized.
And gives an error. Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.

ffmpeg version N-90085-gacdea9e

ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.

How to make QSV work in these versions ? (3.3.6 and N-90085-gacdea9e)

./sys_analyzer_linux.py
--------------------------
Hardware readiness checks:
--------------------------
 [ OK ] Processor name: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
--------------------------
OS readiness checks:
--------------------------
 [ OK ] GPU visible to OS
--------------------------
Media Server Studio Install:
--------------------------
 [ OK ] user in video group
 [ OK ] libva.so.1 found
 [ OK ] vainfo reports valid codec entry points
 [ OK ] /dev/dri/renderD128 connects to Intel i915
--------------------------
Component Smoke Tests:
--------------------------
 [ OK ] Media SDK HW API level:1.19
 [ OK ] Media SDK SW API level:1.19
 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK

Attachments (1)

ffmpeg-20180424-153510.log (76.9 KB) - added by lizhong1008 4 months ago.

Download all attachments as: .zip

Change History (22)

comment:1 Changed 6 months ago by cehoyos

  • Keywords qsv regression added
  • Version changed from 3.4 to git-master

Please find the commit that introduced the regression.

comment:2 Changed 6 months ago by palich2000

In the b26dd5a works.
But on the dde7b1d does not work anymore.

comment:3 Changed 6 months ago by cehoyos

Please run git bisect to find the change that introduced the regression.

comment:4 Changed 6 months ago by palich2000

git bisect good
b0cd14fb1dab4b044f7fe6b53ac635409849de77 is the first bad commit
commit b0cd14fb1dab4b044f7fe6b53ac635409849de77
Author: Mark Thompson <sw@jkqxz.net>
Date: Thu Oct 26 00:18:47 2017 +0100

ffmpeg: Use codec hardware config to configure hwaccels


Removes specific support for all hwaccels supported by the generic code
(DXVA2, D3D11VA, NVDEC, VAAPI and VDPAU).

:040000 040000 db8084af9c0723e94edafc6d4beb29386dee6288 a1b4cd2717b8f391249024ea335241ac7788f843 M fftools

comment:5 Changed 6 months ago by cehoyos

  • Status changed from new to open

comment:6 Changed 6 months ago by palich2000

The author of the commit Mark Thompson <​sw@jkqxz.net> had some grandiose idea but he did not realize it to the end. I do not know how the hardware decoders work, but the hardware coders do not work after this commit. The function hw_device_setup_for_encode suggests that there is a hardware configuration for each encoder, but they are not declared. And hardware encoders do not connect to the codec. This should either be removed or completed.

comment:7 Changed 6 months ago by palich2000

This patch solves the problem, but only for h264_qsv encoder.

diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c
index 09e4c0e..5f60675 100644
--- a/libavcodec/qsvenc_h264.c
+++ b/libavcodec/qsvenc_h264.c
@@ -34,6 +34,7 @@
 #include "qsv.h"
 #include "qsv_internal.h"
 #include "qsvenc.h"
+#include "hwaccel.h"
 
 typedef struct QSVH264EncContext {
     AVClass *class;
@@ -169,6 +170,16 @@ static const AVCodecDefault qsv_enc_defaults[] = {
     { NULL },
 };
 
+#define HWACCEL_QSV(codec) \
+    HW_CONFIG_HWACCEL(1, 1, 1, QSV,          QSV,          ff_ ## codec ## _qsv_hwaccel)
+
+const AVHWAccel ff_h264_qsv_hwaccel = {
+    .name                 = "h264_qsv",
+    .type                 = AVMEDIA_TYPE_VIDEO,
+    .id                   = AV_CODEC_ID_H264,
+    .pix_fmt              = AV_PIX_FMT_QSV
+};
+                
 AVCodec ff_h264_qsv_encoder = {
     .name           = "h264_qsv",
     .long_name      = NULL_IF_CONFIG_SMALL("H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration)"),
@@ -186,5 +197,9 @@ AVCodec ff_h264_qsv_encoder = {
     .priv_class     = &class,
     .defaults       = qsv_enc_defaults,
     .caps_internal  = FF_CODEC_CAP_INIT_CLEANUP,
+    .hw_configs            = (const AVCodecHWConfigInternal*[]) {
+                       HWACCEL_QSV(h264),
+    NULL
+    },
     .wrapper_name   = "qsv",
 };

Last edited 6 months ago by palich2000 (previous) (diff)

comment:8 Changed 6 months ago by cehoyos

Please send your patch - made with git format-patch - to the FFmpeg development mailing list where it can be reviewed.

comment:9 Changed 6 months ago by palich2000

OK
Sended.

comment:10 Changed 6 months ago by jkqxz

This works fine for me. At a guess you are somehow opening the DRM master device rather than a render node and therefore the unused top-level device has exclusive access to the GPU and stops the encoder from opening it?

If that's the problem then not creating the unused top-level device (remove the -init_hw_device option) should make it work for you. (That device did automatically get passed to an encoder in 3.4, but there was disagreement about that feature being useful because ffmpeg was unable to tell whether the device was required or not (it isn't here, the internal session works for isolated encoders) and offered unhelpful warnings.)

comment:11 Changed 6 months ago by palich2000

Without this parameter, the situation is as follows.
The codec tries to code programmatically and it does not work.

[h264_qsv @ 0x1e87ec0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x1e8a540] Qavg: 62368.684

comment:12 Changed 6 months ago by jkqxz

Then I suggest providing more information about your setup, and a complete log of a failing command.

comment:13 follow-up: Changed 6 months ago by palich2000

Look above, there I described everything.

comment:14 Changed 6 months ago by heleppkes

The message even has a hint of what to do - try choosing a different rate control mode.

comment:15 Changed 6 months ago by palich2000

Believe me, this will not help.
You can choose the parameters for which there will be no error. But the encoding will occur on the CPU and not on the GPU.

comment:16 in reply to: ↑ 13 Changed 6 months ago by jkqxz

Replying to palich2000:

Look above, there I described everything.

If might help if you could increase the scope of "everything" to include:

  • Operating system details, including kernel version.
  • What version of the Media SDK you are using (some specific proprietary version, the open-source one?).
  • A complete log of ffmpeg output for the failing case.

comment:17 Changed 6 months ago by palich2000

Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
 [ OK ] Media SDK HW API level:1.19
 [ OK ] Media SDK SW API level:1.19
 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK
ffmpeg -t 30 -v verbose -i rtmp://xxxxx \
    -c:v h264_qsv -b:v 1000k\
    -strict experimental -acodec aac -ab 64k -ar 44100 \
    -f flv rtmp://127.0.0.1/vip/_77_
ffmpeg version N-90198-g7056d06 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11)
  configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-test//include -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-test//lib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --disable-doc --enable-libmp3lame --disable-stripping --enable-debug=3 --enable-runtime-cpudetect --enable-libmfx
  libavutil      56.  8.100 / 56.  8.100
  libavcodec     58. 13.100 / 58. 13.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 12.100 /  7. 12.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Routing option strict to both codec and muxer layer
Parsing...
Parsed protocol: 0
Parsed host    : vr0.dpl-streaming.com
Parsed app     : virtual
RTMP_Connect1, ... connected, handshaking
HandShake: Type Answer   : 03
HandShake: Server Uptime : 134225664
HandShake: FMS Version   : 0.0.0.0
HandShake: Handshaking finished....
RTMP_Connect1, handshaked
Invoking connect
HandleServerBW: server BW = 5000000
HandleClientBW: client BW = 5000000 2
HandleChangeChunkSize, received: chunk size change to 4096
RTMP_ClientPacket, received: invoke 190 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    1.00>
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:             fmsVer, STRING:    FMS/3,0,1,123>
Property: <Name:       capabilities, NUMBER:    31.00>
(object end)
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetConnection.Connect.Success>
Property: <Name:        description, STRING:    Connection succeeded.>
Property: <Name:     objectEncoding, NUMBER:    0.00>
(object end)
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <connect>
sending ctrl. type: 0x0003
Invoking createStream
RTMP_ClientPacket, received: invoke 29 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    2.00>
Property: NULL
Property: <Name:           no-name., NUMBER:    1.00>
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <createStream>
SendPlay, seekTime=0, stopTime=0, sending play: vrm_foot_live
Invoking play
sending ctrl. type: 0x0003
HandleCtrl, received ctrl. type: 0, len: 6
HandleCtrl, Stream Begin 1
RTMP_ClientPacket, received: invoke 96 bytes
(object begin)
Property: <Name:           no-name., STRING:    onStatus>
Property: <Name:           no-name., NUMBER:    0.00>
Property: NULL
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetStream.Play.Start>
Property: <Name:        description, STRING:    Start live>
(object end)
(object end)
HandleInvoke, server invoking <onStatus>
HandleInvoke, onStatus: NetStream.Play.Start
RTMP_ClientPacket, received: notify 24 bytes
(object begin)
Property: <Name:           no-name., STRING:    |RtmpSampleAccess>
Property: <Name:           no-name., BOOLEAN:   TRUE>
Property: <Name:           no-name., BOOLEAN:   TRUE>
(object end)
RTMP_ClientPacket, received: invoke 110 bytes
(object begin)
Property: <Name:           no-name., STRING:    onStatus>
Property: <Name:           no-name., NUMBER:    0.00>
Property: NULL
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetStream.Play.PublishNotify>
Property: <Name:        description, STRING:    Start publishing>
(object end)
(object end)
HandleInvoke, server invoking <onStatus>
HandleInvoke, onStatus: NetStream.Play.PublishNotify
RTMP_ClientPacket, received: notify 387 bytes
(object begin)
Property: <Name:           no-name., STRING:    onMetaData>
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:             Server, STRING:    NGINX RTMP (github.com/arut/nginx-rtmp-module)>
Property: <Name:              width, NUMBER:    1280.00>
Property: <Name:             height, NUMBER:    720.00>
Property: <Name:       displayWidth, NUMBER:    1280.00>
Property: <Name:      displayHeight, NUMBER:    720.00>
Property: <Name:           duration, NUMBER:    0.00>
Property: <Name:          framerate, NUMBER:    25.00>
Property: <Name:                fps, NUMBER:    25.00>
Property: <Name:      videodatarate, NUMBER:    976.00>
Property: <Name:       videocodecid, NUMBER:    7.00>
Property: <Name:      audiodatarate, NUMBER:    62.00>
Property: <Name:       audiocodecid, NUMBER:    10.00>
Property: <Name:            profile, STRING:    >
Property: <Name:              level, STRING:    >
(object end)
(object end)
Metadata:
  Server                NGINX RTMP (github.com/arut/nginx-rtmp-module)
  width                 1280.00
  height                720.00
  displayWidth          1280.00
  displayHeight         720.00
  duration              0.00
  framerate             25.00
  fps                   25.00
  videodatarate         976.00
  videocodecid          7.00
  audiodatarate         62.00
  audiocodecid          10.00
[h264 @ 0x138b040] Reinit context to 1280x720, pix_fmt: yuv420p
Input #0, live_flv, from 'rtmp://xxxxxxxx':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1280
    displayHeight   : 720
    fps             : 25
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 100110.018000, bitrate: N/A
    Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720 [SAR 1:1 DAR 16:9], 999 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc
    Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 63 kb/s
Parsing...
Parsed protocol: 0
Parsed host    : 127.0.0.1
Parsed app     : vip
RTMP_Connect1, ... connected, handshaking
HandShake: Type Answer   : 03
HandShake: Server Uptime : 134227824
HandShake: FMS Version   : 0.0.0.0
HandShake: Handshaking finished....
RTMP_Connect1, handshaked
Invoking connect
HandleServerBW: server BW = 5000000
HandleClientBW: client BW = 5000000 2
HandleChangeChunkSize, received: chunk size change to 1024
RTMP_ClientPacket, received: invoke 190 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    1.00>
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:             fmsVer, STRING:    FMS/3,0,1,123>
Property: <Name:       capabilities, NUMBER:    31.00>
(object end)
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetConnection.Connect.Success>
Property: <Name:        description, STRING:    Connection succeeded.>
Property: <Name:     objectEncoding, NUMBER:    0.00>
(object end)
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <connect>
Invoking releaseStream
Invoking FCPublish
Invoking createStream
RTMP_ClientPacket, received: invoke 29 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    4.00>
Property: NULL
Property: <Name:           no-name., NUMBER:    1.00>
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <createStream>
Invoking publish
RTMP_ClientPacket, received: invoke 105 bytes
(object begin)
Property: <Name:           no-name., STRING:    onStatus>
Property: <Name:           no-name., NUMBER:    0.00>
Property: NULL
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetStream.Publish.Start>
Property: <Name:        description, STRING:    Start publishing>
(object end)
(object end)
HandleInvoke, server invoking <onStatus>
HandleInvoke, onStatus: NetStream.Publish.Start
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_qsv))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[graph_1_in_0_1 @ 0x168c680] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3
[h264 @ 0x14b4f40] Reinit context to 1280x720, pix_fmt: yuv420p
[graph 0 input from stream 0:0 @ 0x139b400] w:1280 h:720 pixfmt:yuv420p tb:1/1000 fr:25/1 sar:1/1 sws_param:flags=2
[auto_scaler_0 @ 0x13f6600] w:iw h:ih flags:'bicubic' interl:0
[format @ 0x148d5c0] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format'
[auto_scaler_0 @ 0x13f6600] w:1280 h:720 fmt:yuv420p sar:1/1 -> w:1280 h:720 fmt:nv12 sar:1/1 flags:0x4
[h264_qsv @ 0x13903c0] Initialized an internal MFX session using hardware accelerated implementation
[h264_qsv @ 0x13903c0] Using the VBR with lookahead (LA) ratecontrol method
[h264_qsv @ 0x13903c0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Invoking FCUnpublish
Invoking deleteStream
[aac @ 0x14854c0] Qavg: 62580.895
[aac @ 0x14854c0] 2 frames left in the queue on closing
Invoking deleteStream
Conversion failed!

comment:18 Changed 4 months ago by lizhong1008

I've tried to reproduce it on ffmpeg-4.0 branch:
./ffmpeg -loglevel verbose -init_hw_device qsv=hw -i /samba/anonymous/Videos/bbb_sunflower_1080p_30fps_normal.mp4 -c:v h264_qsv -b:v 1000k test.mp4

The transcoding pipeline didn't crash, but reported "Encoder will work with partial HW acceleration, Warning in encoder initialization: partial acceleration (4)". The transcoding speed was very slow (about 20 fps), and it shows the gpu usage is zero when use metrics_monitor to check GPU usage.
It means software path was used instead of GPU accelerated.

comment:19 Changed 4 months ago by lizhong1008

More detail:
OS info:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.3.1611 (Core)
Release: 7.3.1611
Codename: Core

MSDK version: MediaServerStudioEssentials2017R3 (API version 1.23)

FFmpeg log (see the attachment)

Check GPU usage with metrics_monitor

./metrics_monitor
RENDER usage: 0.00,     VIDEO usage: 0.00,      VIDEO_E usage: 0.00     VIDEO2 usage: 0.00      GT Freq: 300.00

Changed 4 months ago by lizhong1008

comment:20 Changed 4 months ago by lizhong1008

  • Cc zhong.li@intel.com added

comment:21 Changed 7 weeks ago by lizhong1008

Not sure why it works well on Mark's side as https://trac.ffmpeg.org/ticket/7030#comment:10 (Maybe it just broken on Linux but works well on Wondows? I can understand "-init_hw_device" maybe not needed on Windows since the commit 1f26a231bb065276cd80ce02957c759f3197edfa removing ff_qsv_set_display_handle() has no impact on Windows).
But on Linux, passing a decoded frame by software decoder to qsv encoder directly is broken on FFmpeg 4.0 (this is the problem of this ticket). Passing a qsv decoded frame to a software encoder is also broken.(Reproduce: ffmpeg -c:v h264_qsv -i bbb_sunflower_1080p_30fps_normal.h264 -vframes 100 -c:v libx264 test.h264). These paths are supported on ffmpeg-3.4.
On FFmpeg 4.0, looks like we must use "hwupload" or "hwdownload" for these cases.
An example of this issue, an alternative way without palich's patch is using hwupload:
./ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -i input.mp4 -vframes 100 -vf "hwupload=extra_hw_frames=64, format=qsv" -c:v h264_qsv out.mp4

Last edited 7 weeks ago by lizhong1008 (previous) (diff)
Note: See TracTickets for help on using tickets.