qsv in ffmpeg

[palich2000]

Testing was conducted on the same equipment on the same video stream.
configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-testinclude -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-testlib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-libmp3lame --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --enable-runtime-cpudetect --enable-decklink --enable-libmfx

ffmpeg version 3.2.10

ffmpeg -i rtmp://...
-c:v h264_qsv -b:v 1000k -g 50 -r 25
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

%CPU LOAD (top) 29,7
%GPU LOAD (metrics_monitor)
RENDER usage: 15.15, VIDEO usage: 6.06, VIDEO_E usage: 0.00 GT Freq: 350.00

This works well.

ffmpeg version 3.4.2
add -init_hw_device qsv:hw

ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

%CPU LOAD (top) 36.0
%GPU LOAD (metrics_monitor)
RENDER usage: 14.14, VIDEO usage: 5.05, VIDEO_E usage: 0.00 GT Freq: 350.00

This works well.

ffmpeg version 3.3.6

ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

Unrecognized option 'init_hw_device'.

Without this option, QSV is not initialized.
And gives an error. Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.

ffmpeg version N-90085-gacdea9e

ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1

Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.

How to make QSV work in these versions ? (3.3.6 and N-90085-gacdea9e)

./sys_analyzer_linux.py
--------------------------
Hardware readiness checks:
--------------------------
 [ OK ] Processor name: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
--------------------------
OS readiness checks:
--------------------------
 [ OK ] GPU visible to OS
--------------------------
Media Server Studio Install:
--------------------------
 [ OK ] user in video group
 [ OK ] libva.so.1 found
 [ OK ] vainfo reports valid codec entry points
 [ OK ] /dev/dri/renderD128 connects to Intel i915
--------------------------
Component Smoke Tests:
--------------------------
 [ OK ] Media SDK HW API level:1.19
 [ OK ] Media SDK SW API level:1.19
 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK

[cehoyos]

Please find the commit that introduced the regression.

[palich2000]

In the b26dd5a works.
But on the dde7b1d does not work anymore.

[cehoyos]

Please run git bisect to find the change that introduced the regression.

[palich2000]

git bisect good
b0cd14fb1dab4b044f7fe6b53ac635409849de77 is the first bad commit
commit b0cd14fb1dab4b044f7fe6b53ac635409849de77
Author: Mark Thompson <​sw@jkqxz.net>
Date: Thu Oct 26 00:18:47 2017 +0100

ffmpeg: Use codec hardware config to configure hwaccels

Removes specific support for all hwaccels supported by the generic code
(DXVA2, D3D11VA, NVDEC, VAAPI and VDPAU).

:040000 040000 db8084af9c0723e94edafc6d4beb29386dee6288 a1b4cd2717b8f391249024ea335241ac7788f843 M fftools

[palich2000]

The author of the commit Mark Thompson <​​sw@jkqxz.net> had some grandiose idea but he did not realize it to the end. I do not know how the hardware decoders work, but the hardware coders do not work after this commit. The function hw_device_setup_for_encode suggests that there is a hardware configuration for each encoder, but they are not declared. And hardware encoders do not connect to the codec. This should either be removed or completed.

[palich2000]

This patch solves the problem, but only for h264_qsv encoder.

diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c
index 09e4c0e..5f60675 100644
--- a/libavcodec/qsvenc_h264.c
+++ b/libavcodec/qsvenc_h264.c
@@ -34,6 +34,7 @@
 #include "qsv.h"
 #include "qsv_internal.h"
 #include "qsvenc.h"
+#include "hwaccel.h"
 
 typedef struct QSVH264EncContext {
     AVClass *class;
@@ -169,6 +170,16 @@ static const AVCodecDefault qsv_enc_defaults[] = {
     { NULL },
 };
 
+#define HWACCEL_QSV(codec) \
+    HW_CONFIG_HWACCEL(1, 1, 1, QSV,          QSV,          ff_ ## codec ## _qsv_hwaccel)
+
+const AVHWAccel ff_h264_qsv_hwaccel = {
+    .name                 = "h264_qsv",
+    .type                 = AVMEDIA_TYPE_VIDEO,
+    .id                   = AV_CODEC_ID_H264,
+    .pix_fmt              = AV_PIX_FMT_QSV
+};
+                
 AVCodec ff_h264_qsv_encoder = {
     .name           = "h264_qsv",
     .long_name      = NULL_IF_CONFIG_SMALL("H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration)"),
@@ -186,5 +197,9 @@ AVCodec ff_h264_qsv_encoder = {
     .priv_class     = &class,
     .defaults       = qsv_enc_defaults,
     .caps_internal  = FF_CODEC_CAP_INIT_CLEANUP,
+    .hw_configs            = (const AVCodecHWConfigInternal*[]) {
+                       HWACCEL_QSV(h264),
+    NULL
+    },
     .wrapper_name   = "qsv",
 };

[cehoyos]

Please send your patch - made with git format-patch - to the FFmpeg development mailing list where it can be reviewed.

[palich2000]

OK
Sended.

[jkqxz]

This works fine for me. At a guess you are somehow opening the DRM master device rather than a render node and therefore the unused top-level device has exclusive access to the GPU and stops the encoder from opening it?

If that's the problem then not creating the unused top-level device (remove the -init_hw_device option) should make it work for you. (That device did automatically get passed to an encoder in 3.4, but there was disagreement about that feature being useful because ffmpeg was unable to tell whether the device was required or not (it isn't here, the internal session works for isolated encoders) and offered unhelpful warnings.)

[palich2000]

Without this parameter, the situation is as follows.
The codec tries to code programmatically and it does not work.

[h264_qsv @ 0x1e87ec0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x1e8a540] Qavg: 62368.684

[jkqxz]

Then I suggest providing more information about your setup, and a complete log of a failing command.

[palich2000]

Look above, there I described everything.

[heleppkes]

The message even has a hint of what to do - try choosing a different rate control mode.

[palich2000]

Believe me, this will not help.
You can choose the parameters for which there will be no error. But the encoding will occur on the CPU and not on the GPU.

[jkqxz]

Replying to palich2000:

Look above, there I described everything.

If might help if you could increase the scope of "everything" to include:

• Operating system details, including kernel version.
• What version of the Media SDK you are using (some specific proprietary version, the open-source one?).
• A complete log of ffmpeg output for the failing case.

[palich2000]

Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

 [ OK ] Media SDK HW API level:1.19
 [ OK ] Media SDK SW API level:1.19
 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK

ffmpeg -t 30 -v verbose -i rtmp://xxxxx \
    -c:v h264_qsv -b:v 1000k\
    -strict experimental -acodec aac -ab 64k -ar 44100 \
    -f flv rtmp://127.0.0.1/vip/_77_

ffmpeg version N-90198-g7056d06 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11)
  configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-test//include -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-test//lib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --disable-doc --enable-libmp3lame --disable-stripping --enable-debug=3 --enable-runtime-cpudetect --enable-libmfx
  libavutil      56.  8.100 / 56.  8.100
  libavcodec     58. 13.100 / 58. 13.100
  libavformat    58. 10.100 / 58. 10.100
  libavdevice    58.  2.100 / 58.  2.100
  libavfilter     7. 12.100 /  7. 12.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Routing option strict to both codec and muxer layer
Parsing...
Parsed protocol: 0
Parsed host    : vr0.dpl-streaming.com
Parsed app     : virtual
RTMP_Connect1, ... connected, handshaking
HandShake: Type Answer   : 03
HandShake: Server Uptime : 134225664
HandShake: FMS Version   : 0.0.0.0
HandShake: Handshaking finished....
RTMP_Connect1, handshaked
Invoking connect
HandleServerBW: server BW = 5000000
HandleClientBW: client BW = 5000000 2
HandleChangeChunkSize, received: chunk size change to 4096
RTMP_ClientPacket, received: invoke 190 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    1.00>
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:             fmsVer, STRING:    FMS/3,0,1,123>
Property: <Name:       capabilities, NUMBER:    31.00>
(object end)
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetConnection.Connect.Success>
Property: <Name:        description, STRING:    Connection succeeded.>
Property: <Name:     objectEncoding, NUMBER:    0.00>
(object end)
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <connect>
sending ctrl. type: 0x0003
Invoking createStream
RTMP_ClientPacket, received: invoke 29 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    2.00>
Property: NULL
Property: <Name:           no-name., NUMBER:    1.00>
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <createStream>
SendPlay, seekTime=0, stopTime=0, sending play: vrm_foot_live
Invoking play
sending ctrl. type: 0x0003
HandleCtrl, received ctrl. type: 0, len: 6
HandleCtrl, Stream Begin 1
RTMP_ClientPacket, received: invoke 96 bytes
(object begin)
Property: <Name:           no-name., STRING:    onStatus>
Property: <Name:           no-name., NUMBER:    0.00>
Property: NULL
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetStream.Play.Start>
Property: <Name:        description, STRING:    Start live>
(object end)
(object end)
HandleInvoke, server invoking <onStatus>
HandleInvoke, onStatus: NetStream.Play.Start
RTMP_ClientPacket, received: notify 24 bytes
(object begin)
Property: <Name:           no-name., STRING:    |RtmpSampleAccess>
Property: <Name:           no-name., BOOLEAN:   TRUE>
Property: <Name:           no-name., BOOLEAN:   TRUE>
(object end)
RTMP_ClientPacket, received: invoke 110 bytes
(object begin)
Property: <Name:           no-name., STRING:    onStatus>
Property: <Name:           no-name., NUMBER:    0.00>
Property: NULL
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetStream.Play.PublishNotify>
Property: <Name:        description, STRING:    Start publishing>
(object end)
(object end)
HandleInvoke, server invoking <onStatus>
HandleInvoke, onStatus: NetStream.Play.PublishNotify
RTMP_ClientPacket, received: notify 387 bytes
(object begin)
Property: <Name:           no-name., STRING:    onMetaData>
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:             Server, STRING:    NGINX RTMP (github.com/arut/nginx-rtmp-module)>
Property: <Name:              width, NUMBER:    1280.00>
Property: <Name:             height, NUMBER:    720.00>
Property: <Name:       displayWidth, NUMBER:    1280.00>
Property: <Name:      displayHeight, NUMBER:    720.00>
Property: <Name:           duration, NUMBER:    0.00>
Property: <Name:          framerate, NUMBER:    25.00>
Property: <Name:                fps, NUMBER:    25.00>
Property: <Name:      videodatarate, NUMBER:    976.00>
Property: <Name:       videocodecid, NUMBER:    7.00>
Property: <Name:      audiodatarate, NUMBER:    62.00>
Property: <Name:       audiocodecid, NUMBER:    10.00>
Property: <Name:            profile, STRING:    >
Property: <Name:              level, STRING:    >
(object end)
(object end)
Metadata:
  Server                NGINX RTMP (github.com/arut/nginx-rtmp-module)
  width                 1280.00
  height                720.00
  displayWidth          1280.00
  displayHeight         720.00
  duration              0.00
  framerate             25.00
  fps                   25.00
  videodatarate         976.00
  videocodecid          7.00
  audiodatarate         62.00
  audiocodecid          10.00
[h264 @ 0x138b040] Reinit context to 1280x720, pix_fmt: yuv420p
Input #0, live_flv, from 'rtmp://xxxxxxxx':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1280
    displayHeight   : 720
    fps             : 25
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 100110.018000, bitrate: N/A
    Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720 [SAR 1:1 DAR 16:9], 999 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc
    Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 63 kb/s
Parsing...
Parsed protocol: 0
Parsed host    : 127.0.0.1
Parsed app     : vip
RTMP_Connect1, ... connected, handshaking
HandShake: Type Answer   : 03
HandShake: Server Uptime : 134227824
HandShake: FMS Version   : 0.0.0.0
HandShake: Handshaking finished....
RTMP_Connect1, handshaked
Invoking connect
HandleServerBW: server BW = 5000000
HandleClientBW: client BW = 5000000 2
HandleChangeChunkSize, received: chunk size change to 1024
RTMP_ClientPacket, received: invoke 190 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    1.00>
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:             fmsVer, STRING:    FMS/3,0,1,123>
Property: <Name:       capabilities, NUMBER:    31.00>
(object end)
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetConnection.Connect.Success>
Property: <Name:        description, STRING:    Connection succeeded.>
Property: <Name:     objectEncoding, NUMBER:    0.00>
(object end)
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <connect>
Invoking releaseStream
Invoking FCPublish
Invoking createStream
RTMP_ClientPacket, received: invoke 29 bytes
(object begin)
Property: <Name:           no-name., STRING:    _result>
Property: <Name:           no-name., NUMBER:    4.00>
Property: NULL
Property: <Name:           no-name., NUMBER:    1.00>
(object end)
HandleInvoke, server invoking <_result>
HandleInvoke, received result for method call <createStream>
Invoking publish
RTMP_ClientPacket, received: invoke 105 bytes
(object begin)
Property: <Name:           no-name., STRING:    onStatus>
Property: <Name:           no-name., NUMBER:    0.00>
Property: NULL
Property: <Name:           no-name., OBJECT>
(object begin)
Property: <Name:              level, STRING:    status>
Property: <Name:               code, STRING:    NetStream.Publish.Start>
Property: <Name:        description, STRING:    Start publishing>
(object end)
(object end)
HandleInvoke, server invoking <onStatus>
HandleInvoke, onStatus: NetStream.Publish.Start
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_qsv))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[graph_1_in_0_1 @ 0x168c680] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3
[h264 @ 0x14b4f40] Reinit context to 1280x720, pix_fmt: yuv420p
[graph 0 input from stream 0:0 @ 0x139b400] w:1280 h:720 pixfmt:yuv420p tb:1/1000 fr:25/1 sar:1/1 sws_param:flags=2
[auto_scaler_0 @ 0x13f6600] w:iw h:ih flags:'bicubic' interl:0
[format @ 0x148d5c0] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format'
[auto_scaler_0 @ 0x13f6600] w:1280 h:720 fmt:yuv420p sar:1/1 -> w:1280 h:720 fmt:nv12 sar:1/1 flags:0x4
[h264_qsv @ 0x13903c0] Initialized an internal MFX session using hardware accelerated implementation
[h264_qsv @ 0x13903c0] Using the VBR with lookahead (LA) ratecontrol method
[h264_qsv @ 0x13903c0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Invoking FCUnpublish
Invoking deleteStream
[aac @ 0x14854c0] Qavg: 62580.895
[aac @ 0x14854c0] 2 frames left in the queue on closing
Invoking deleteStream
Conversion failed!

[lizhong1008]

I've tried to reproduce it on ffmpeg-4.0 branch:
./ffmpeg -loglevel verbose -init_hw_device qsv=hw -i /samba/anonymous/Videos/bbb_sunflower_1080p_30fps_normal.mp4 -c:v h264_qsv -b:v 1000k test.mp4

The transcoding pipeline didn't crash, but reported "Encoder will work with partial HW acceleration, Warning in encoder initialization: partial acceleration (4)". The transcoding speed was very slow (about 20 fps), and it shows the gpu usage is zero when use metrics_monitor to check GPU usage.
It means software path was used instead of GPU accelerated.

[lizhong1008]

More detail:
OS info:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.3.1611 (Core)
Release: 7.3.1611
Codename: Core

MSDK version: MediaServerStudioEssentials2017R3 (API version 1.23)

FFmpeg log (see the attachment)

Check GPU usage with metrics_monitor

./metrics_monitor
RENDER usage: 0.00,     VIDEO usage: 0.00,      VIDEO_E usage: 0.00     VIDEO2 usage: 0.00      GT Freq: 300.00

[lizhong1008]

Not sure why it works well on Mark's side as ​https://trac.ffmpeg.org/ticket/7030#comment:10 (Maybe it just broken on Linux but works well on Wondows? I can understand "-init_hw_device" maybe not needed on Windows since the commit 1f26a231bb065276cd80ce02957c759f3197edfa removing ff_qsv_set_display_handle() has no impact on Windows).
But on Linux, passing a decoded frame by software decoder to qsv encoder directly is broken on FFmpeg 4.0 (this is the problem of this ticket). Passing a qsv decoded frame to a software encoder is also broken.(Reproduce: ffmpeg -c:v h264_qsv -i bbb_sunflower_1080p_30fps_normal.h264 -vframes 100 -c:v libx264 test.h264). These paths are supported on ffmpeg-3.4.
On FFmpeg 4.0, looks like we must use "hwupload" or "hwdownload" for these cases.
An example of this issue, an alternative way without palich's patch is using hwupload:
./ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -i input.mp4 -vframes 100 -vf "hwupload=extra_hw_frames=64, format=qsv" -c:v h264_qsv out.mp4