Opened 12 months ago
Last modified 8 months ago
#7030 open defect
qsv in ffmpeg
Reported by: | palich2000 | Owned by: | |
---|---|---|---|
Priority: | important | Component: | avcodec |
Version: | git-master | Keywords: | qsv regression |
Cc: | zhong.li@intel.com | Blocked By: | |
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
Testing was conducted on the same equipment on the same video stream.
configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-testinclude -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-testlib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-libmp3lame --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --enable-runtime-cpudetect --enable-decklink --enable-libmfx
ffmpeg version 3.2.10
ffmpeg -i rtmp://...
-c:v h264_qsv -b:v 1000k -g 50 -r 25
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
%CPU LOAD (top) 29,7
%GPU LOAD (metrics_monitor)
RENDER usage: 15.15, VIDEO usage: 6.06, VIDEO_E usage: 0.00 GT Freq: 350.00
This works well.
ffmpeg version 3.4.2
add -init_hw_device qsv:hw
ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
%CPU LOAD (top) 36.0
%GPU LOAD (metrics_monitor)
RENDER usage: 14.14, VIDEO usage: 5.05, VIDEO_E usage: 0.00 GT Freq: 350.00
This works well.
ffmpeg version 3.3.6
ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
Unrecognized option 'init_hw_device'.
Without this option, QSV is not initialized.
And gives an error. Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
ffmpeg version N-90085-gacdea9e
ffmpeg -v info -i rtmp://....
-init_hw_device qsv:hw -c:v h264_qsv -b:v 1000k
-strict experimental -acodec aac -ab 64k -ar 44100
-f flv rtmp://127.0.0.1/virtual/1
Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode.
How to make QSV work in these versions ? (3.3.6 and N-90085-gacdea9e)
./sys_analyzer_linux.py -------------------------- Hardware readiness checks: -------------------------- [ OK ] Processor name: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz -------------------------- OS readiness checks: -------------------------- [ OK ] GPU visible to OS -------------------------- Media Server Studio Install: -------------------------- [ OK ] user in video group [ OK ] libva.so.1 found [ OK ] vainfo reports valid codec entry points [ OK ] /dev/dri/renderD128 connects to Intel i915 -------------------------- Component Smoke Tests: -------------------------- [ OK ] Media SDK HW API level:1.19 [ OK ] Media SDK SW API level:1.19 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK
Attachments (1)
Change History (22)
comment:1 Changed 12 months ago by cehoyos
- Keywords qsv regression added
- Version changed from 3.4 to git-master
comment:2 Changed 12 months ago by palich2000
In the b26dd5a works.
But on the dde7b1d does not work anymore.
comment:3 Changed 12 months ago by cehoyos
Please run git bisect to find the change that introduced the regression.
comment:4 Changed 12 months ago by palich2000
git bisect good
b0cd14fb1dab4b044f7fe6b53ac635409849de77 is the first bad commit
commit b0cd14fb1dab4b044f7fe6b53ac635409849de77
Author: Mark Thompson <sw@jkqxz.net>
Date: Thu Oct 26 00:18:47 2017 +0100
ffmpeg: Use codec hardware config to configure hwaccels
Removes specific support for all hwaccels supported by the generic code
(DXVA2, D3D11VA, NVDEC, VAAPI and VDPAU).
:040000 040000 db8084af9c0723e94edafc6d4beb29386dee6288 a1b4cd2717b8f391249024ea335241ac7788f843 M fftools
comment:5 Changed 12 months ago by cehoyos
- Status changed from new to open
comment:6 Changed 12 months ago by palich2000
The author of the commit Mark Thompson <sw@jkqxz.net> had some grandiose idea but he did not realize it to the end. I do not know how the hardware decoders work, but the hardware coders do not work after this commit. The function hw_device_setup_for_encode suggests that there is a hardware configuration for each encoder, but they are not declared. And hardware encoders do not connect to the codec. This should either be removed or completed.
comment:7 Changed 12 months ago by palich2000
This patch solves the problem, but only for h264_qsv encoder.
diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c index 09e4c0e..5f60675 100644 --- a/libavcodec/qsvenc_h264.c +++ b/libavcodec/qsvenc_h264.c @@ -34,6 +34,7 @@ #include "qsv.h" #include "qsv_internal.h" #include "qsvenc.h" +#include "hwaccel.h" typedef struct QSVH264EncContext { AVClass *class; @@ -169,6 +170,16 @@ static const AVCodecDefault qsv_enc_defaults[] = { { NULL }, }; +#define HWACCEL_QSV(codec) \ + HW_CONFIG_HWACCEL(1, 1, 1, QSV, QSV, ff_ ## codec ## _qsv_hwaccel) + +const AVHWAccel ff_h264_qsv_hwaccel = { + .name = "h264_qsv", + .type = AVMEDIA_TYPE_VIDEO, + .id = AV_CODEC_ID_H264, + .pix_fmt = AV_PIX_FMT_QSV +}; + AVCodec ff_h264_qsv_encoder = { .name = "h264_qsv", .long_name = NULL_IF_CONFIG_SMALL("H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration)"), @@ -186,5 +197,9 @@ AVCodec ff_h264_qsv_encoder = { .priv_class = &class, .defaults = qsv_enc_defaults, .caps_internal = FF_CODEC_CAP_INIT_CLEANUP, + .hw_configs = (const AVCodecHWConfigInternal*[]) { + HWACCEL_QSV(h264), + NULL + }, .wrapper_name = "qsv", };
comment:8 Changed 12 months ago by cehoyos
Please send your patch - made with git format-patch - to the FFmpeg development mailing list where it can be reviewed.
comment:9 Changed 12 months ago by palich2000
OK
Sended.
comment:10 Changed 12 months ago by jkqxz
This works fine for me. At a guess you are somehow opening the DRM master device rather than a render node and therefore the unused top-level device has exclusive access to the GPU and stops the encoder from opening it?
If that's the problem then not creating the unused top-level device (remove the -init_hw_device option) should make it work for you. (That device did automatically get passed to an encoder in 3.4, but there was disagreement about that feature being useful because ffmpeg was unable to tell whether the device was required or not (it isn't here, the internal session works for isolated encoders) and offered unhelpful warnings.)
comment:11 Changed 12 months ago by palich2000
Without this parameter, the situation is as follows.
The codec tries to code programmatically and it does not work.
[h264_qsv @ 0x1e87ec0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode. Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height [aac @ 0x1e8a540] Qavg: 62368.684
comment:12 Changed 12 months ago by jkqxz
Then I suggest providing more information about your setup, and a complete log of a failing command.
comment:13 follow-up: ↓ 16 Changed 12 months ago by palich2000
Look above, there I described everything.
comment:14 Changed 12 months ago by heleppkes
The message even has a hint of what to do - try choosing a different rate control mode.
comment:15 Changed 12 months ago by palich2000
Believe me, this will not help.
You can choose the parameters for which there will be no error. But the encoding will occur on the CPU and not on the GPU.
comment:16 in reply to: ↑ 13 Changed 12 months ago by jkqxz
Replying to palich2000:
Look above, there I described everything.
If might help if you could increase the scope of "everything" to include:
- Operating system details, including kernel version.
- What version of the Media SDK you are using (some specific proprietary version, the open-source one?).
- A complete log of ffmpeg output for the failing case.
comment:17 Changed 12 months ago by palich2000
Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[ OK ] Media SDK HW API level:1.19 [ OK ] Media SDK SW API level:1.19 [ OK ] OpenCL check:platform:Intel(R) OpenCL GPU OK CPU OK
ffmpeg -t 30 -v verbose -i rtmp://xxxxx \ -c:v h264_qsv -b:v 1000k\ -strict experimental -acodec aac -ab 64k -ar 44100 \ -f flv rtmp://127.0.0.1/vip/_77_
ffmpeg version N-90198-g7056d06 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-11) configuration: --prefix=/home/palich/ffmpeg-test/ --extra-cflags='-I/home/palich/ffmpeg-test//include -Ofast -march=native' --extra-ldflags='-L/home/palich/ffmpeg-test//lib -ldl' --enable-avresample --enable-static --enable-shared --enable-nonfree --enable-gpl --enable-version3 --enable-libx264 --enable-libfdk-aac --enable-librtmp --enable-gnutls --enable-libfreetype --enable-filters --enable-postproc --disable-doc --enable-libmp3lame --disable-stripping --enable-debug=3 --enable-runtime-cpudetect --enable-libmfx libavutil 56. 8.100 / 56. 8.100 libavcodec 58. 13.100 / 58. 13.100 libavformat 58. 10.100 / 58. 10.100 libavdevice 58. 2.100 / 58. 2.100 libavfilter 7. 12.100 / 7. 12.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 0.101 / 5. 0.101 libswresample 3. 0.101 / 3. 0.101 libpostproc 55. 0.100 / 55. 0.100 Routing option strict to both codec and muxer layer Parsing... Parsed protocol: 0 Parsed host : vr0.dpl-streaming.com Parsed app : virtual RTMP_Connect1, ... connected, handshaking HandShake: Type Answer : 03 HandShake: Server Uptime : 134225664 HandShake: FMS Version : 0.0.0.0 HandShake: Handshaking finished.... RTMP_Connect1, handshaked Invoking connect HandleServerBW: server BW = 5000000 HandleClientBW: client BW = 5000000 2 HandleChangeChunkSize, received: chunk size change to 4096 RTMP_ClientPacket, received: invoke 190 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 1.00> Property: <Name: no-name., OBJECT> (object begin) Property: <Name: fmsVer, STRING: FMS/3,0,1,123> Property: <Name: capabilities, NUMBER: 31.00> (object end) Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetConnection.Connect.Success> Property: <Name: description, STRING: Connection succeeded.> Property: <Name: objectEncoding, NUMBER: 0.00> (object end) (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <connect> sending ctrl. type: 0x0003 Invoking createStream RTMP_ClientPacket, received: invoke 29 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 2.00> Property: NULL Property: <Name: no-name., NUMBER: 1.00> (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <createStream> SendPlay, seekTime=0, stopTime=0, sending play: vrm_foot_live Invoking play sending ctrl. type: 0x0003 HandleCtrl, received ctrl. type: 0, len: 6 HandleCtrl, Stream Begin 1 RTMP_ClientPacket, received: invoke 96 bytes (object begin) Property: <Name: no-name., STRING: onStatus> Property: <Name: no-name., NUMBER: 0.00> Property: NULL Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetStream.Play.Start> Property: <Name: description, STRING: Start live> (object end) (object end) HandleInvoke, server invoking <onStatus> HandleInvoke, onStatus: NetStream.Play.Start RTMP_ClientPacket, received: notify 24 bytes (object begin) Property: <Name: no-name., STRING: |RtmpSampleAccess> Property: <Name: no-name., BOOLEAN: TRUE> Property: <Name: no-name., BOOLEAN: TRUE> (object end) RTMP_ClientPacket, received: invoke 110 bytes (object begin) Property: <Name: no-name., STRING: onStatus> Property: <Name: no-name., NUMBER: 0.00> Property: NULL Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetStream.Play.PublishNotify> Property: <Name: description, STRING: Start publishing> (object end) (object end) HandleInvoke, server invoking <onStatus> HandleInvoke, onStatus: NetStream.Play.PublishNotify RTMP_ClientPacket, received: notify 387 bytes (object begin) Property: <Name: no-name., STRING: onMetaData> Property: <Name: no-name., OBJECT> (object begin) Property: <Name: Server, STRING: NGINX RTMP (github.com/arut/nginx-rtmp-module)> Property: <Name: width, NUMBER: 1280.00> Property: <Name: height, NUMBER: 720.00> Property: <Name: displayWidth, NUMBER: 1280.00> Property: <Name: displayHeight, NUMBER: 720.00> Property: <Name: duration, NUMBER: 0.00> Property: <Name: framerate, NUMBER: 25.00> Property: <Name: fps, NUMBER: 25.00> Property: <Name: videodatarate, NUMBER: 976.00> Property: <Name: videocodecid, NUMBER: 7.00> Property: <Name: audiodatarate, NUMBER: 62.00> Property: <Name: audiocodecid, NUMBER: 10.00> Property: <Name: profile, STRING: > Property: <Name: level, STRING: > (object end) (object end) Metadata: Server NGINX RTMP (github.com/arut/nginx-rtmp-module) width 1280.00 height 720.00 displayWidth 1280.00 displayHeight 720.00 duration 0.00 framerate 25.00 fps 25.00 videodatarate 976.00 videocodecid 7.00 audiodatarate 62.00 audiocodecid 10.00 [h264 @ 0x138b040] Reinit context to 1280x720, pix_fmt: yuv420p Input #0, live_flv, from 'rtmp://xxxxxxxx': Metadata: Server : NGINX RTMP (github.com/arut/nginx-rtmp-module) displayWidth : 1280 displayHeight : 720 fps : 25 profile : level : Duration: 00:00:00.00, start: 100110.018000, bitrate: N/A Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 1280x720 [SAR 1:1 DAR 16:9], 999 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 63 kb/s Parsing... Parsed protocol: 0 Parsed host : 127.0.0.1 Parsed app : vip RTMP_Connect1, ... connected, handshaking HandShake: Type Answer : 03 HandShake: Server Uptime : 134227824 HandShake: FMS Version : 0.0.0.0 HandShake: Handshaking finished.... RTMP_Connect1, handshaked Invoking connect HandleServerBW: server BW = 5000000 HandleClientBW: client BW = 5000000 2 HandleChangeChunkSize, received: chunk size change to 1024 RTMP_ClientPacket, received: invoke 190 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 1.00> Property: <Name: no-name., OBJECT> (object begin) Property: <Name: fmsVer, STRING: FMS/3,0,1,123> Property: <Name: capabilities, NUMBER: 31.00> (object end) Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetConnection.Connect.Success> Property: <Name: description, STRING: Connection succeeded.> Property: <Name: objectEncoding, NUMBER: 0.00> (object end) (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <connect> Invoking releaseStream Invoking FCPublish Invoking createStream RTMP_ClientPacket, received: invoke 29 bytes (object begin) Property: <Name: no-name., STRING: _result> Property: <Name: no-name., NUMBER: 4.00> Property: NULL Property: <Name: no-name., NUMBER: 1.00> (object end) HandleInvoke, server invoking <_result> HandleInvoke, received result for method call <createStream> Invoking publish RTMP_ClientPacket, received: invoke 105 bytes (object begin) Property: <Name: no-name., STRING: onStatus> Property: <Name: no-name., NUMBER: 0.00> Property: NULL Property: <Name: no-name., OBJECT> (object begin) Property: <Name: level, STRING: status> Property: <Name: code, STRING: NetStream.Publish.Start> Property: <Name: description, STRING: Start publishing> (object end) (object end) HandleInvoke, server invoking <onStatus> HandleInvoke, onStatus: NetStream.Publish.Start Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_qsv)) Stream #0:1 -> #0:1 (aac (native) -> aac (native)) Press [q] to stop, [?] for help [graph_1_in_0_1 @ 0x168c680] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3 [h264 @ 0x14b4f40] Reinit context to 1280x720, pix_fmt: yuv420p [graph 0 input from stream 0:0 @ 0x139b400] w:1280 h:720 pixfmt:yuv420p tb:1/1000 fr:25/1 sar:1/1 sws_param:flags=2 [auto_scaler_0 @ 0x13f6600] w:iw h:ih flags:'bicubic' interl:0 [format @ 0x148d5c0] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format' [auto_scaler_0 @ 0x13f6600] w:1280 h:720 fmt:yuv420p sar:1/1 -> w:1280 h:720 fmt:nv12 sar:1/1 flags:0x4 [h264_qsv @ 0x13903c0] Initialized an internal MFX session using hardware accelerated implementation [h264_qsv @ 0x13903c0] Using the VBR with lookahead (LA) ratecontrol method [h264_qsv @ 0x13903c0] Selected ratecontrol mode is not supported by the QSV runtime. Choose a different mode. Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height Invoking FCUnpublish Invoking deleteStream [aac @ 0x14854c0] Qavg: 62580.895 [aac @ 0x14854c0] 2 frames left in the queue on closing Invoking deleteStream Conversion failed!
comment:18 Changed 10 months ago by lizhong1008
I've tried to reproduce it on ffmpeg-4.0 branch:
./ffmpeg -loglevel verbose -init_hw_device qsv=hw -i /samba/anonymous/Videos/bbb_sunflower_1080p_30fps_normal.mp4 -c:v h264_qsv -b:v 1000k test.mp4
The transcoding pipeline didn't crash, but reported "Encoder will work with partial HW acceleration, Warning in encoder initialization: partial acceleration (4)". The transcoding speed was very slow (about 20 fps), and it shows the gpu usage is zero when use metrics_monitor to check GPU usage.
It means software path was used instead of GPU accelerated.
comment:19 Changed 10 months ago by lizhong1008
More detail:
OS info:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.3.1611 (Core)
Release: 7.3.1611
Codename: Core
MSDK version: MediaServerStudioEssentials2017R3 (API version 1.23)
FFmpeg log (see the attachment)
Check GPU usage with metrics_monitor
./metrics_monitor RENDER usage: 0.00, VIDEO usage: 0.00, VIDEO_E usage: 0.00 VIDEO2 usage: 0.00 GT Freq: 300.00
Changed 10 months ago by lizhong1008
comment:20 Changed 10 months ago by lizhong1008
- Cc zhong.li@intel.com added
comment:21 Changed 8 months ago by lizhong1008
Not sure why it works well on Mark's side as https://trac.ffmpeg.org/ticket/7030#comment:10 (Maybe it just broken on Linux but works well on Wondows? I can understand "-init_hw_device" maybe not needed on Windows since the commit 1f26a231bb065276cd80ce02957c759f3197edfa removing ff_qsv_set_display_handle() has no impact on Windows).
But on Linux, passing a decoded frame by software decoder to qsv encoder directly is broken on FFmpeg 4.0 (this is the problem of this ticket). Passing a qsv decoded frame to a software encoder is also broken.(Reproduce: ffmpeg -c:v h264_qsv -i bbb_sunflower_1080p_30fps_normal.h264 -vframes 100 -c:v libx264 test.h264). These paths are supported on ffmpeg-3.4.
On FFmpeg 4.0, looks like we must use "hwupload" or "hwdownload" for these cases.
An example of this issue, an alternative way without palich's patch is using hwupload:
./ffmpeg -init_hw_device qsv=hw -filter_hw_device hw -i input.mp4 -vframes 100 -vf "hwupload=extra_hw_frames=64, format=qsv" -c:v h264_qsv out.mp4
Please find the commit that introduced the regression.