Can't initialize h264_qsv encoder post Aug 29 2015 - Ivy Bridge CPU

[Andrew Van Til]

Summary of the bug:
How to reproduce:

% ffmpeg -i -c:v h264_qsv -i h264file.mp4 -c:v h264_qsv outputfile.mp4

ffmpeg version
git-master

built on ...
Anything after August 29th

Platform
Windows (built on Linux)

The version of ffmpeg built on August 29th works, after August 30ths does not, on my Ivy Bridge PC (i7-3770S). Post-Aug30 builds do work on my Haswell PC (i7-4770K). Most likely culprit is ​https://github.com/FFmpeg/FFmpeg/commit/67e87f8050cb8cc61e016cb77be137c18653cbfe probably a call that assumes a newer SDK version than is supported on IVB.

Thanks

[Carl Eugen Hoyos]

Please provide your failing command line together with the complete, uncut console output to make this a valid ticket and please verify which change introduced the regression.

[Andrew Van Til]

Screencap added showing cmdline and output from console.

The change I highlighted above was made on the 30th (when it stopped working). Reverting the hs264 qsv encoder to commits before it work on IVB (i.e. the 29th build and the builds using 29th codebase), everything since doesn't. What exactly are you looking for?

[IvUs]

Replying to babgvant:

Please try to set lower features version into the /libavcodec/qsv_internal.h:
change
#define QSV_VERSION_MINOR 9
to
#define QSV_VERSION_MINOR 6
and recompile the ffmpeg.

[Andrew Van Til]

Replying to IvUs:

Replying to babgvant:

Please try to set lover features version into the /libavcodec/qsv_internal.h:
change
#define QSV_VERSION_MINOR 9
to
#define QSV_VERSION_MINOR 6
and recompile the ffmpeg.

Works

[Andrew Van Til]

Do you know when this change will be merged in?

Thanks

[Andrew Van Til]

Hmm... pulled latest, made the change and the old behavior is back (can't init encoder).

[Hendrik]

@IvUs:
Please only close tickets once the corresponding fix has actually been commited to the repository (or its been decided no changes will be made)

[Hendrik]

The commit that increased the value from 1 does not explain why it needs to be increased, in fact, it seems unrelated to me?

See db89f45535aa3e99bceb5f6bf957c90e7ca39841

[Chris]

I am having similar issues:
I can't initialize the h264_qsv encoder with the FFMpeg 2.8 version that just released.
I am building the ffmpeg lib files and then building them into the open opencv_ffmpeg300.dll (openCV 3.0 project sources\3rdparty\ffmpeg) and calling that from my application.
I had to slightly modify the opencv\sources\modules\videoio\src\cap_ffmpeg_impl.hpp
to specify the nv12 codec format, or it wouldn't even try to initialize the codec.
After that change, it get to the initialization function in ffmpeg\libavcodec\qsvenc.c, but MFXVideoENCODE_Init returns error -15 (MFX_ERR_INVALID_VIDEO_PARAM).
At this point, I don't think the problem is on the OpenCV side, but on the FFMpeg side.
I am trying to build a video file with color 640x480 images, so I don't think alignment is an issue.
Updating the minor version as suggested did not fix the issue for me.
Which version of Intel INDE has ffmpeg been tested with and is recommended to install?
Any help would be greatly appreciated.

Best regards.

[IvUs]

Replying to heleppkes:

@IvUs:
Please only close tickets once the corresponding fix has actually been commited to the repository (or its been decided no changes will be made)

Ok.
I believe replacing QSV_VERSION_MINOR can not be the common fix because it just does disable a lot features available on more fresh platforms and more fresh SDK version. I.e. I can not commit it.
The best way should by dynamic features enabling instead pre-configured compilation for different platforms but it is quite big separate task.

[Andrew Van Til]

The driver provides the client SDK version when you create a session. Applications should use this to determine whether to use/configure different attributes on the encoder so modern and older systems can both benefit up to their ability. It shouldn't be too difficult to put in the if/then logic that enables this approach.

[IvUs]

Replying to heleppkes:

The commit that increased the value from 1 does not explain why it needs to be increased, in fact, it seems unrelated to me?

See db89f45535aa3e99bceb5f6bf957c90e7ca39841

The Intel MFX SDK 1.1 (i.e. with MINOR=1) is VERY old. It was announced with Sandy Bridge and does not have many current features like h265 processing etc. Latest Intel SDK has version 1.16. So using 1.9 is some kind of compromise to have rich enough set of necessary features.
In general it is not comfortable to change MINOR depending by hardware and installed driver and each time recompile the ffmpeg, I believe here some more smart tuning necessary.

[Andrew Van Til]

After a certain point (2yrs IIRC) Intel doesn't update the MSDK version in the latest driver. So while they may release a new driver to fix a critical issue for an IVB (which has pretty good h.264 encoding support) they won't update it to the latest MSDK, so it's frozen forever at this level.

I totally agree that flipping the min level recompiling isn't a good option. I think that (like every MSDK supporting app that I'm aware of) ffmpeg should query the MSDK level and selectively enable/configure features based on it (which might be what you're saying as well).

[Hendrik]

The version passed to mfxInit is just the minimal version, not the version the actual runtime will end up being in the end.

So what happens when we request runtime 1.1 on a modern GPU?
From what I can tell, nothing changes at all, since the actual runtime version is still the same.

What changes on an old GPU? Well, I can't be 100% sure about that. Maybe it still fails because unsupported options are set, but from what I can tell only the LookAhead RC mode is "new" enough, and HEVC support. So would it work, and only fail if LA or HEVC is used?

[IvUs]

Replying to heleppkes:

So what happens when we request runtime 1.1 on a modern GPU?

Currently we will do not have HEVC codec an LA modes.

What changes on an old GPU? Well, I can't be 100% sure about that. Maybe it still fails because unsupported options are set, but from what I can tell only the LookAhead RC mode is "new" enough, and HEVC support. So would it work, and only fail if LA or HEVC is used?

I still not tested yet (my own hardware is haswell). But looks like ivy bridge should to work even with build compiled under MINOR=9 if look_ahead=0 specified in command line. If yes than it will bad idea to make look_ahead=1 by default and all that is necessary is only correct default ratecontrol.

[IvUs]

Replying to babgvant:

I totally agree that flipping the min level recompiling isn't a good option. I think that (like every MSDK supporting app that I'm aware of) ffmpeg should query the MSDK level and selectively enable/configure features based on it (which might be what you're saying as well).

Do you have an ability to revert QSV_VERSION_MINOR back to 9 or just take and compile latest ffmpeg sourcedrop but during test explicitly specify look_ahead=0 in the ffmpeg command line? Will such combination work on ivy bridge cpu?

[akae]

Hi,
I'm also getting an error initializing the encoder with last git version. I've tried the modification to #define QSV_VERSION_MINOR to 6 in last, 2.7.1 and 2.7.2 versions and recompiled all of them with no good results.

The error in last version is

[h264_qsv @ 0x249c1e0] mfx init: /dev/dri/renderD128 fd open failed
[h264_qsv @ 0x249c1e0] mfx init: /dev/dri/renderD129 fd open failed
[h264_qsv @ 0x249c1e0] mfx init: /dev/dri/renderD130 fd open failed
libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_35
libva info: va_openDriver() returns 0
[h264_qsv @ 0x249c1e0] mfx initialization: /dev/dri/card0 vaInitialize successful
[h264_qsv @ 0x249c1e0] Initialized an internal MFX session using software implementation
[h264_qsv @ 0x249c1e0] Using the lookahead (LA) ratecontrol method
[h264_qsv @ 0x249c1e0] Error initializing the encoder

I thought I should paste this here because the original error is an attached jpg and it's not properly indexed.

The error I get in modified older versions is a segmentation fault.

[Andrew Van Til]

Replying to heleppkes:

So what happens when we request runtime 1.1 on a modern GPU?
From what I can tell, nothing changes at all, since the actual runtime version is still the same.

I think initializing at 1.1 is the correct approach (it's what I do in my MSDK transcoder). Initializing at this level doesn't limit what the application can support.

After init use m_pmfxSession->QueryVersion(&version) to retrieve the client's level and cascade the encoder config based on that:

if(version.Major >= 1)
{

if(version.Minor >= 8)
{

introduced in API 1.8
m_CodingOptions2.BRefType = MFX_B_REF_UNKNOWN; default is MFX_B_REF_UNKNOWN
m_CodingOptions2.RepeatPPS = MFX_CODINGOPTION_ON;
m_CodingOptions2.LookAheadDS = pInParams->nLADS; MFX_LOOKAHEAD_DS_UNKNOWN default

}

if(version.Minor >= 7)
{

m_CodingOptions2.Trellis = pInParams->nTrellis;
m_CodingOptions2.LookAheadDepth = pInParams->nLAD;

}

...
}

If you try to configure an option that isn't supported by your SDK version the encoding session will fail.

[Hendrik]

I agree with babgvant, thats the only sensible approach. And today the number of conditions would be very small as well, so it shouldn't be much effort.

If noone else is going to build those checks, I might do it myself, but it might take a bit until I get to it.

[IvUs]

Replying to babgvant:

After init use m_pmfxSession->QueryVersion(&version) to retrieve the client's level and cascade the encoder config based on that:

By the way, what exactly version the QueryVersion(&version) does return at the problem system?

[Andrew Van Til]

Replying to IvUs:

Replying to babgvant:

After init use m_pmfxSession->QueryVersion(&version) to retrieve the client's level and cascade the encoder config based on that:

By the way, what exactly version the QueryVersion(&version) does return at the problem system?

1.11

[Hendrik]

We request 1.9, and you have 1.11, and it still doesn't work? How odd.
Maybe some of the features we use are actually higher in the API? LookAhead requires HW support on top of API support - if its the default now, that might be a problem.

How would one detect HW compat with LA at runtime?

[Andrew Van Til]

Replying to heleppkes:

We request 1.9, and you have 1.11, and it still doesn't work? How odd.
Maybe some of the features we use are actually higher in the API? LookAhead requires HW support on top of API support - if its the default now, that might be a problem.

Generally the min version is intended to act just as an application baseline. If you configure features that require a higher version than is available then the encoder will fail to init.

There are two ways I can think of to approach this:

1) Set the min version to 1.1 then use the actual queried version to only configure options that are available at the client's version (pretty easy with if statements and the SDK docs), reporting via the log or console that a feature that was requested could not be enabled if it was explicitly set by the user. This is the approach I use in my app, I think it is the most approachable and allows for better application specific defaults (i.e. overriding MSDK defaults with mine).

2) Determine the min version required based on the features requested by the user (i.e. if they specify lookahead set the minor version to the appropriate level) and allow the session to fail init if it can't be met.

I'm not aware of any applications that use 2.

How would one detect HW compat with LA at runtime?

The queried version + SDK documentation (1.7 for basic LA, 1.8+ for more).

[Hendrik]

LA is a hardware feature, the queried version alone won't help you there.
Your Ivy apparently returns 1.11, but it does not support LA.

So there must be more checks to be done.

[Andrew Van Til]

Do mean whether LA will actually happen or if the encoder will fail to init/encode?

Here's output from my app, which I added some logging to show that it is able to configure LA depth (1.7) & LA downsampling (1.8) on my IVB box.

[output]
qstranscode v1.0.4.1
Intel(R) QuickSync Transcoding With FFMPEG
IPG: Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz
GFX Driver: 10.18.10.4226

Microsoft Windows Server 2012 R2 (build 9600), 64-bit defaulting to DX11
Selected audio track: 1
Selected video track: 0
Selected streams: video = 0 audio = 1
SDK: v1.11 IMPL: HARDWARE <-- reported SDK version
Configure LADS: 2 <-- m_CodingOptions2.LookAheadDS set
Configure LAD: 100 <-- m_CodingOptions2.LookAheadDepth set
[mp4 @ 007032c0] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
Input video: AVC
Output video: AVC
MFX dll: C:\Program Files\Intel\Media SDK\libmfxhw32.dll

Transcoding started
............
Transcoding finished

Common transcoding time is 14.48 sec
MFX session 0 transcoding PASSED:
Processing time: 14.48 sec
Number of processed frames: 11412

All done...
[end output]

I can't say whether it actually does the LA feature inside the MSDK, but it doesn't fail to encode the file.

[Hendrik]

Then why does current ffmpeg master fail for you? We request version 9 and request LA, but something causes it to fail regardless.

[Andrew Van Til]

I will run it down and let you know ASAP.

[Hendrik]

I should get my old IVB dusted off and running again one of these days. Would be a good system to test future changes for compatibility.

[Andrew Van Til]

version testing patch

[Andrew Van Til]

I made a mistake when I was testing earlier and didn't set the rate control to LA on the IVB system. Specifying -look_ahead 0 enables the ffmpeg qsv encoder, I didn't realize it was the default rate control until I looked at the code.

I don't know how to test the hardware for feature support so I asked on the Intel MSDK forum. They are usually pretty good about responding; hopefully it is possible. In the meantime maybe LA shouldn't be the default rate control.

While I was looking at it I made some of the version testing changes mentioned earlier (see attached patch).

[Hendrik]

Thanks for testing, makes sense. Once we know how to test hardware support, I'll clean the patch and get it on the ML and Git. We probably want to keep the #ifdefs to enable building with older MSDK's

[Carl Eugen Hoyos]

The patch contains trailing whitespace that will be refused by our git repository.
Both tools/patcheck and grep ' $' can help identifying the offending lines.

[Andrew Van Til]

Replying to heleppkes:

Thanks for testing, makes sense. Once we know how to test hardware support, I'll clean the patch and get it on the ML and Git. We probably want to keep the #ifdefs to enable building with older MSDK's

Good point. In that case you'll probably want to init the session using a different baseline like 1.3, if the switch to opaque memory is made.

[IvUs]

Let's just disable enabled LA by default and the issue on old CPUs will solved.

[Hendrik]

LA should give improved quality, so if we can figure out how to detect hardware support, turning it on there would be the best choice, wouldn't it?

[IvUs]

Replying to heleppkes:

LA should give improved quality, so if we can figure out how to detect hardware support, turning it on there would be the best choice, wouldn't it?

I believe we can left it to user's choice. Of course LA mode looks better but for this case I believe the compatibility is more important than quality.
I believe it is bad idea to use different default rate control mode depending by hardware because same command line on different hardware will produce different output. It is not good.
And for any case to quality encoding it is necessary to set a rich command line where LA can be activated additionally too.

[Hendrik]

Replying to IvUs:

I believe it is bad idea to use different default rate control mode depending by hardware because same command line on different hardware will produce different output.

Different hardware will produce different output no matter what, as the encoder improved in every generation. So using the best features available is not a bad thing.

And IMHO it would still be nice to somehow offer a "automatic" LA mode, otherwise every GUI on top of ffmpeg encoding would have to implement such a hardware check.

[Andrew Van Til]

MFXVideoENCODE_Query should validate/correct the encoding parameters based on actual hardware (i.e. if you ask for LA it should auto correct it in the output params). I haven't been able to get it to work correctly though so hopefully it's just something stupid that I'm [not] doing.

[Andrew Van Til]

We were able to track down the issue with MFXVideoENCODE_Query. The documentation did not state that you must copy ExtParam before calling Query if they are present.

mfxVideoParam outParam;
MSDK_ZERO_MEMORY(outParam);
outParam.mfx.CodecId = m_mfxEncParams.mfx.CodecId;
if (m_mfxEncParams.NumExtParam > 0)
{

outParam.NumExtParam = m_mfxEncParams.NumExtParam;
outParam.ExtParam = m_mfxEncParams.ExtParam;

}

[Hendrik]

So calling MFXVideoENCODE_Query with in and outparam the same struct is probably not the right solution (like 5d4a3563f23d39338bf6807caaec0e90cce90f3c), huh.

I'll test on my Ivy and see about fixing it if needed.

[Andrew Van Til]

Probably not best to pass q->param to in & out. I noticed a couple other things: MFX_ERR_INVALID_VIDEO_PARAM is a valid return value and doesn't indicate failure (indicates that issues were corrected in out), and q->req (arg passed to QueryIOSurf) doesn't appear to be used later (it suggest how many surfaces should be allocated per function), but I could have missed it.

Also, while Query can correct many things it appears that the rate control method isn't one of them, so that will need to be handled manually. Either by informing the user, or reconfiguring encoding params manually.

[code]

ret = MFXVideoENCODE_Query(q->session, &q->param, &outParam);
if (MFX_ERR_INVALID_VIDEO_PARAM==ret) {

av_log(avctx, AV_LOG_WARNING, "Encoder settings have been corrected\n");

} else if (ret < 0) {

if (q->look_ahead) {

av_log(avctx, AV_LOG_ERROR, "Error querying encoder params. Try adding -look_ahead 0 to the cmdline\n");

} else {

av_log(avctx, AV_LOG_ERROR, "Error querying encoder params.\n");

}
return ff_qsv_error(ret);

}
ret = MFXVideoENCODE_Init(q->session, &outParam);

code

[Hendrik]

It seems like the return value for corrected params would be "MFX_WRN_INCOMPATIBLE_VIDEO_PARAM", at least according to the docs I have.

[Andrew Van Til]

Sorry, you are correct. It should be MFX_WRN_INCOMPATIBLE_VIDEO_PARAM.

Ideally, I think a was/is of the delta should be logged for user edification.

[Hendrik]

A printed delta might only be meaningful if the user has an influence on the value anyway, which they not necessarily do for all. But something to keep in mind, I guess.

In any case, the function can be used to detect support for LA (even if it doesn't auto-correct it), which answers a question from earlier how to do that. Its on my list to make all that smarter and make the auto-mode smart enough to not use LA when not supported. But probably not before next week, got stuff to finish first.

[ervinshiznit]

Any chance this will get fixed? I could really use this feature...

[rkan]

I'd also appreciate something in regarding this one... It's stupid newer hardware and their related SDKs affect older hardware and 'disable' features from them..

Plenty of nice Thinkpads with good HW that could support QSV without problems. With these laptops you'd still appreciate the hardware encoding capabilities and thus extended battery life.

Also.. Many FFmpeg builds that are offered on Windows (OBS and Zeranoe for example) work fine with QSV even on Sandy Bridge.

[Hendrik]

Zeranoe just builds the ffmpeg code. This has likely been fixed in the code sometime last year.

[Zeranoe]

Is this the same issue as ​https://trac.ffmpeg.org/ticket/5324?

That issue was resolved with 5d542936680e1f3b67bd48265fc56c9227436e48

[Dennis E. Mungai]

Concerning platform support for QuickSync: If you're using Intel's Media SDK, they've been known for deprecating older hardware with newer releases.

If you're able, stick to the latest stable version of the SDK that indicates support for your processor generation. For instance, the open source iMSDK and the related intel-media-driver only indicate support for Broadwell, Skylake, ApolloLake and Cannonlake, leaving owners of Kabylake, Coffeelake, etc out of support.

[Balling]

Was this fixed?