HEVC encoder crashes including OS on high performance environment

[Nobody2112]

Hi all,

this is my first bug report, please give feedback if something is missing or not in the correct form.

My problem HEVC and h264 crashes including the OS during encoding on some environments.

Originally the crash was observed using a software (written in C# using Autogen) on a few high performance systems. On other systems the application runs without any problems.

It is also possible to reproduce the problem with an ffmpeg.exe call.

Description:

• Usage version “N-93387-gdef18ac43b”, also other tested
• Usage of the Zerano-“dll” build of ffmpeg, x64
• Error occurs on fast multi core systems under win10 x64
• Access violation R/W crashes windows
• Frame size 720 x 567 (also with 1920 * 1200 and generated source frames reproducible)

After more investigations and many system crashes, I have the following details:

• Access violation read or write in the method call “avcodec_receive_packet“
• Usage of reference counted frames -> crash with active or inactive
• Usage of CPU flags -> also reproducible without any CPU flag
• Reuse of source image frames or reuse of the destination packet -> also with creating a new frame or packet for every call (no reusage of memory, planned memory leek for testing) the crash occurs.
• Tested with source frames from a decoded video and also created frames with test pixel -> Crash with all image sources
• Test environment using one or multiple threads for decoding / encoding -> no effect. Also with encoding and static image the crash occurs
• Settings for thread pool and frame threads set to 1 -> no effect still crash
• Testing different presets -> Using “medium” or higher settings reproduceable within the first 2500 encoded frames. Using “fast” or lower settings the crash rate goes to zero.
• Tested on different systems -> VMs (4 cores, 8 GB Ram, win10) OK; I7 980 OK, X7820 (8 cores + HT, 32 GB, win10) crash; Xeon 5154 (18 cores + HT, 192 GB, wins 2016) crash.

The problem is reproduceable using the following command and an “mp4” source file. Using a “.mpeg” file as source its harder to reproduce, but as described before, the problem is not related to the decoder. It also occurs if no decoder is involved.

"H:\PRG\Projecte\VideoCut\VideoCut 1.0\VideoCut\VideoCut\bin\Debug\ffmpeg" -i "u:\test.mkv" -c:v libx265 -preset medium -x265-params lossless=1 "w:\testh265.mkv" -b:v 1M

It is not easy to get additional information, because typically the OS crashes. Attached the information delivered by the debugger if I get a exception without OS crash.

Exception
„Ausnahme ausgelöst bei 0x00007FFD6FBB7B4C (avcodec-58.dll) in VideoCut.exe: 0xC0000005: Zugriffsverletzung beim Schreiben an Position 0x00007FFD6FA78D2F.“

Call stack

avcodec-58.dll!00007ffd6fbb7b4c() Unbekannt

avcodec-58.dll!00007ffd6fb028ba() Unbekannt
avcodec-58.dll!00007ffd6fb045bf() Unbekannt
avcodec-58.dll!00007ffd6fa7944b() Unbekannt
avcodec-58.dll!00007ffd6fa83ef2() Unbekannt
avcodec-58.dll!00007ffd6fa8253b() Unbekannt
avcodec-58.dll!00007ffd6fa8253b() Unbekannt
avcodec-58.dll!00007ffd6fa8253b() Unbekannt
avcodec-58.dll!00007ffd6fa856d0() Unbekannt
avcodec-58.dll!00007ffd6fa41564() Unbekannt
avcodec-58.dll!00007ffd6fa3f255() Unbekannt
avcodec-58.dll!00007ffd6fb23383() Unbekannt
avcodec-58.dll!00007ffd6fb23739() Unbekannt
avcodec-58.dll!00007ffd6fb24e7a() Unbekannt
kernel32.dll!00007ffdad062774() Unbekannt
ntdll.dll!00007ffdaee60d51() Unbekannt

Disassembly
00007FFD6FBB7ADF jne 00007FFD6FBB7A63
00007FFD6FBB7AE1 ret
00007FFD6FBB7AE2 nop dword ptr [rax+rax]
00007FFD6FBB7AEA nop word ptr [rax+rax]
00007FFD6FBB7AF0 mov r9d,r9d
00007FFD6FBB7AF3 add r9d,r9d
00007FFD6FBB7AF6 lea r11,[rdx+rdx*2]
00007FFD6FBB7AFA lea rax,[r9+r9*2]
00007FFD6FBB7AFE mov r10d,2
00007FFD6FBB7B04 movdqa xmm4,xmmword ptr [7FFD70D5E0C0h]
00007FFD6FBB7B0C movdqa xmm5,xmmword ptr [7FFD70D55380h]
00007FFD6FBB7B14 movq xmm0,mmword ptr [rcx]
00007FFD6FBB7B18 punpcklbw xmm0,xmm4
00007FFD6FBB7B1C pmaddubsw xmm0,xmm5
00007FFD6FBB7B21 movq xmm1,mmword ptr [rcx+rdx]
00007FFD6FBB7B26 punpcklbw xmm1,xmm4
00007FFD6FBB7B2A pmaddubsw xmm1,xmm5
00007FFD6FBB7B2F movq xmm2,mmword ptr [rcx+rdx*2]
00007FFD6FBB7B34 punpcklbw xmm2,xmm4
00007FFD6FBB7B38 pmaddubsw xmm2,xmm5
00007FFD6FBB7B3D movq xmm3,mmword ptr [rcx+r11]
00007FFD6FBB7B43 punpcklbw xmm3,xmm4
00007FFD6FBB7B47 pmaddubsw xmm3,xmm5
00007FFD6FBB7B4C movdqu xmmword ptr [r8],xmm0
00007FFD6FBB7B51 movdqu xmmword ptr [r8+r9],xmm1
00007FFD6FBB7B57 movdqu xmmword ptr [r8+r9*2],xmm2
00007FFD6FBB7B5D movdqu xmmword ptr [r8+rax],xmm3
00007FFD6FBB7B63 lea rcx,[rcx+rdx*4]
00007FFD6FBB7B67 lea r8,[r8+r9*4]
00007FFD6FBB7B6B dec r10d
00007FFD6FBB7B6E jne 00007FFD6FBB7B14
00007FFD6FBB7B70 ret
00007FFD6FBB7B71 nop dword ptr [rax+rax]
00007FFD6FBB7B79 nop dword ptr [rax]
00007FFD6FBB7B80 mov r9d,r9d
00007FFD6FBB7B83 add r9d,r9d
00007FFD6FBB7B86 lea r11,[rdx+rdx*2]
00007FFD6FBB7B8A lea rax,[r9+r9*2]
00007FFD6FBB7B8E mov r10d,1
00007FFD6FBB7B94 movdqa xmm4,xmmword ptr [7FFD70D5E0C0h]
00007FFD6FBB7B9C movdqa xmm5,xmmword ptr [7FFD70D55380h]
00007FFD6FBB7BA4 movq xmm0,mmword ptr [rcx]
00007FFD6FBB7BA8 punpcklbw xmm0,xmm4
00007FFD6FBB7BAC pmaddubsw xmm0,xmm5
00007FFD6FBB7BB1 movq xmm1,mmword ptr [rcx+rdx]
00007FFD6FBB7BB6 punpcklbw xmm1,xmm4
00007FFD6FBB7BBA pmaddubsw xmm1,xmm5
00007FFD6FBB7BBF movq xmm2,mmword ptr [rcx+rdx*2]
00007FFD6FBB7BC4 punpcklbw xmm2,xmm4
00007FFD6FBB7BC8 pmaddubsw xmm2,xmm5
00007FFD6FBB7BCD movq xmm3,mmword ptr [rcx+r11]
00007FFD6FBB7BD3 punpcklbw xmm3,xmm4
00007FFD6FBB7BD7 pmaddubsw xmm3,xmm5
00007FFD6FBB7BDC movdqu xmmword ptr [r8],xmm0
00007FFD6FBB7BE1 movdqu xmmword ptr [r8+r9],xmm1
00007FFD6FBB7BE7 movdqu xmmword ptr [r8+r9*2],xmm2
00007FFD6FBB7BED movdqu xmmword ptr [r8+rax],xmm3
00007FFD6FBB7BF3 lea rcx,[rcx+rdx*4]
00007FFD6FBB7BF7 lea r8,[r8+r9*4]
00007FFD6FBB7BFB dec r10d
00007FFD6FBB7BFE jne 00007FFD6FBB7BA4
00007FFD6FBB7C00 ret
00007FFD6FBB7C01 nop dword ptr [rax+rax]
00007FFD6FBB7C09 nop dword ptr [rax]
00007FFD6FBB7C10 mov r9d,r9d
00007FFD6FBB7C13 add r9d,r9d
00007FFD6FBB7C16 lea r11,[rdx+rdx*2]

X265 lib info
x265 [info]: HEVC encoder version 3.0_Au+10-74a8672ea4f7
x265 [info]: build info [Windows][GCC 8.2.1][64 bit] 8bit+10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 1 / wpp(9 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 3 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1000 kbps / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 rskip signhide tmvp strong-intra-smoothing
x265 [info]: tools: deblock sao

[Carl Eugen Hoyos]

To make this a valid ticket please test current FFmpeg git head and provide the (simplified) command line you tested together with the complete, uncut console output. Please confirm if a specific input file is needed to reproduce or if the crash is also reproducible with -f lavfi -i testsrc2.

[Cigaes]

If the OS crashes, then the OS is at fault. Report it to the vendor (or sell it on the dark web). When it is fixed then we can try to see if there is something faulty in FFmpeg or x265.

[Nobody2112]

Console log generic test

[Nobody2112]

Thanks for the fast response.

As reported in the initial ticket. The problem is independent of the source video file and also the ffmpeg.exe. In my application I use API calls for my application not the exe with parameters.

I have created a API call based test using generated YUV frames and no thread functionality from my site, which has no decoding functionality included an show the same result.

The I tried if the problem also occurs using the “EXE” with parameters. And it exists also with the exe, but only on very fast systems.

ffmpeg -f lavfi -i testsrc2=duration=300:rate=25:size=pal -pix_fmt yuv420p -c:v libx265 -preset medium -x265-params lossless=1 "w:\testh265.mkv" -b:v 1M >>w:\log.txt 2>&1

If I use fullHD (hd1080) it not occurs.

I’m not able to build FFMpeg by myself. I downloaded the last build from Zeranoe (today), for the attached tests. The included x265 part is the same I used for my test before.
Due to the fact, that the complete OS crashes when the error occurs, I was not able to get a full console log. I add what I have.

To the response (Cigaes), I should talk to the OS vendor if it crashes.
As reported there is a reproducible access valuation of ffmpeg, writing in memory areas not allocated by ffmpeg or it’s processes. Why is the following crash a problem of the OS vendor?

[Cigaes]

A modern operating system is not supposed to crash on software problem. It may crash on severe hardware malfunction of course. But if any application running in the OS does something invalid, the OS is supposed to stand it: it kills the application and everything else goes on.

If your OS does not behave that way, your OS is at fault.