Opened 17 months ago

Last modified 7 months ago

#9603 new defect

Heap corruption using d3d11va hwaccel without pool

Reported by: Adion Owned by:
Priority: important Component: avutil
Version: git-master Keywords: d3d11va crash
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Summary of the bug:
d3d11va_alloc_single() which is used when no pool is used makes a call to wrap_texture_buf()
This in turn writes to frames_hwctx->texture_infos, but this would not be initialized if no pool is used, causing heap corruption and eventually a crash.

Possible solution:
Adding check to wrap_texture_buf seems to work fine to only update texture_infos if it's safe.
if (s->nb_surfaces_used<s->nb_surfaces)

Additional error logging/check might be needed in case pool is used (s->nb_surfaces>0) but I did not check this use case.

Change History (2)

comment:1 by Carl Eugen Hoyos, 7 months ago

Please either provide source that compiles and allows to reproduce the issue or send your patch - made with git format-patch - to the FFmpeg development mailing list.

comment:2 by Carl Eugen Hoyos, 7 months ago

Keywords: crash added
Priority: normalimportant
Note: See TracTickets for help on using tickets.