Opened 17 months ago

Last modified 10 months ago

#10111 new defect

Memory leak with vaapi encoder

Reported by: Steve Browne Owned by:
Priority: normal Component: avcodec
Version: 4.4.3 Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by Steve Browne)

Summary of the bug:
Calling avcodec_send_frame multiple times in a row without an avcodec_receive_packet in between with an AVFrame that does not store data in data[0] will result in a memory leak. The example where I ran into this was with VAAPI. This could potentially apply to other hardware encoders that don't store anything in data[0] as well though.

The problem lies in encode_send_frame_internal because it uses dst->data[0] to determine if avci->buffer_frame is already pointing to something. Since vaapi contexts only store the surface ID in data[3] it will gladly think buffer_frame is unused and overwrite it without actually encoding it or freeing it. I don't know if all contexts should just store something in data[0] or if that check something else like buf[0], but it's very easy for this to cause a memory leak.

The simple workaround for this is to just set data[0] = data[3] for vaapi.

How to reproduce:
Call avcodec_send_frame multiple times in a row without avcodec_receive_packet.

It looks like this was actually fixed with 67aceaf4ad641a4d34c3ec70b532efdc60483e3d, but I'm using an older version. It would probably be nice if this were backpatched to avoid others running into this.

Feel free to close this, but I just wanted to write something for it because I couldn't find anything indicating a memory leak with vaapi encoding when searching and there's nothing in the changelog that would suggest that resolves a memory leak either.

Change History (1)

comment:1 by Steve Browne, 10 months ago

Description: modified (diff)
Note: See TracTickets for help on using tickets.