Opened 8 years ago

Closed 8 years ago

#4989 closed defect (invalid)

avfilter_graph_create_filter() eventually hits pthread::EAGAIN on multi-threading

Reported by: czhang384 Owned by:
Priority: normal Component: undetermined
Version: unspecified Keywords:
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

I was trying to convert OPUS to PCM format for my project. And I modified transcoding.c example from the source.

I did use one other library as well but chances for its memory problems are really low, so I am guessing some APIs in transcoding.c example may cause memory problem?

There was no specific way to reproduce my problem: my program uses ffmpeg to transcode one file, and then passes trancoded file/packet to the other library.

I create an infinite loop to test its limit, and it ranges 300~1500 loops of reading one single file (every loop will have 2 ffmpeg to read the same file). But the error is consistent: [AVFilterGraph @ 0x12171640] Error initializing threading. I check with source code, and it gives pthread::EAGAIN, which means the pthread hits the limit.

Here are some of my investigations so far:

  1. avfilter_graph_create_filter() will call thread_init_internal() internally, and then it creats 25 pthreads each. This means I need to limit # of threads. I use 4 threads, every thread uses 2 ffmpeg to loop the file, so it is 25*4*3 = 200 threads. Given original 4 threads, it expects [4, 204] active threads at any time. I tested it, and it number was correct.
  2. I checked memory with valgrind: by running 1-5 thread and loops few times. The consistent memory problem is "40 bytes still reachable". From my research, it is ffmpeg ticket: https://trac.ffmpeg.org/ticket/3194, where one internal mutex lock is not freed and occupies that 40 bytes memory.
  3. I run only 1 thread. And so far it has been loop ~1900 times and looks alright.

So I am wandering whether any APIs in trancoding.c may allocate some secret memory in multi-threading. If not, I guess I can check other non-ffmpeg-related sections on it.

Change History (2)

comment:1 by czhang384, 8 years ago

There is some implementation I need to mention with respect to this:

I wrap transcoding.c to a c++ class Transcoder. And to support 2 ffmpeg, I use map<int, Transcoder> to get/set Transcoder.
I notice there are some multi-thread problems for ffmpeg in other posts: http://libav-users.943685.n4.nabble.com/Threads-safe-issue-with-ffmpeg-c-td945683.html.

So I create separate map<int, Transcoder> object on every single thread, but potentially I guess these two transcoders may leave some small memory sometimes in multi-thread?

The convertion is correct, but it is just the memory problem I have no clue about.

comment:2 by czhang384, 8 years ago

Resolution: invalid
Status: newclosed

I guess I finger out the reason. It is exactly for my investigation 1: 25 threads initialized for thread_init_internal().

When I use 4 threads and each threads call 2 times, sooner or later the threads uses up the resources (mine just takes potential hours). I notice there is lock for thread-operation overall. So I guess some time the lock is released to create but not destroy, so that is why it hits the limit and crash.

Leave my comment here in case there is future similar problem here.

Note: See TracTickets for help on using tickets.