Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#6859 closed defect (needs_more_info)

2pass moov not written and block ffmpeg if RTMP input stream stops before

Reported by: zermok Owned by:
Priority: important Component: undetermined
Version: git-master Keywords: librtmp mov regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
there is no way to get a valid MP4 file if the input rtmp live stream stops before ffmpeg, resulting 100% cpu and no way other than Ctr+C to exit ffmpeg but with a corrupted file as result since moov portion is not written as the second pass

How to reproduce:

% /usr/local/bin/ffmpeg -fflags +genpts -analyzeduration 0 -r 10 -threads 0 -rtmp_conn 'S:pass Z: Z: Z: S:ffmpeg Z: Z: Z: Z: Z:' -i 'rtmp://xxxx:1935/app/instance/mp4:livestreamm live=1 swfVfy=1 swfUrl=https://xxx/swf flashVer=11,2,0,0 buffer=9 timeout=30' -map 0 -c:a libfdk_aac -ar 16000 -c:v libopenh264 -g 10 -allow_skip_frames 1 -profile:v baseline -q:v 1 -pix_fmt yuv420p -async 1 -vsync 1 -vf 'scale=w=1920:h=1200:force_original_aspect_ratio=decrease' -movflags +faststart -y '/out.mp4'

ffmpeg version N-89190-g6db511a

built on Linux 4.13.13-rt5 #1 SMP PREEMPT x86_64 GNU/Linux Fedora 25

Attachments (2)

output.txt (13.1 KB ) - added by zermok 6 years ago.
consoleLog-3.4.txt (5.9 KB ) - added by zermok 6 years ago.
good encdoing log with version 3.4 release

Download all attachments as: .zip

Change History (88)

comment:1 by Carl Eugen Hoyos, 6 years ago

Priority: importantnormal

Please provide a simplified (!) command line that allows to reproduce the issue together with the complete, uncut console output to make this a valid ticket.

comment:2 by zermok, 6 years ago

ffmpeg version N-89190-g6db511a Copyright (c) 2000-2017 the FFmpeg developers

built with gcc 6.4.1 (GCC) 20170727 (Red Hat 6.4.1-1)
configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-libspeex --enable-librtmp --enable-libfreetype --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libfdk-aac --enable-libopenh264 --enable-libcaca --enable-avresample --enable-libgsm --enable-thumb --enable-libfontconfig --enable-static --disable-shared --enable-nvenc --enable-vaapi --enable-opengl --cpu=native --enable-vdpau --enable-nvdec --enable-libvpx --enable-cuvid
libavutil 56. 0.100 / 56. 0.100
libavcodec 58. 3.105 / 58. 3.105
libavformat 58. 2.102 / 58. 2.102
libavdevice 58. 0.100 / 58. 0.100
libavfilter 7. 2.100 / 7. 2.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 0.101 / 5. 0.101
libswresample 3. 0.101 / 3. 0.101
libpostproc 55. 0.100 / 55. 0.100

comment:3 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: newclosed

Please reopen this ticket if you can provide a simplified command line that allows to reproduce the issue you see together with the complete, uncut console output.

comment:4 by zermok, 6 years ago

Hi,

% /usr/local/bin/ffmpeg - i'rtmp://xxxx:1935/app/instance/mp4:livestreamm live=1 swfVfy=1 swfUrl=https://xxx/swf flashVer=11,2,0,0 buffer=9 timeout=30' -map 0 -movflags +faststart -y '/out.mp4'

I had to regress to ffmpeg 3.4 release to make it work again.
the actual git version just blocks ffmpeg and does not write the 2pass where moov block is written in case of rtmp live interruption.

comment:5 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened

comment:6 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: reopenedclosed

Please reopen this ticket if you can provide the command line that allows to reproduce the issue you see together with the complete, uncut console output.

comment:7 by zermok, 6 years ago

I provided the command line already
/usr/local/bin/ffmpeg - i'rtmp://xxxx:1935/app/instance/mp4:livestreamm live=1 swfVfy=1 swfUrl=​https://xxx/swf flashVer=11,2,0,0 buffer=9 timeout=30' -map 0 -movflags +faststart -y '/out.mp4'
for now I cannot reinstall the git version since my server is in production mode only.
what I can say is if a live rtmp stream interrupts so ffmpeg stalled at the end,
showing after the time out RTMP HEADER blablabla in red and no 2pass moov is written at all, ffmpeg just stalls for ever, making 100% cpu btw
what else do you need?

comment:8 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened

comment:9 by Timo R., 6 years ago

An actually usable commandline that does not rely on a server called "xxxx", ideally does not rely on a server at all and is as minimal as possible.
And the full uncut output, which you have been asked to provide 3 times already.

in reply to:  7 comment:10 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: reopenedclosed

Replying to zermok:

for now I cannot reinstall the git version since my server is in production mode only.

It is not necessary to install FFmpeg to be able to test and you don't have to use your production server for testing, any system is sufficient.

comment:11 by zermok, 6 years ago

"xxx" is an example....
provide the command line to get the full uncut console please

comment:12 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened

comment:13 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: reopenedclosed

by zermok, 6 years ago

Attachment: output.txt added

comment:14 by zermok, 6 years ago

Here the uncut console log..
so it blocks at "Cannot READ RTMP HEADER"
and ffmpeg goes to 100% cpu orphan
stop ffmpeg encoding while the live streaming is still running is ok
it happens only when the live stream stops before ffmpeg.
This issue is not noticeable on release version 3.4.
If you need an rtmp live stream so I can create one permanent for your tests

Last edited 6 years ago by zermok (previous) (diff)

comment:15 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened

comment:16 by Carl Eugen Hoyos, 6 years ago

What happens if you press Ctrl-c?

comment:17 by zermok, 6 years ago

it exits ffmpeg but does not write the 2pass moov block
and result in a corrupted mp4 file

comment:18 by Carl Eugen Hoyos, 6 years ago

Could you show the console output in this case?
I did not succeed trying to write an invalid file pressing Ctrl-c.

comment:19 by zermok, 6 years ago

It's the file I already uploaded above
output.txt

Last edited 6 years ago by zermok (previous) (diff)

comment:20 by zermok, 6 years ago

to reproduce it:
encode an rtmp live stream
stop the live stream
then good ffmpeg version use the rtmp timeout
and write 2pass and exit
the actual git blocks on the red message "CANNOT ReAD RTMP HeADER"
raise to 100% cpu and never quit.
making Ctrl+C does nothing, it has to do Ctrl+C TWICE
to quit ffmpeg but resulting as a corrupted file

comment:21 by Carl Eugen Hoyos, 6 years ago

Do you see the same issue when you compile without --enable-librtmp and test the native implementation?

comment:22 by zermok, 6 years ago

dunno, I'm going to try it now and will tell you asap...

comment:23 by zermok, 6 years ago

I get an error
[rtmp @ 0x2771b80] Detected librtmp style URL parameters, these aren't supported by the libavformat internal RTMP handler currently enabled. See the documentation for the correct way to pass parameters. - [rtmp @ 0x2771b80] Unknown connect error (unsupported authentication method?) - [rtmp @ 0x2771b80] Server error: Connection failed

I need to pass connection paramaters otherwise I cannot connect to my server
for now I use -rtmp_conn, what's the equivalent for the native ffmpeg librtmp?

comment:24 by zermok, 6 years ago

anyhow, if it was librtmp so it also won't work with ffmpeg 3.4 release...
I suggest to take a look at the differences between the git source and the 3.4 release.

Last edited 6 years ago by zermok (previous) (diff)

comment:25 by Carl Eugen Hoyos, 6 years ago

Please run git bisect to find the change introducing the issue.

comment:26 by zermok, 6 years ago

into the last git or the release 3.4?
sorry I never used how to use git bisect and what to do with

Last edited 6 years ago by zermok (previous) (diff)

comment:27 by Carl Eugen Hoyos, 6 years ago

To find out which commit introduced the regression you are reporting, you need to compile yourself and use git bisect.

comment:28 by zermok, 6 years ago

ok I did
git bisect start
git bisect bad
git bisect good 3.4

but
git bisect good 3.4
fatal: Needed a single revision
Bad rev input: 3.4

Last edited 6 years ago by zermok (previous) (diff)

comment:29 by Carl Eugen Hoyos, 6 years ago

Use this version instead of 3.4: 80154b1b

comment:30 by zermok, 6 years ago

ok thanks I started to test it now step by step

comment:31 by zermok, 6 years ago

sorry but how to refresh the repository to the right revision
after git bisect good 80154b1b ??
update: ok I made git checkout 80154b1b,
but after compiled the original 3.4 (80154b1b) sources
I did git bisect good
and git jumped to the next commit...
well there is 835 commits since the last git so
it would be better to jump between revision first, how to do that with git bisect?

Last edited 6 years ago by zermok (previous) (diff)

comment:32 by zermok, 6 years ago

ok I got it. don't know why it didn't work before
I did again
git bisect start
git bisect bad
git bisect good 80154b1b
and then
git bisect good
so it jumped to the next step

comment:33 by zermok, 6 years ago

Ok I'm lucky (well at least)
the issue comes just after the 3.4 80154b1b revision

bad one starts at revision 912ceba61b0d45caa8ba8664ddf7b18e2121ddf3

comment:34 by Carl Eugen Hoyos, 6 years ago

To completely rule out any kind of mistake, please provide command and console output for the working version bb4c9d0a

in reply to:  32 comment:35 by Carl Eugen Hoyos, 6 years ago

Replying to zermok:

git bisect start
git bisect bad

In my (very long) experience, this is not a good idea:
You should first compile current FFmpeg, confirm that your newly built binary allows to reproduce the issue you want to bisect, and only then run git bisect bad.

git bisect good 80154b1b

Then you should compile and test the old version, only after you have confirmed that it is a good version, run git bisect good.

So in your case, try something like:

$ make distclean
$ git bisect reset
$ git checkout master
$ ./configure && make ffmpeg
Test and verify a fail
$ make distclean && git bisect bad
$ git checkout 80154b1b
$ ./configure && make ffmpeg
Test and verify a success
$ make distclean && git bisect good

and so on. If the first of above tests does not fail or if the second does not succeed, you verified that there is another issue.

in reply to:  33 comment:36 by Carl Eugen Hoyos, 6 years ago

Replying to zermok:

bad one starts at revision 912ceba61b0d45caa8ba8664ddf7b18e2121ddf3

To elaborate why I am quite certain this is not the relevant commit:
It is supposed to only change hardware decoding, even if something went wrong, it only changes vc1 decoding, which is not used by your command line.

comment:37 by zermok, 6 years ago

It's what I noticed Cehoyos,
just compiled the next revision after the 3.4 release
and the issue happened.... don't know what to say...
I'm going to do what you said above again

Last edited 6 years ago by zermok (previous) (diff)

in reply to:  37 comment:38 by Carl Eugen Hoyos, 6 years ago

Replying to zermok:

It's what I noticed Cehoyos,
just compiled the next revision after the 3.4 release

Not sure what you mean with "next revision": You have to test until git bisect tells you the change introducing the regression.

comment:39 by zermok, 6 years ago

I confirm that the last good one was the version 3.4
version 80154b1b3a0a9e83a9cbaff8b922440f60998128
I made git log to the first bad after this revision
maybe it helps

comment:40 by Carl Eugen Hoyos, 6 years ago

Then please also confirm that 3.4 works for you (and provide the command line including console output): http://ffmpeg.org/releases/ffmpeg-3.4.tar.bz2

comment:41 by zermok, 6 years ago

sorry the log of the bad revision is 28MB, but I guess you already have it.
yes, the version 3.4 works perfectly, I use it for now in production mode
as I said the issue is if the rtmp live stream stops while ffmpeg is encoding
so ffmpeg wait until the rtmp timeout and block for ever with 100% cpu after the last red message "Cannot read RTMP header".
the version 3.4 release just write the 2pass moov and quit correctly.
I attached the full console log of the version 3.4

by zermok, 6 years ago

Attachment: consoleLog-3.4.txt added

good encdoing log with version 3.4 release

comment:42 by Carl Eugen Hoyos, 6 years ago

And 7bec3f78 also works fine? (Sorry, but again, you point to a commit that is an unlikely candidate.)

comment:43 by zermok, 6 years ago

ok I try it now....

(Sorry, but again, you point to a commit that is an unlikely candidate.)

sorry I cannot help you ;), maybe a typo or a mistake happened.. who knows

comment:44 by zermok, 6 years ago

7bec3f78 looks good, no issue...

comment:45 by Carl Eugen Hoyos, 6 years ago

Keywords: librtmp mov regression added
Priority: normalimportant
Status: reopenedopen

Sorry, I was hoping the resulting commit would show something useful.

comment:46 by zermok, 6 years ago

bah, me too :)

comment:47 by Carl Eugen Hoyos, 6 years ago

If you want to help, you can find out which of the changes in 80154b1b introduced the regression.

comment:48 by zermok, 6 years ago

well, I'm absolutely not a C programmer, I'm at work for now too...
what I suspect to see and where? which file?
also must I compare 80154b1b with the next revision?

Last edited 6 years ago by zermok (previous) (diff)

comment:49 by Carl Eugen Hoyos, 6 years ago

If you click on the link that you posted in comment:39 you can see the nine trivial changes. Test if reverting some of them makes the bug go away.

comment:50 by zermok, 6 years ago

but... it only shows headers changed on this commit.
does it mean thant maybe the next version of these libraries
introduced the issue?

comment:51 by zermok, 6 years ago

ok I started to modify every change

comment:52 by zermok, 6 years ago

I tested all 9 successfully, no issue when I regressed the minor versions :(
to be sure I recreated a new last git clone but the issue is there....
What a frustration :((((

comment:53 by zermok, 6 years ago

a top -c shows
28392 nobody 20 0 1502412 91980 19280 R 100.0 0.3 2:35.52 /usr/local/bin/ffmpeg -fflags +genpts -analyzeduration 0 -r 10 -threads 0 -rtmp_conn S:pass Z: Z: Z:....

100% cpu for ever and no second pass to write the moov container....

in reply to:  52 comment:54 by Carl Eugen Hoyos, 6 years ago

Replying to zermok:

I tested all 9 successfully, no issue when I regressed the minor versions :(

I don't understand: Which unchanged version did you test first?
And what did you change and what was the effect?

comment:55 by zermok, 6 years ago

I regressed the minor versions of all the 9 changes made on this revision

comment:56 by zermok, 6 years ago

anyhow I restarted a bisect again...

comment:57 by zermok, 6 years ago

f...ck, same results... but I confused myself, it's the second revision after the 80154b1b
not the first

comment:59 by zermok, 6 years ago

don't know what number is a good reference
0f5ad12ba2b538cb329c507ecc914e06bfa70194

comment:60 by zermok, 6 years ago

[root@node228 ffmpeg]# git bisect log
git bisect start
# bad: [a0e4c41d086bbc32dfefef0b81ed0f59fe04d4ab] avfilter/vf_pseudocolor: add support for more formats
git bisect bad a0e4c41d086bbc32dfefef0b81ed0f59fe04d4ab
# good: [80154b1b3a0a9e83a9cbaff8b922440f60998128] Bump version for master after 3.4 branchpoint
git bisect good 80154b1b3a0a9e83a9cbaff8b922440f60998128
# bad: [ec703910af668b8deb8432d2953222385b821b2c] Merge commit '0f5ad12ba2b538cb329c507ecc914e06bfa70194'
git bisect bad ec703910af668b8deb8432d2953222385b821b2c

comment:61 by Carl Eugen Hoyos, 6 years ago

You have to continue building and testing until git bisect tells you that the offending commit has been found.

comment:62 by zermok, 6 years ago

yes I got it thanks, is this info enough for you to correct the bug?

Last edited 6 years ago by zermok (previous) (diff)

comment:63 by Carl Eugen Hoyos, 6 years ago

I believe you have never finished running git bisect, that's at least what your git bisect log output indicates: It should be longer.

comment:64 by zermok, 6 years ago

ha ok, until the last git I must do it?
but if one of the revision becomes bad, so it will be bad until the end right?

Last edited 6 years ago by zermok (previous) (diff)

in reply to:  64 comment:65 by Carl Eugen Hoyos, 6 years ago

Replying to zermok:

ha ok, until the last git I must do it?

Until git bisect tells you that the offending commit was found.

but if one of the revision becomes bad, so it will be bad until the end right?

No, I don't think so.

comment:66 by zermok, 6 years ago

ok so I should continue to do git bisect good/bad until the current git right?

comment:67 by Carl Eugen Hoyos, 6 years ago

No, only until git bisect tells you that the offending commit was found.

comment:68 by zermok, 6 years ago

ok... I'm full of work now, will do when time allows me..

comment:69 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: openclosed

Please reopen this ticket if you can point us to the commit introducing the regression.

comment:70 by zermok, 6 years ago

ok will do thanks

comment:71 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened
Summary: 2pass moov not written and block ffmpeg if input live stream stops before2pass moov not written and block ffmpeg if RTMP input stream stops before

on month after I recompiled the today last git (January 25th)
and the issue persists, something changed on ffmpeg source after the version 3.4.x
since this version is working as expected.
I'm sorry to not have time to make the tests you asked (my free time is very short, work, family, kids etc...) So the only offer I can do is you can have access to my server and so make your own tests.
Thanks

comment:72 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: reopenedclosed

Please reopen this ticket if you can point us to the commit introducing the regression or provide a command line that I can use to reproduce and run git bisect.

comment:73 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened

Hi, sorry I'm not enough familiar with git bisect. tried it but very confusing and not clear how to find regression since there are so many changes after the version 3.4.x
as I said above, the simpliest command to reproduce the issue
is to get a live stream as input and convert it in real time in mp4 or whatever.
then disconnect the rtmp live stream while ffmpeg is encoding so you get the issue which
is infinite 100% CPU and no moov write leaving a corrupted mp4 file.

comment:74 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: reopenedclosed

I am willing to help you with git bisect: What did you try and what did not work?

comment:75 by zermok, 6 years ago

I don't understand how git bisect works.
I'm not a pro Compiler, just a simple user and don't make any money with ffmpeg.
so I used my time to report this bug, tried many times git bisect but it's so confusing so it needs a clear tutorial of how to debug ffmpeg with git bisect. I lost too much time already for nothing

in reply to:  75 comment:76 by zermok, 6 years ago

I think the best is I give to you an access to my server so I'm sure you can check it in 5mn and avoid I spend hours for the same result.
As I said the issue is clear and you can reproduce it with any kind of rtmp live stream as input encoding in mp4 as output.
just disconnect the live stream while it's encoding you will reproduce the issue.
only ffmpeg version 3.4.x is ok for now

in reply to:  75 comment:77 by Carl Eugen Hoyos, 6 years ago

Replying to zermok:

I don't understand how git bisect works.

It would be more helpful if you had explained what you tried and what didn't work.
Anyway:
Run git clone to get an FFmpeg directory including git history
Compile FFmpeg and confirm that you can reproduce the issue
Run make distclean && git bisect bad to start bisecting.
Checkout a version that you know works well
Compile and test that version to confirm (I always do this and I believe it is a very good idea)
Run make distclean && git bisect good
Now loop the following: Compile, test and run one of the two lines above: bad if the test fails and good if it succeeds until git tells you which commit introduced the regression.
Usually takes 30 minutes.

comment:78 by zermok, 6 years ago

it takes 30mn for you, not for a newbie like me...
anyhow, ffmpeg version 4.0 still has the same issue.
I'm stuck with 3.4.2 for now....

comment:79 by zermok, 6 years ago

Resolution: needs_more_info
Status: closedreopened

comment:80 by Carl Eugen Hoyos, 6 years ago

Resolution: needs_more_info
Status: reopenedclosed

Please reopen this ticket if you can point us to the commit introducing the regression or provide a command line that I can use to reproduce and run git bisect.

comment:81 by zermok, 6 years ago

...

Last edited 6 years ago by zermok (previous) (diff)

comment:82 by zermok, 6 years ago

Resolution: needs_more_infofixed

Tested last git today N-91801-gad9b4ecc26
and works back like a charm, I can finally update ffmpeg!! :D
thanks!

comment:83 by Carl Eugen Hoyos, 6 years ago

Resolution: fixedneeds_more_info

comment:84 by zermok, 6 years ago

do you need any from me?

comment:85 by Carl Eugen Hoyos, 6 years ago

I believe it makes sense for all tickets on this bug tracker to contain every information necessary on how to reproduce an issue, a commit if the issue is a regression and (if applicable) the commit that fixes the issue. If all three are missing (for a fixed regression) like in this case, I don't think the ticket has much value.

comment:86 by zermok, 6 years ago

ok I understand

Note: See TracTickets for help on using tickets.