Opened 3 weeks ago

Last modified 36 hours ago

#6859 open defect

2pass moov not written and block ffmpeg if input live stream stops before

Reported by: zermok Owned by:
Priority: important Component: undetermined
Version: git-master Keywords: librtmp mov regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
there is no way to get a valid MP4 file if the input rtmp live stream stops before ffmpeg, resulting 100% cpu and no way other than Ctr+C to exit ffmpeg but with a corrupted file as result since moov portion is not written as the second pass

How to reproduce:

% /usr/local/bin/ffmpeg -fflags +genpts -analyzeduration 0 -r 10 -threads 0 -rtmp_conn 'S:pass Z: Z: Z: S:ffmpeg Z: Z: Z: Z: Z:' -i 'rtmp://xxxx:1935/app/instance/mp4:livestreamm live=1 swfVfy=1 swfUrl=https://xxx/swf flashVer=11,2,0,0 buffer=9 timeout=30' -map 0 -c:a libfdk_aac -ar 16000 -c:v libopenh264 -g 10 -allow_skip_frames 1 -profile:v baseline -q:v 1 -pix_fmt yuv420p -async 1 -vsync 1 -vf 'scale=w=1920:h=1200:force_original_aspect_ratio=decrease' -movflags +faststart -y '/out.mp4'

ffmpeg version N-89190-g6db511a

built on Linux 4.13.13-rt5 #1 SMP PREEMPT x86_64 GNU/Linux Fedora 25

Attachments (2)

output.txt (13.1 KB) - added by zermok 2 weeks ago.
consoleLog-3.4.txt (5.9 KB) - added by zermok 3 days ago.
good encdoing log with version 3.4 release

Download all attachments as: .zip

Change History (70)

comment:1 Changed 3 weeks ago by cehoyos

  • Priority changed from important to normal

Please provide a simplified (!) command line that allows to reproduce the issue together with the complete, uncut console output to make this a valid ticket.

comment:2 Changed 3 weeks ago by zermok

ffmpeg version N-89190-g6db511a Copyright (c) 2000-2017 the FFmpeg developers

built with gcc 6.4.1 (GCC) 20170727 (Red Hat 6.4.1-1)
configuration: --enable-gpl --enable-version3 --enable-nonfree --enable-libspeex --enable-librtmp --enable-libfreetype --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libfdk-aac --enable-libopenh264 --enable-libcaca --enable-avresample --enable-libgsm --enable-thumb --enable-libfontconfig --enable-static --disable-shared --enable-nvenc --enable-vaapi --enable-opengl --cpu=native --enable-vdpau --enable-nvdec --enable-libvpx --enable-cuvid
libavutil 56. 0.100 / 56. 0.100
libavcodec 58. 3.105 / 58. 3.105
libavformat 58. 2.102 / 58. 2.102
libavdevice 58. 0.100 / 58. 0.100
libavfilter 7. 2.100 / 7. 2.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 0.101 / 5. 0.101
libswresample 3. 0.101 / 3. 0.101
libpostproc 55. 0.100 / 55. 0.100

comment:3 Changed 3 weeks ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from new to closed

Please reopen this ticket if you can provide a simplified command line that allows to reproduce the issue you see together with the complete, uncut console output.

comment:4 Changed 3 weeks ago by zermok

Hi,

% /usr/local/bin/ffmpeg - i'rtmp://xxxx:1935/app/instance/mp4:livestreamm live=1 swfVfy=1 swfUrl=https://xxx/swf flashVer=11,2,0,0 buffer=9 timeout=30' -map 0 -movflags +faststart -y '/out.mp4'

I had to regress to ffmpeg 3.4 release to make it work again.
the actual git version just blocks ffmpeg and does not write the 2pass where moov block is written in case of rtmp live interruption.

comment:5 Changed 3 weeks ago by zermok

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

comment:6 Changed 3 weeks ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from reopened to closed

Please reopen this ticket if you can provide the command line that allows to reproduce the issue you see together with the complete, uncut console output.

comment:7 follow-up: Changed 2 weeks ago by zermok

I provided the command line already
/usr/local/bin/ffmpeg - i'rtmp://xxxx:1935/app/instance/mp4:livestreamm live=1 swfVfy=1 swfUrl=​https://xxx/swf flashVer=11,2,0,0 buffer=9 timeout=30' -map 0 -movflags +faststart -y '/out.mp4'
for now I cannot reinstall the git version since my server is in production mode only.
what I can say is if a live rtmp stream interrupts so ffmpeg stalled at the end,
showing after the time out RTMP HEADER blablabla in red and no 2pass moov is written at all, ffmpeg just stalls for ever, making 100% cpu btw
what else do you need?

comment:8 Changed 2 weeks ago by zermok

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

comment:9 Changed 2 weeks ago by oromit

An actually usable commandline that does not rely on a server called "xxxx", ideally does not rely on a server at all and is as minimal as possible.
And the full uncut output, which you have been asked to provide 3 times already.

comment:10 in reply to: ↑ 7 Changed 2 weeks ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from reopened to closed

Replying to zermok:

for now I cannot reinstall the git version since my server is in production mode only.

It is not necessary to install FFmpeg to be able to test and you don't have to use your production server for testing, any system is sufficient.

comment:11 Changed 2 weeks ago by zermok

"xxx" is an example....
provide the command line to get the full uncut console please

comment:12 Changed 2 weeks ago by zermok

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

comment:13 Changed 2 weeks ago by cehoyos

  • Resolution set to needs_more_info
  • Status changed from reopened to closed

Changed 2 weeks ago by zermok

comment:14 Changed 2 weeks ago by zermok

Here the uncut console log..
so it blocks at "Cannot READ RTMP HEADER"
and ffmpeg goes to 100% cpu orphan
stop ffmpeg encoding while the live streaming is still running is ok
it happens only when the live stream stops before ffmpeg.
This issue is not noticeable on release version 3.4.
If you need an rtmp live stream so I can create one permanent for your tests

Last edited 2 weeks ago by zermok (previous) (diff)

comment:15 Changed 2 weeks ago by zermok

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

comment:16 Changed 2 weeks ago by cehoyos

What happens if you press Ctrl-c?

comment:17 Changed 13 days ago by zermok

it exits ffmpeg but does not write the 2pass moov block
and result in a corrupted mp4 file

comment:18 Changed 13 days ago by cehoyos

Could you show the console output in this case?
I did not succeed trying to write an invalid file pressing Ctrl-c.

comment:19 Changed 11 days ago by zermok

It's the file I already uploaded above
output.txt

Last edited 11 days ago by zermok (previous) (diff)

comment:20 Changed 11 days ago by zermok

to reproduce it:
encode an rtmp live stream
stop the live stream
then good ffmpeg version use the rtmp timeout
and write 2pass and exit
the actual git blocks on the red message "CANNOT ReAD RTMP HeADER"
raise to 100% cpu and never quit.
making Ctrl+C does nothing, it has to do Ctrl+C TWICE
to quit ffmpeg but resulting as a corrupted file

comment:21 Changed 11 days ago by cehoyos

Do you see the same issue when you compile without --enable-librtmp and test the native implementation?

comment:22 Changed 8 days ago by zermok

dunno, I'm going to try it now and will tell you asap...

comment:23 Changed 7 days ago by zermok

I get an error
[rtmp @ 0x2771b80] Detected librtmp style URL parameters, these aren't supported by the libavformat internal RTMP handler currently enabled. See the documentation for the correct way to pass parameters. - [rtmp @ 0x2771b80] Unknown connect error (unsupported authentication method?) - [rtmp @ 0x2771b80] Server error: Connection failed

I need to pass connection paramaters otherwise I cannot connect to my server
for now I use -rtmp_conn, what's the equivalent for the native ffmpeg librtmp?

comment:24 Changed 7 days ago by zermok

anyhow, if it was librtmp so it also won't work with ffmpeg 3.4 release...
I suggest to take a look at the differences between the git source and the 3.4 release.

Last edited 7 days ago by zermok (previous) (diff)

comment:25 Changed 7 days ago by cehoyos

Please run git bisect to find the change introducing the issue.

comment:26 Changed 7 days ago by zermok

into the last git or the release 3.4?
sorry I never used how to use git bisect and what to do with

Last edited 7 days ago by zermok (previous) (diff)

comment:27 Changed 6 days ago by cehoyos

To find out which commit introduced the regression you are reporting, you need to compile yourself and use git bisect.

comment:28 Changed 5 days ago by zermok

ok I did
git bisect start
git bisect bad
git bisect good 3.4

but
git bisect good 3.4
fatal: Needed a single revision
Bad rev input: 3.4

Last edited 5 days ago by zermok (previous) (diff)

comment:29 Changed 5 days ago by cehoyos

Use this version instead of 3.4: 80154b1b

comment:30 Changed 4 days ago by zermok

ok thanks I started to test it now step by step

comment:31 Changed 4 days ago by zermok

sorry but how to refresh the repository to the right revision
after git bisect good 80154b1b ??
update: ok I made git checkout 80154b1b,
but after compiled the original 3.4 (80154b1b) sources
I did git bisect good
and git jumped to the next commit...
well there is 835 commits since the last git so
it would be better to jump between revision first, how to do that with git bisect?

Last edited 4 days ago by zermok (previous) (diff)

comment:32 follow-up: Changed 4 days ago by zermok

ok I got it. don't know why it didn't work before
I did again
git bisect start
git bisect bad
git bisect good 80154b1b
and then
git bisect good
so it jumped to the next step

comment:33 follow-up: Changed 4 days ago by zermok

Ok I'm lucky (well at least)
the issue comes just after the 3.4 80154b1b revision

bad one starts at revision 912ceba61b0d45caa8ba8664ddf7b18e2121ddf3

comment:34 Changed 4 days ago by cehoyos

To completely rule out any kind of mistake, please provide command and console output for the working version bb4c9d0a

comment:35 in reply to: ↑ 32 Changed 4 days ago by cehoyos

Replying to zermok:

git bisect start
git bisect bad

In my (very long) experience, this is not a good idea:
You should first compile current FFmpeg, confirm that your newly built binary allows to reproduce the issue you want to bisect, and only then run git bisect bad.

git bisect good 80154b1b

Then you should compile and test the old version, only after you have confirmed that it is a good version, run git bisect good.

So in your case, try something like:

$ make distclean
$ git bisect reset
$ git checkout master
$ ./configure && make ffmpeg
Test and verify a fail
$ make distclean && git bisect bad
$ git checkout 80154b1b
$ ./configure && make ffmpeg
Test and verify a success
$ make distclean && git bisect good

and so on. If the first of above tests does not fail or if the second does not succeed, you verified that there is another issue.

comment:36 in reply to: ↑ 33 Changed 4 days ago by cehoyos

Replying to zermok:

bad one starts at revision 912ceba61b0d45caa8ba8664ddf7b18e2121ddf3

To elaborate why I am quite certain this is not the relevant commit:
It is supposed to only change hardware decoding, even if something went wrong, it only changes vc1 decoding, which is not used by your command line.

comment:37 follow-up: Changed 4 days ago by zermok

It's what I noticed Cehoyos,
just compiled the next revision after the 3.4 release
and the issue happened.... don't know what to say...
I'm going to do what you said above again

Last edited 4 days ago by zermok (previous) (diff)

comment:38 in reply to: ↑ 37 Changed 4 days ago by cehoyos

Replying to zermok:

It's what I noticed Cehoyos,
just compiled the next revision after the 3.4 release

Not sure what you mean with "next revision": You have to test until git bisect tells you the change introducing the regression.

comment:39 Changed 3 days ago by zermok

I confirm that the last good one was the version 3.4
version 80154b1b3a0a9e83a9cbaff8b922440f60998128
I made git log to the first bad after this revision
maybe it helps

comment:40 Changed 3 days ago by cehoyos

Then please also confirm that 3.4 works for you (and provide the command line including console output): http://ffmpeg.org/releases/ffmpeg-3.4.tar.bz2

comment:41 Changed 3 days ago by zermok

sorry the log of the bad revision is 28MB, but I guess you already have it.
yes, the version 3.4 works perfectly, I use it for now in production mode
as I said the issue is if the rtmp live stream stops while ffmpeg is encoding
so ffmpeg wait until the rtmp timeout and block for ever with 100% cpu after the last red message "Cannot read RTMP header".
the version 3.4 release just write the 2pass moov and quit correctly.
I attached the full console log of the version 3.4

Changed 3 days ago by zermok

good encdoing log with version 3.4 release

comment:42 Changed 3 days ago by cehoyos

And 7bec3f78 also works fine? (Sorry, but again, you point to a commit that is an unlikely candidate.)

comment:43 Changed 3 days ago by zermok

ok I try it now....

(Sorry, but again, you point to a commit that is an unlikely candidate.)

sorry I cannot help you ;), maybe a typo or a mistake happened.. who knows

comment:44 Changed 3 days ago by zermok

7bec3f78 looks good, no issue...

comment:45 Changed 3 days ago by cehoyos

  • Keywords librtmp mov regression added
  • Priority changed from normal to important
  • Status changed from reopened to open

Sorry, I was hoping the resulting commit would show something useful.

comment:46 Changed 3 days ago by zermok

bah, me too :)

comment:47 Changed 3 days ago by cehoyos

If you want to help, you can find out which of the changes in 80154b1b introduced the regression.

comment:48 Changed 3 days ago by zermok

well, I'm absolutely not a C programmer, I'm at work for now too...
what I suspect to see and where? which file?
also must I compare 80154b1b with the next revision?

Last edited 3 days ago by zermok (previous) (diff)

comment:49 Changed 3 days ago by cehoyos

If you click on the link that you posted in comment:39 you can see the nine trivial changes. Test if reverting some of them makes the bug go away.

comment:50 Changed 3 days ago by zermok

but... it only shows headers changed on this commit.
does it mean thant maybe the next version of these libraries
introduced the issue?

comment:51 Changed 3 days ago by zermok

ok I started to modify every change

comment:52 follow-up: Changed 3 days ago by zermok

I tested all 9 successfully, no issue when I regressed the minor versions :(
to be sure I recreated a new last git clone but the issue is there....
What a frustration :((((

comment:53 Changed 3 days ago by zermok

a top -c shows
28392 nobody 20 0 1502412 91980 19280 R 100.0 0.3 2:35.52 /usr/local/bin/ffmpeg -fflags +genpts -analyzeduration 0 -r 10 -threads 0 -rtmp_conn S:pass Z: Z: Z:....

100% cpu for ever and no second pass to write the moov container....

comment:54 in reply to: ↑ 52 Changed 3 days ago by cehoyos

Replying to zermok:

I tested all 9 successfully, no issue when I regressed the minor versions :(

I don't understand: Which unchanged version did you test first?
And what did you change and what was the effect?

comment:55 Changed 3 days ago by zermok

I regressed the minor versions of all the 9 changes made on this revision

comment:56 Changed 3 days ago by zermok

anyhow I restarted a bisect again...

comment:57 Changed 3 days ago by zermok

f...ck, same results... but I confused myself, it's the second revision after the 80154b1b
not the first

comment:59 Changed 3 days ago by zermok

don't know what number is a good reference
0f5ad12ba2b538cb329c507ecc914e06bfa70194

comment:60 Changed 3 days ago by zermok

[root@node228 ffmpeg]# git bisect log
git bisect start
# bad: [a0e4c41d086bbc32dfefef0b81ed0f59fe04d4ab] avfilter/vf_pseudocolor: add support for more formats
git bisect bad a0e4c41d086bbc32dfefef0b81ed0f59fe04d4ab
# good: [80154b1b3a0a9e83a9cbaff8b922440f60998128] Bump version for master after 3.4 branchpoint
git bisect good 80154b1b3a0a9e83a9cbaff8b922440f60998128
# bad: [ec703910af668b8deb8432d2953222385b821b2c] Merge commit '0f5ad12ba2b538cb329c507ecc914e06bfa70194'
git bisect bad ec703910af668b8deb8432d2953222385b821b2c

comment:61 Changed 3 days ago by cehoyos

You have to continue building and testing until git bisect tells you that the offending commit has been found.

comment:62 Changed 3 days ago by zermok

yes I got it thanks, is this info enough for you to correct the bug?

Last edited 3 days ago by zermok (previous) (diff)

comment:63 Changed 3 days ago by cehoyos

I believe you have never finished running git bisect, that's at least what your git bisect log output indicates: It should be longer.

comment:64 follow-up: Changed 3 days ago by zermok

ha ok, until the last git I must do it?
but if one of the revision becomes bad, so it will be bad until the end right?

Last edited 3 days ago by zermok (previous) (diff)

comment:65 in reply to: ↑ 64 Changed 3 days ago by cehoyos

Replying to zermok:

ha ok, until the last git I must do it?

Until git bisect tells you that the offending commit was found.

but if one of the revision becomes bad, so it will be bad until the end right?

No, I don't think so.

comment:66 Changed 2 days ago by zermok

ok so I should continue to do git bisect good/bad until the current git right?

comment:67 Changed 2 days ago by cehoyos

No, only until git bisect tells you that the offending commit was found.

comment:68 Changed 36 hours ago by zermok

ok... I'm full of work now, will do when time allows me..

Note: See TracTickets for help on using tickets.