Opened 6 years ago

Closed 5 years ago

Last modified 5 years ago

#1312 closed enhancement (fixed)

Support multi-threaded resolution change

Reported by: jyavenard Owned by:
Priority: wish Component: avcodec
Version: git-master Keywords: h264
Cc: tmatth Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Apple provides some samples of HTTP Live Streaming playlist.
In particular, one provides 4 streams at the following bitrates (in bit/s)
stream1 : 200000
stream2 : 311111
stream3 : 484444
stream4 : 37777

http://devimages.apple.com/iphone/samples/bipbop/gear4/prog_index.m3u8

I implemented a HLS player, allowing to switch streams according to the download bandwidth available. Unfortunately, FFmpeg is unable to play the stream generated when a fallback streams is selected.

The samples attached are made of 10s block made from the 4 streams.
bipbopX.ts is the corresponding TS file where X goes from 1 to 4,

bipbopXY.ts is a stream made of 10s of streamX followed by 10s of streamY

bipbopXYZ.ts is a stream made of 10s of streamX, 10s of streamY and 10s of stream Z
etc...

QuickTime/Safari? can play all but VBR streams with stream 1 (eg: bipbop12, bipob21, bipbop1234, bipbop4321)

ffplay only plays the following sequences:
43 and 34...
All the others fail.

This prevent implementing a HLS player that properly handle change of streams on the fly.

Attachments (14)

bipbop2.ts (808.7 KB) - added by jyavenard 6 years ago.
bipbop1.ts (501.0 KB) - added by jyavenard 6 years ago.
bipbop3.ts (1.2 MB) - added by jyavenard 6 years ago.
bipbop4.ts (1.8 MB) - added by jyavenard 6 years ago.
bipbop43.ts (1.5 MB) - added by jyavenard 6 years ago.
bipbop32.ts (1.0 MB) - added by jyavenard 6 years ago.
bipbop21.ts (661.5 KB) - added by jyavenard 6 years ago.
bipbop12.ts (648.3 KB) - added by jyavenard 6 years ago.
bipbop23.ts (1.0 MB) - added by jyavenard 6 years ago.
bipbop34.ts (1.5 MB) - added by jyavenard 6 years ago.
bipbop1234.ts (2.2 MB) - added by jyavenard 6 years ago.
bipbop4321.ts (2.1 MB) - added by jyavenard 6 years ago.
bipbop234.ts (1.9 MB) - added by jyavenard 6 years ago.
bipbop432.ts (1.9 MB) - added by jyavenard 6 years ago.

Change History (33)

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

Changed 6 years ago by jyavenard

comment:1 Changed 6 years ago by jyavenard

Should have added that of course, ffplay properly plays bipbopX.ts with 1 <= X <= 4

comment:2 Changed 6 years ago by mjs973

  • Keywords segments added
  • Status changed from new to open
  • Type changed from defect to enhancement

You should provide more information if you want to get someone else interested in looking at this problem. Your bug description is "ffplay fails." That is too little information.

You don't even say if you are talking about audio or video.

What is the failure you see or hear? The more detailed, the better.

What characteristic changes in these input files (that might cause trouble)? At the very least, give us an 'ffmpeg -i infile' dump for files 1-4.

comment:3 Changed 6 years ago by jyavenard

Sorry, I thought it was self-explanatory that ffplay fails to handle the transitions occurring at 10, 20 or 30s (whenever there's a bitrate or resolution change) depending on the sample file played.

The samples are mpeg-ts file with a H264 video stream and a AAC mono audio stream.
Every 10s, the H264 stream change, either just the bitrate change, or the resolution (and as such bitrate) change.

it's the whole avcodec_decode_video2 that fails at the transition between bitstreams change and/or video resolution transition.
avcodec_decode_video2 will just return an error as soon as you reach the transition (exactly occurring at 10, 20 or 30s in the samples provided

I only mentioned ffplay failing, as an example on how to easily reproduce the problem.

This is not an enhancement request.. This is a bug !
The H264 decoder will crap out when the H264 stream changes, and you can't recover from that error.

[h264 @ 0x7fba43843600] Width/height/bit depth/chroma idc changing with threads is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x7fba43843600] decode_slice_header error
[h264 @ 0x7fba43843600] Width/height/bit depth/chroma idc changing with threads is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x7fba43843600] decode_slice_header error
[h264 @ 0x7fba43843600] Width/height/bit depth/chroma idc changing with threads is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

It's also not possible to detect the new resolution/bitrate change before avcodec_decode_video2 fails. so one can't take proactive measure to perform some kind of reset.

comment:4 Changed 6 years ago by cehoyos

Is the problem only reproducible with ffplay or also with ffmpeg?
If it is reproducible with ffmpeg, please do not report the problem for ffplay (ffplay depends on an external library that is known to contain bugs, so ffmpeg is the primary application).

In any case: Complete, uncut console output missing.

comment:5 Changed 6 years ago by jyavenard

The problem isn't in ffplay, but with ffmpeg libs (precisely libavcodec H264 decoder)

Again, I only mentioned ffplay because I could easily reproduce the problem there too, same symptoms

It would have only taken 30s of your time to reproduce the problem.

The work around for the time being is to detect manually the change of resolution, close the decoder, and re-open it.

e.g.:

    // definition: AVStream *stream;
    if (stream->codec->codec)
        avcodec_close(stream->codec);
    AVCodec *codec = avcodec_find_decoder(stream-codec->codec_id);
    avcodec_open2(stream->codec, codec, NULL);

comment:6 Changed 6 years ago by reimar

What you quote quite obviously is _not_ a bug, the message is quite clear: Changing resultions is an unsupported feature in frame-multithreaded decode.
If you need support for resolution changes, you cannot use frame multithreading.
Thus yes, this clearly is a feature request.
It would only be a bug if none of the properties the message says actually changes.

comment:7 in reply to: ↑ description Changed 6 years ago by cehoyos

  • Keywords segments removed
  • Version changed from unspecified to git-master

Replying to jyavenard:

QuickTime/Safari? can play all but VBR streams with stream 1 (eg: bipbop12, bipob21, bipbop1234, bipbop4321)

Is there a player that can play the streams that contain stream 1?

comment:8 Changed 6 years ago by jyavenard

bipbop1.ts on its own ; almost everyone will play it.

For one transitioning from or to bipbop1 ; then I don't know.
Neither mplayer, ffmpeg or Quicktime will play them.

comment:9 Changed 6 years ago by cehoyos

Please allow me to rephrase my badly worded question:
Is there a player that can play the streams that contain stream 1 and another stream?
If not, is there any reason to assume the resulting (combined) stream is valid?

comment:10 follow-up: Changed 6 years ago by jyavenard

then the answer is yes.

This type of stream could theorically occur when you are streaming HTTP live streaming.
This stream for example:
http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8

on an iPhone or iPad, it starts streaming from the bipbop1 stream, then realise that there is sufficient bandwidth and change to a higher bitrate stream (bipbop2, 3 or 4). So what you see is bipbop1 for a few seconds, and then it switch to another one. it works.

For a player, it is possible to reset the decoder on the fly as you can detect the change of streams, but say you are recording the stream content instead, you would be unable to play back the resulting mpeg-ts.

MythTV now allows to record HLS stream, and can do switch of streams on the fly, according to the available bandwidth.

That no current player is able to manage that mpeg-ts doesn't mean it can't be done :)

comment:11 in reply to: ↑ 10 Changed 6 years ago by cehoyos

Replying to jyavenard:

That no current player is able to manage that mpeg-ts doesn't mean it can't be done :)

I did not try to imply this, on the contrary, afaict, FFmpeg supports changing from 2 to 3 or 4, from 3 to 2 or 4 and from 4 to 2 or 3 fine. Imo, this suggests that there might be a problem with stream 1.

comment:12 follow-up: Changed 6 years ago by jyavenard

If you use the default multi-threaded h264 decoder, ffmpeg doesn't handle any of them...
mplayer does the change on the fly by resetting the decoder, but it's rather heavy handed and shouldn't be necessary

comment:13 Changed 6 years ago by cehoyos

  • Component changed from undetermined to avcodec
  • Keywords h264 added
  • Priority changed from normal to wish
  • Summary changed from FFmpeg doesn't handle bitrate transition on some mpeg-ts stream to Support multi-threaded resolution change

comment:14 Changed 6 years ago by cehoyos

  • Reproduced by developer set

comment:15 in reply to: ↑ 12 ; follow-up: Changed 5 years ago by tmatth

Replying to jyavenard:

If you use the default multi-threaded h264 decoder, ffmpeg doesn't handle any of them...
mplayer does the change on the fly by resetting the decoder, but it's rather heavy handed and shouldn't be necessary

Just out of curiosity, where does mplayer do this?

comment:16 in reply to: ↑ 15 Changed 5 years ago by cehoyos

  • Cc tmatth added

Replying to tmatth:

Replying to jyavenard:

mplayer does the change on the fly by resetting the decoder, but it's rather heavy handed and shouldn't be necessary

Just out of curiosity, where does mplayer do this?

It does not.

comment:17 Changed 5 years ago by cehoyos

  • Resolution set to fixed
  • Status changed from open to closed

Implemented by Janne Grunau.

comment:18 Changed 5 years ago by jyavenard

excellent news... any chance to get this ever backported to release/1.0 ?

comment:19 Changed 5 years ago by reimar

Personally I think that features should only in very special circumstances ever be backported.
And with such extensive and kind of risky changes I don't think it's a good idea, if you're willing to risk regressions for this feature you can just use git, if you're not willing to take that risk backporting it would be a disservice.

Note: See TracTickets for help on using tickets.