Opened 9 years ago

Closed 8 years ago

#4797 closed defect (fixed)

Trying to save m3u8 as mp4 giving "could not find corresponding trex" error.

Reported by: Jonzo Owned by: Anssi Hannula
Priority: important Component: avformat
Version: git-master Keywords: hls regression debian
Cc: anssi.hannula@iki.fi Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Summary of the bug:

Trying to save an m3u8 file with error. Tried many different versions and command line options.

I am able to save the files in VLC and play it in ffplay.

How to reproduce:

ffmpeg -v 9 -loglevel 99 -i http://a126.phobos.apple.com/us/r30/PurpleVideo1/v4/27/e0/fe/27e0feb3-6f4b-4445-9aa0-35ca9c90a7ce/P37213012_default.m3u8 -acodec copy -vcodec copy  -y -loglevel info -f mp4 myNewVideo.mp4
ffmpeg version N-74505-gc1507db Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      54. 30.100 / 54. 30.100
  libavcodec     56. 57.101 / 56. 57.101
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 34.100 /  5. 34.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'http://a126.phobos.apple.com/us/r30/PurpleVideo1/v4/27/e0/fe/27e0feb3-6f4b-4445-9aa0-35ca9c90a7ce/P37213012_default.m3u8'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'copy'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'copy'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'info'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'mp4'.
Reading option 'myNewVideo.mp4' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Applying option y (overwrite output files) with argument 1.
Applying option loglevel (set logging level) with argument info.
[http @ 000000000477ad40] No trailing CRLF found in HTTP header.
[mov,mp4,m4a,3gp,3g2,mj2 @ 000000000478a940] could not find corresponding trex
[mov,mp4,m4a,3gp,3g2,mj2 @ 000000000478a940] error reading header
http://a126.phobos.apple.com/us/r30/PurpleVideo1/v4/27/e0/fe/27e0feb3-6f4b-4445-9aa0-35ca9c90a7ce/P37213012_default.m3u8: Invalid data found when processing input

FFprobe output:

 ffprobe -loglevel debug http://apptrailers.itunes.apple.com/apple-assets-us-std-000001/PurpleVideo69/v4/91/3b/f0/913bf074-62df-9e4b-8a6b-9db2ecfb50bf/P37213012__video_gr40.m3u8
ffprobe version N-74505-gc1507db Copyright (c) 2007-2015 the FFmpeg developers
  built with gcc 4.9.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      54. 30.100 / 54. 30.100
  libavcodec     56. 57.101 / 56. 57.101
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 34.100 /  5. 34.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[http @ 000000000491d060] request: GET /apple-assets-us-std-000001/PurpleVideo69/v4/91/3b/f0/913bf074-62df-9e4b-8a6b-9db2ecfb50bf/P37213012__video_gr40.m3u8 HTTP/1.1
User-Agent: Lavf/56.40.101
Accept: */*
Range: bytes=0-
Connection: close
Host: apptrailers.itunes.apple.com
Icy-MetaData: 1


[hls,applehttp @ 000000000491c120] Format hls,applehttp probed with size=2048 and score=100
[hls,applehttp @ 000000000491c120] HLS request for url 'http://apptrailers.itunes.apple.com/apple-assets-us-std-000001/PurpleVideo69/v4/91/3b/f0/913bf074-62df-9e4b-8a6b-9db2ecfb50bf/P37213012__video_gr40.mp4', offset 792, playlist 0
[http @ 000000000493a920] No trailing CRLF found in HTTP header.
[http @ 000000000493a920] request: GET /apple-assets-us-std-000001/PurpleVideo69/v4/91/3b/f0/913bf074-62df-9e4b-8a6b-9db2ecfb50bf/P37213012__video_gr40.mp4 HTTP/1.1
User-Agent: Lavf/56.40.101
Accept: */*
Range: bytes=792-1369426
Connection: close
Host: apptrailers.itunes.apple.com
Icy-MetaData: 1



Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 000000000491f0a0] could not find corresponding trex
[mov,mp4,m4a,3gp,3g2,mj2 @ 000000000491f0a0] error reading header
[AVIOContext @ 00000000049270e0] Statistics: 658 bytes read, 0 seeks
http://apptrailers.itunes.apple.com/apple-assets-us-std-000001/PurpleVideo69/v4/91/3b/f0/913bf074-62df-9e4b-8a6b-9db2ecfb50bf/P37213012__video_gr40.m3u8: Invalid data found when processing input

FFprobe output of saved file through VLC

ffprobe.exe -loglevel debug vlc_saved_output_candy_bear_higher_quality.mp4
ffprobe version N-74505-gc1507db Copyright (c) 2007-2015 the FFmpeg developers
  built with gcc 4.9.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      54. 30.100 / 54. 30.100
  libavcodec     56. 57.101 / 56. 57.101
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 34.100 /  5. 34.100
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] ISO: File Type Major Brand: mp42
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] Before avformat_find_stream_info() pos: 9568680 bytes read:196608 seeks:5
[h264 @ 000000000480d960] Increasing reorder buffer to 1
[h264 @ 000000000480d960] no picture
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 29.666667 0.016451
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 29.750000 0.009253
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 29.833333 0.004113
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 29.916667 0.001028
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 30.000000 0.000000
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 60.000000 0.000000
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 120.000000 0.000000
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 240.000000 0.000000
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 29.970030 0.000133
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] rfps: 59.940060 0.000532
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000047fc080] After avformat_find_stream_info() pos: 380304 bytes read:589157 seeks:6 frames:41
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'vlc_saved_output_candy_bear_higher_quality.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isomhlsf
    creation_time   : 2015-08-08 13:04:49
  Duration: 00:00:25.97, start: 0.033333, bitrate: 3179 kb/s
    Stream #0:0(und), 41, 1/30: Video: h264 (High), 5 reference frames (avc1 / 0x31637661), yuv420p(tv, bt709, topleft), 1108x830 (1120x832) [SAR 1:1 DAR 554:415], 1/60, 3279 kb/s, 30 fps, 30 tbr, 30 tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2015-08-08 13:04:49
      handler_name    : Core Media Video
detected 8 logical cores
[AVIOContext @ 0000000004805000] Statistics: 589157 bytes read, 6 seeks

Attachments (3)

media_input_candy_bear.m3u8 (4.1 KB ) - added by Jonzo 9 years ago.
vlc_saved_output_candy_bear_low_quality.mp4 (1.8 MB ) - added by Jonzo 9 years ago.
CNN.m3u (180 bytes ) - added by Carl Eugen Hoyos 9 years ago.

Download all attachments as: .zip

Change History (10)

by Jonzo, 9 years ago

Attachment: media_input_candy_bear.m3u8 added

in reply to:  description comment:1 by Carl Eugen Hoyos, 9 years ago

Component: undeterminedavformat
Keywords: hls regression added; m3u8 trex error itunes video removed
Priority: normalimportant
Reproduced by developer: set
Status: newopen
Version: unspecifiedgit-master

Replying to Jonzo:

 ffprobe -loglevel debug http://apptrailers.itunes.apple.com/apple-assets-us-std-000001/PurpleVideo69/v4/91/3b/f0/913bf074-62df-9e4b-8a6b-9db2ecfb50bf/P37213012__video_gr40.m3u8

This is a regression since da7759b3579de3e98deb1ac58e642b861280ba54

comment:2 by Anssi Hannula, 9 years ago

Cc: anssi.hannula@iki.fi added
Owner: set to Anssi Hannula

Before da7759b3579de3e98deb1ac58e642b861280ba54 the EXT-X-BYTERANGE in the playlist weren't handled at all so FFmpeg would just play back the entire file 6 times (there are 6 segments in the playlist), which I guess is a better result from user's perspective than not working at all ;)

The playlist seems to contain some unsupported feature(s) causing some bytes to be missing from the stream and thus demuxing fails, I'll take a look at implementing them.

by Carl Eugen Hoyos, 9 years ago

Attachment: CNN.m3u added

comment:3 by Carl Eugen Hoyos, 9 years ago

Keywords: debian added

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798189 seems to describe the same issue, at least it is also a regression since da7759b3.
I attached the file CNN.m3u that allows to test the issue, reproducible both with and without --enable-librtmp.

$ ffmpeg -loglevel 99 -i CNN.m3u
ffmpeg version N-75008-g1964aea Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl
  libavutil      55.  0.100 / 55.  0.100
  libavcodec     57.  0.100 / 57.  0.100
  libavformat    57.  0.100 / 57.  0.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6.  0.100 /  6.  0.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'CNN.m3u'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument 99.
Successfully parsed a group of options.
Parsing a group of options: input file CNN.m3u.
Successfully parsed a group of options.
Opening an input file: CNN.m3u.
Probing hls,applehttp score:100 size:180
[hls,applehttp @ 0x2fb32a0] Format hls,applehttp probed with size=2048 and score=100
[hls,applehttp @ 0x2fb32a0] HLS request for url 'rtmp://b.cdn.newschat.tv/edge/cnn_live swfUrl=http://newschat.tv/player.swf pageUrl=http://www.livenewschat.eu/top/', offset 0, playlist 0
[rtmp @ 0x2fb74e0] Detected librtmp style URL parameters, these aren't supported by the libavformat internal RTMP handler currently enabled. See the documentation for the correct way to pass parameters.
[rtmp @ 0x2fb74e0] Handshaking...
[rtmp @ 0x2fb74e0] Type answer 3
[rtmp @ 0x2fb74e0] Server version 13.14.10.13
[rtmp @ 0x2fb74e0] Proto = rtmp, path = /edge/cnn_live, app = edge, fname = cnn_live
[rtmp @ 0x2fb74e0] Server bandwidth = 5000000
[rtmp @ 0x2fb74e0] Client bandwidth = 5000000
[rtmp @ 0x2fb74e0] New incoming chunk size = 16384
[rtmp @ 0x2fb74e0] Creating stream...
[rtmp @ 0x2fb74e0] Sending play command for 'cnn_live'
[hls,applehttp @ 0x2fb32a0] Unable to seek to offset 0 of HLS segment 'rtmp://b.cdn.newschat.tv/edge/cnn_live swfUrl=http://newschat.tv/player.swf pageUrl=http://www.livenewschat.eu/top/'
[rtmp @ 0x2fb74e0] Deleting stream...
[hls,applehttp @ 0x2fb32a0] Failed to open segment of playlist 0
[hls,applehttp @ 0x2fb32a0] Error when loading first segment 'rtmp://b.cdn.newschat.tv/edge/cnn_live swfUrl=http://newschat.tv/player.swf pageUrl=http://www.livenewschat.eu/top/'
[AVIOContext @ 0x2fbbe20] Statistics: 180 bytes read, 0 seeks
CNN.m3u: Invalid data found when processing input

comment:4 by Carl Eugen Hoyos, 9 years ago

Another stream from ticket #4846 that also does not work since da7759b3:

$ ffmpeg -i "https://toutvuniver1-vh.akamaihd.net/i/012/mp4/c/2015-09-11_19_00_00_cheforeille_0001_,500,800,1200,2000,3000,.mp4.csmil/master.m3u8?hdnea=st=1442150589~exp=1444742589~acl=/i/012/mp4/c/2015-09-11_19_00_00_cheforeille_0001_*~hmac=84d9371012603adc84fc95b49714153067267921e1fd1657aa4dbde37a135c90"
ffmpeg version N-75274-g39c61d8 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (SUSE Linux)
  configuration: --enable-gpl --enable-gnutls
  libavutil      55.  2.100 / 55.  2.100
  libavcodec     57.  1.100 / 57.  1.100
  libavformat    57.  0.100 / 57.  0.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6.  3.100 /  6.  3.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100
[https @ 0x3727bc0] No trailing CRLF found in HTTP header.
[http @ 0x3727bc0] HTTP error 403 Forbidden
[crypto @ 0x377cfa0] Unable to open resource: http://toutvuniver1-vh.akamaihd.net/i/012/mp4/c/2015-09-11_19_00_00_cheforeille_0001_,500,800,1200,2000,3000,.mp4.csmil/segment1_0_av.ts?null=&id=AgBQbmw8aFMElRPW9VWJqhwWsWEs563L0f+nx8N7fxLnJzL3I7%2fGJht44xunR2RgQBF06lHGQvbarA%3d%3d
[hls,applehttp @ 0x3715580] Failed to open segment of playlist 0

in reply to:  4 comment:5 by Carl Eugen Hoyos, 8 years ago

Replying to cehoyos:

Another stream from ticket #4846 that also does not work since da7759b3:

$ ffmpeg -i "https://toutvuniver1-vh.akamaihd.net/i/012/mp4/c/2015-09-11_19_00_00_cheforeille_0001_,500,800,1200,2000,3000,.mp4.csmil/master.m3u8?hdnea=st=1442150589~exp=1444742589~acl=/i/012/mp4/c/2015-09-11_19_00_00_cheforeille_0001_*~hmac=84d9371012603adc84fc95b49714153067267921e1fd1657aa4dbde37a135c90"

This stream was fixed in 26eb2940079d0ec433cf9b2deae24560707cbcf8

in reply to:  3 comment:6 by Carl Eugen Hoyos, 8 years ago

Replying to cehoyos:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798189 seems to describe the same issue, at least it is also a regression since da7759b3.

This stream was fixed by Andreas Cadhalpun in f9f0b4c08e7dc2c178ede137af4f64ca84ab2deb
The original "trex" issue is still reproducible.

comment:7 by Anssi Hannula, 8 years ago

Resolution: fixed
Status: openclosed

Sorry about the delay.

This should now be fixed in 909907948846dedf57a730a4d115d04d1117f9e5.

Seeking for this stream doesn't seem to work, though, as unlike with mpegts we do not seem to get the "real" dts from the mp4 subdemuxer after seeking the stream from under its feet. Not immediately sure what is the best way to handle that, but that is a separate issue.

Note: See TracTickets for help on using tickets.