Opened 3 years ago

Last modified 3 years ago

#8998 new defect

Getting "Error when loading first segment" in some of the version FFmpeg

Reported by: Nabi Owned by:
Priority: important Component: avformat
Version: git-master Keywords: hls regression
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

I have remote index.m3u8 file and I want to download it with ffmpeg.
The contents file is:

#EXTM3U
#EXT-X-TARGETDURATION:6
#EXT-X-ALLOW-CACHE:YES
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-KEY:METHOD=AES-128,URI="https://example.com/encryption.key?x=AVCoPj..."
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:6.000,
https://example.com/seg-1-v1-a1.ts?x=AVCoPj...
#EXTINF:6.000,
https://example.com/seg-2-v1-a1.ts?x=AVCoPj...
#EXTINF:6.000,
https://example.com/seg-3-v1-a1.ts?x=AVCoPj...
#EXTINF:6.000,
:
:

I tried this command:

ffmpeg -headers "User-Agent: a" -i "https://example.com/index.m3u8?x=AVCoPj..." -c copy "video.mp4"

In the some of versions of ffmpeg I getting this errors:

[AVIOContext @ 000001cef2816ec0] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] Error when loading first segment 'https://example.com/seg-1-v1-a1.ts?x=AVCoPj...'
[AVIOContext @ 000001cef21c8f00] Statistics: 24167 bytes read, 0 seeks
https://example.com/index.m3u8?x=AVCoPj...: Invalid data found when processing input

You can see all logs here:

# ffmpeg -headers "User-Agent: a" -i "https://example.com/index.m3u8?x=AVCoPj..." -c copy "video.mp4" -loglevel debug
ffmpeg version 2020-11-18-git-e3081d6f4f-full_build-www.gyan.dev Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 10.2.0 (Rev3, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      56. 60.100 / 56. 60.100
  libavcodec     58.112.103 / 58.112.103
  libavformat    58. 64.100 / 58. 64.100
  libavdevice    58. 11.102 / 58. 11.102
  libavfilter     7. 90.100 /  7. 90.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Splitting the commandline.
Reading option '-headers' ... matched as AVOption 'headers' with argument 'User-Agent: a'.
Reading option '-i' ... matched as input url with argument 'https://example.com/index.m3u8?x=AVCoPj...'.
Reading option '-c' ... matched as option 'c' (codec name) with argument 'copy'.
Reading option 'video.mp4' ... matched as output url.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url https://example.com/index.m3u8?x=AVCoPj....
Successfully parsed a group of options.
Opening an input file: https://example.com/index.m3u8?x=AVCoPj....
[NULL @ 000001cef21be3c0] Opening 'https://example.com/index.m3u8?x=AVCoPj...' for reading
[https @ 000001cef21bed80] Setting default whitelist 'http,https,tls,rtp,tcp,udp,crypto,httpproxy'
[https @ 000001cef21bed80] No trailing CRLF found in HTTP header. Adding it.
[tcp @ 000001cef21c1f80] Original list of addresses:
[tcp @ 000001cef21c1f80] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef21c1f80] Interleaved list of addresses:
[tcp @ 000001cef21c1f80] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef21c1f80] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef21c1f80] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef21bed80] request: GET /index.m3u8?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: close
Host: example.com
Icy-MetaData: 1
User-Agent: a


[hls @ 000001cef21be3c0] Format hls probed with size=2048 and score=100
[hls @ 000001cef21be3c0] Skip ('#EXT-X-ALLOW-CACHE:YES')
[hls @ 000001cef21be3c0] Skip ('#EXT-X-VERSION:3')
[https @ 000001cef21bed80] Last chunk received, closing conn
[hls @ 000001cef21be3c0] HLS request for url 'https://example.com/seg-1-v1-a1.ts?x=AVCoPj...', offset 0, playlist 0
[hls @ 000001cef21be3c0] Opening 'https://example.com/encryption.key?x=AVCoPj...' for reading
[tcp @ 000001cef2891bc0] Original list of addresses:
[tcp @ 000001cef2891bc0] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef2891bc0] Interleaved list of addresses:
[tcp @ 000001cef2891bc0] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef2891bc0] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef2891bc0] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef2270900] request: GET /encryption.key?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a


[AVIOContext @ 000001cef2269980] Statistics: 16 bytes read, 0 seeks
[hls @ 000001cef21be3c0] Opening 'crypto+https://example.com/seg-1-v1-a1.ts?x=AVCoPj...' for reading
[tcp @ 000001cef21eb740] Original list of addresses:
[tcp @ 000001cef21eb740] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef21eb740] Interleaved list of addresses:
[tcp @ 000001cef21eb740] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef21eb740] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef21eb740] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef282acc0] request: GET /seg-1-v1-a1.ts?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a


[AVIOContext @ 000001cef28992c0] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] HLS request for url 'https://example.com/seg-2-v1-a1.ts?x=AVCoPj...', offset 0, playlist 0
[hls @ 000001cef21be3c0] Opening 'crypto+https://example.com/seg-2-v1-a1.ts?x=AVCoPj...' for reading
[tcp @ 000001cef21e2a80] Original list of addresses:
[tcp @ 000001cef21e2a80] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef21e2a80] Interleaved list of addresses:
[tcp @ 000001cef21e2a80] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef21e2a80] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef21e2a80] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef282acc0] request: GET /seg-2-v1-a1.ts?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a


[AVIOContext @ 000001cef21c6d40] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] HLS request for url 'https://example.com/seg-3-v1-a1.ts?x=AVCoPj...', offset 0, playlist 0
[hls @ 000001cef21be3c0] Opening 'crypto+https://example.com/seg-3-v1-a1.ts?x=AVCoPj...' for reading
[tcp @ 000001cef282bd40] Original list of addresses:
[tcp @ 000001cef282bd40] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef282bd40] Interleaved list of addresses:
[tcp @ 000001cef282bd40] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef282bd40] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef282bd40] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef2270900] request: GET /seg-3-v1-a1.ts?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a


[AVIOContext @ 000001cef226e6c0] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] HLS request for url 'https://example.com/seg-4-v1-a1.ts?x=AVCoPj...', offset 0, playlist 0
[hls @ 000001cef21be3c0] Opening 'crypto+https://example.com/seg-4-v1-a1.ts?x=AVCoPj...' for reading
[tcp @ 000001cef282bec0] Original list of addresses:
[tcp @ 000001cef282bec0] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef282bec0] Interleaved list of addresses:
[tcp @ 000001cef282bec0] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef282bec0] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef282bec0] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef282acc0] request: GET /seg-4-v1-a1.ts?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a

:
:
:

[AVIOContext @ 000001cef226b9c0] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] HLS request for url 'https://example.com/seg-75-v1-a1.ts?x=AVCoPj...', offset 0, playlist 0
[hls @ 000001cef21be3c0] Opening 'crypto+https://example.com/seg-75-v1-a1.ts?x=AVCoPj...' for reading
[tcp @ 000001cef28dd1c0] Original list of addresses:
[tcp @ 000001cef28dd1c0] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef28dd1c0] Interleaved list of addresses:
[tcp @ 000001cef28dd1c0] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef28dd1c0] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef28dd1c0] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef282ba40] request: GET /seg-75-v1-a1.ts?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a


[AVIOContext @ 000001cef2817800] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] HLS request for url 'https://example.com/seg-76-v1-a1.ts?x=AVCoPj...', offset 0, playlist 0
[hls @ 000001cef21be3c0] Opening 'crypto+https://example.com/seg-76-v1-a1.ts?x=AVCoPj...' for reading
[tcp @ 000001cef2270700] Original list of addresses:
[tcp @ 000001cef2270700] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef2270700] Interleaved list of addresses:
[tcp @ 000001cef2270700] Address 1.2.3.4 port 443
    Last message repeated 1 times
[tcp @ 000001cef2270700] Starting connection attempt to 1.2.3.4 port 443
[tcp @ 000001cef2270700] Successfully connected to 1.2.3.4 port 443
[https @ 000001cef2271600] request: GET /seg-76-v1-a1.ts?x=AVCoPj... HTTP/1.1
Accept: */*
Range: bytes=0-
Connection: keep-alive
Host: example.com
Icy-MetaData: 1
User-Agent: a


[AVIOContext @ 000001cef2816ec0] Statistics: 368 bytes read, 0 seeks
[hls @ 000001cef21be3c0] Error when loading first segment 'https://example.com/seg-1-v1-a1.ts?x=AVCoPj...'
[AVIOContext @ 000001cef21c8f00] Statistics: 24167 bytes read, 0 seeks
https://example.com/index.m3u8?x=AVCoPj...: Invalid data found when processing input

I tried many versions of ffmpeg and get these results:

ffmpeg-1.0.1-win64-static		Error in the pull function.
ffmpeg-1.2-win64-static			Error in the pull function.
ffmpeg-2.0.1-win64-static		Error in the pull function.
ffmpeg-2.2.3-win64-static		Error in the pull function.
ffmpeg-3.0-win64-static			OK
ffmpeg-3.1.5-win64-static		OK warning...
ffmpeg-3.2.4-win64-static		OK
ffmpeg-3.3.4-win64-static		OK
ffmpeg-3.4.2-win64-static		OK
ffmpeg-4.0.2-win64-static		OK
ffmpeg-4.1.4-win64-static		OK
ffmpeg-4.2.3-win64-static		OK
ffmpeg-4.3-win64-static			Error when loading first segment
ffmpeg-4.3.1-win64-static		Error when loading first segment
ffmpeg-2020-11-18-git-e3081d6f4f	Error when loading first segment

Seem after >=4.3 there is a bug in ffmpeg!

Change History (6)

comment:2 by Carl Eugen Hoyos, 3 years ago

Component: ffmpegavformat
Keywords: hls regression added; bug ffmpeg m3u8 removed
Priority: importantnormal

If you cannot provide a sample url you will have to run git bisect yourself.

comment:3 by Nabi, 3 years ago

The final result of git bisect command:

# git bisect good
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[5e829262a6a6068dfed9d76da4a9635f1f4be391] lavf/hls: add http_seekable option for HTTP partial requests

# git bisect bad
5e829262a6a6068dfed9d76da4a9635f1f4be391 is the first bad commit
commit 5e829262a6a6068dfed9d76da4a9635f1f4be391
Author: Jun Zhao <barryjzhao@tencent.com>
Date:   Thu Aug 8 00:12:16 2019 +0800

    lavf/hls: add http_seekable option for HTTP partial requests

    Add http_seekable option for HTTP partial requests, when The
    EXT-X-BYTERANGE tag indicates that a Media Segment is a sub-range
    of the resource identified by its URI, we can use HTTP partial
    requests to get the Media Segment.

    Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
    Signed-off-by: Jun Zhao <barryjzhao@tencent.com>

 doc/demuxers.texi | 4 ++++
 libavformat/hls.c | 9 +++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

https://github.com/FFmpeg/FFmpeg/commit/5e829262a6a6068dfed9d76da4a9635f1f4be391

comment:4 by Carl Eugen Hoyos, 3 years ago

Priority: normalimportant
Version: unspecifiedgit-master

comment:5 by Nabi, 3 years ago

Note: Also in the same system on the chrome browser nothing problem with play this video URL and also can be download with streamlink software.(so not have any problem for network or blocking IP or broken link.)

Last edited 3 years ago by Nabi (previous) (diff)

comment:6 by Jun Zhao, 3 years ago

You can use option http_seekable=0 in this case, the root cause is "Some HLS servers don't like being sent the range header, in this case, need to setting http_seekable = 0 to disable the range header"

Note: See TracTickets for help on using tickets.