Opened 3 years ago

Last modified 3 years ago

#5295 reopened defect

Unusual bug when file name starts with leading %20

Reported by: rjmoses Owned by:
Priority: normal Component: avformat
Version: unspecified Keywords: hls
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

I think I have discovered an unusual bug when trying to download a video file that is an m3u8 file and the file name contains a leading space encoded as a %20.

Short form of the problem.

1) File is "http://d3fk8fzsguno39.cloudfront.net/Directory/%20VideoFile.m3u8"

2) Amazon cloud server returns header frame with resolution info in which the segment name is : #EXT-X-STREAM-INF..."(space)HLS_640).m3u8 HTTP/1.1"

3) ffmpeg formats the segment request list as : "GET /Directory/(space)HLS_640).m3u8".

4) Amazon cloud server returns "HTTP/1.1 400 Bad request (text/html)" and closes connection.

I believe the problem is that the GET request should be "%20" instead of (space).

I have trace other applications (specifically Flash Player) on a windows 7 system and have found that that is what is being sent.

I have looked at the ffmpeg code and I think (may be wrong) that problem is in the parse_playlist() function and the way it works with the function ff_make_absolute_url(). Without more research, I am not sure how to fix this particular issue.

Command line:
ffmpeg http://d3fk8fzsguno39.cloudfront.net/Directory/%20FileName.m3u8 -vcodec copy -acodec copy -absf aac_adtstoasc "OutputFile?.mp4"

For security reasons, I cannot provide the actual file names until I am sure that I am on the correct line of thought. I can then provide actual test cases and wireshark traces of both working and failing situations in private conversations.

linux ffmpeg version info (7.3.0+git+trusty):

ffmpeg version N-78590-g5590ab4 Copyright (c) 2000-2016 the FFmpeg developers

built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --mandir=/usr/share/man --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libpulse --enable-libfreetype --enable-gnutls --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvidstab
libavutil 55. 18.100 / 55. 18.100
libavcodec 57. 24.103 / 57. 24.103
libavformat 57. 25.100 / 57. 25.100
libavdevice 57. 0.101 / 57. 0.101
libavfilter 6. 32.100 / 6. 32.100
libavresample 3. 0. 0 / 3. 0. 0
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.101 / 2. 0.101
libpostproc 54. 0.100 / 54. 0.100

Note: ffmpeg works satisfactorily if a %20 is embedded in the file name (e.g., "File%20Name").

Thanks

Ralph

Attachments (2)

TestCase.gz (15.9 KB) - added by rjmoses 3 years ago.
http.c modifications, test scripts and m3u8 test files (.ts files not included)
TestBad1.log (3.0 KB) - added by rjmoses 3 years ago.
Failing logfile

Download all attachments as: .zip

Change History (9)

comment:1 Changed 3 years ago by cehoyos

  • Keywords Leading space file name removed
  • Resolution set to needs_more_info
  • Status changed from new to closed
  • Version changed from 2.7.3 to unspecified

Please reopen this ticket if you can provide a reproducible test case.

comment:2 Changed 3 years ago by rjmoses

Attached is a gzip file contain two bash test scripts. The modified code for http.c containing a workaround and the m3u8 files for a good case and a failing case.

1) Due to attachment file size restriction of 2.5MB, the actual video files cannot be uploaded (They are 161MB).

2) All testing was done under Linux Mint 17.3 with current updates as date of this posting.

3) Modified code was excerpted from vlc video player and was changed to add '%' and '/' as uri safe characters to enable this work around to work.

4) TestGood?.s script will test a know good file without embedded blanks in the directory or file name.

TestBad1.s script will test a known bad m3u8 file with embedded blanks in the directory or file name.

5) Documentation referenced is from IETF RFC 2396 paragraph 2.3. (http://tools.ietf.org/html/rfc2396#section-3)

6) The modified code in http.c is bracketed by "#ifdef TestRJMChanges"

7) This problem can be reproduced locally by modifying the segment file names in the m3u8 file to embed blank(s) and renaming the .ts files to match the changed m38u file name.

8) The m3u8 video files need to installed under the apache webserver top level directory, e.g., /var/www/html/TestGood(Bad1).

9) If there is another way, I can upload the two complete directories.

I hope this is sufficient to reopen this problem. Please contact me if there are any questions or problems.

Changed 3 years ago by rjmoses

http.c modifications, test scripts and m3u8 test files (.ts files not included)

comment:3 Changed 3 years ago by rjmoses

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

comment:4 Changed 3 years ago by cehoyos

Please paste the command line that allows to reproduce the issue together with the complete, uncut console output to make this a valid ticket.

comment:5 Changed 3 years ago by rjmoses

The command lines are in the test scripts, but here is a simplified version:

ffmpeg -v 9 -loglevel 99 -i "http://test42/TestBad1/ TB1Stream.m3u8" -vcodec copy -acodec copy -absf aac_adtstoasc "TestBad1.mp4" 2>"TestBad1.log"

The test scripts (.s files) reference "./ffmpeg/ffmpeg" as the command. This is my directory for testing. The stderr output (>2"TestBad1.log) is directed to a log file.

I have also attached a failing log file.

Thanks

Changed 3 years ago by rjmoses

Failing logfile

comment:6 Changed 3 years ago by rjmoses

I want apologize upfront--I am a newbie to video processing and my understanding for formats, etc., is extremely limited at this time.

Upon further investigation, it might be appropriate if the escaped character processing was placed in hls.c: parse_playlist(). This accounts for recursive m3u8 file, i.e., m3u8 invokes a lower level m3u8 file which I believe is the "new variant".

E.g.: An m3u8 top level file contains: (I believe this is handled by variant processing.)

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=8327000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"

HLS_1080_.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=7135000,RESOLUTION=1280x720,CODECS="avc1.4d001f,mp4a.40.2"

HLS_720_.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4989000,RESOLUTION=960x540,CODECS="avc1.4d001f,mp4a.40.2"

HLS_960_.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1507000,RESOLUTION=640x360,CODECS="avc1.42001f,mp4a.40.2"

HLS_640_.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=551000,RESOLUTION=480x270,CODECS="avc1.42001e,mp4a.40.2"

HLS_400_.m3u8

However, this does not account for the condition where the directory/subdirectory contains embedded uri unsafe characters such as a space.

I have tested this idea and it works, but I lack the expertise in ffmpeg to determine where the best place to add this processing would be.

Also, perhaps another place would be ff_make_absolute_url() but I'm not sure how that would impact other functions, etc.

Ralph

comment:7 Changed 3 years ago by cehoyos

  • Keywords hls added
Note: See TracTickets for help on using tickets.