avformat/hlsenc produces corrupted HLS stream meta data

[dvrogozh]

Summary of the bug:

Running HLS stream generation w/ ffmpeg produces corrupted meta-data. Specifically, master m3u8 list has invalid name (tream.m3u8 - leading s is missing) for the child list. Child list is created correctly on the disk. Example:

$ ls -1
index.m3u8
stream_0
stream_0.m3u8

$ cat index.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=211198,RESOLUTION=1280x534,CODECS="avc1.64001f,mp4a.40.2"
tream_0.m3u8

How to reproduce (I've been on 99ebb03 commitid):

ffmpeg -c:v h264 -re -i TearsOfSteel.mp4 -c:a copy \
  -c:v libx264 -f hls -hls_time 2 \
  -hls_playlist_type event -master_pl_name index.m3u8 \
  -hls_segment_filename stream_%v/data%06d.ts \
  -use_localtime_mkdir 1 -var_stream_map 'v:0,a:0' stream_%v.m3u8

This seems to be a regression since commitid 7239254 works ok for me. Here is analisys:

$ git bisect start
$ git bisect bad 99ebb03
$ git bisect good 7239254

git bisect log
git bisect start
# bad: [99ebb033d21df58b359b496662b066bc48ae6662] avformat/hlsenc: use av_asprintf()
git bisect bad 99ebb033d21df58b359b496662b066bc48ae6662
# good: [7239254b1da28c1a0d8dff8916edefc98b88775e] avfilter/hue: fix range in comment
git bisect good 7239254b1da28c1a0d8dff8916edefc98b88775e
# good: [5a0575e32c62765549675833a0e523769955b616] avformat/movenc: add write_clli flag to write clli atom
git bisect good 5a0575e32c62765549675833a0e523769955b616
# good: [c59233d50336c8f8eedd94f960ef927ffbae5c29] lavc/sbc: Remove bool usage.
git bisect good c59233d50336c8f8eedd94f960ef927ffbae5c29
# good: [7e4527e8fa1f8f0bba6a7f7cadc407813f17fbd1] avfilter/vf_derain.c: put all the calculation in model file.
git bisect good 7e4527e8fa1f8f0bba6a7f7cadc407813f17fbd1
# good: [3935c891e96c0819439da43d1b862652bbbdf065] avcodec/flacdsp_template: Fix invalid shifts in decorrelate
git bisect good 3935c891e96c0819439da43d1b862652bbbdf065
# good: [de8f6a403350dca74a4d395a2c8b85592138cf0c] avformat/hlsenc: Fix memleak when deleting old segments
git bisect good de8f6a403350dca74a4d395a2c8b85592138cf0c
# bad: [73dc87c4f07f82b9756b3bbf4d4e9f18e46ef2e3] avformat/hlsplaylist: simplify code for checking whether the string is empty
git bisect bad 73dc87c4f07f82b9756b3bbf4d4e9f18e46ef2e3
# bad: [e2e8ef5076bfe20d70c98bed9f142bc4b94c1e2d] avformat/hlsenc: remove the first slash of the relative path line in the master m3u8 file
git bisect bad e2e8ef5076bfe20d70c98bed9f142bc4b94c1e2d
# good: [c9cd0a20b209144648f8402f6b4ed0970cd54502] avformat/hlsenc: Use AVBPrint to avoid allocations of strings
git bisect good c9cd0a20b209144648f8402f6b4ed0970cd54502
# first bad commit: [e2e8ef5076bfe20d70c98bed9f142bc4b94c1e2d] avformat/hlsenc: remove the first slash of the relative path line in the master m3u8 file

So, this points out to:

commit e2e8ef5076bfe20d70c98bed9f142bc4b94c1e2d
Author: Limin Wang <lance.lmwang@gmail.com>
Date:   Thu Mar 26 21:56:57 2020 +0800

    avformat/hlsenc: remove the first slash of the relative path line in the master m3u8 file

    Please testing with the following command:
    ./ffmpeg -y -i input.mkv \
     -b:v:0 5250k -c:v h264 -pix_fmt yuv420p -profile:v main -level 4.1 \
     -b:a:0 256k \
     -c:a mp2 -ar 48000 -ac 2 -map 0:v -map 0:a:0\
     -f hls -var_stream_map "v:0,a:0" \
     -master_pl_name master.m3u8 -t 300 -hls_time 10 -hls_init_time 4 -hls_list_size \
     10 -master_pl_publish_rate 10  -hls_flags \
     delete_segments+discont_start+split_by_time ./tmp/video.m3u8

    then cat ./tmp/master.m3u8
    before:
    #EXTM3U
    #EXT-X-VERSION:3
    #EXT-X-STREAM-INF:BANDWIDTH=6056600,RESOLUTION=1280x720,CODECS="avc1.4d4829,mp4a.40.33"
    /video.m3u8

    $ ./ffmpeg -i  ./tmp/master.m3u8 -c:v copy -c:a mp2 ./test.mkv
    [hls @ 0x7f82f9000000] Skip ('#EXT-X-VERSION:3')
    [hls @ 0x7f82f9000000] Opening '/video.m3u8' for reading
    [hls @ 0x7f82f9000000] parse_playlist error No such file or directory [/video.m3u8]
    ./tmp/master.m3u8: No such file or directory

    after:
    #EXTM3U
    #EXT-X-VERSION:3
    #EXT-X-STREAM-INF:BANDWIDTH=6056600,RESOLUTION=1280x720,CODECS="avc1.4d4829,mp4a.40.33"
    video.m3u8

    Signed-off-by: Limin Wang <lance.lmwang@gmail.com>

[Carl Eugen Hoyos]

Would you like to tell us how we can reproduce the issue? Please provide a simplified command line including the complete, uncut console output to make this a valid ticket.

[dvrogozh]

How to reproduce is described in the bug description. Here is more specific example with the log:

$ wget https://fate-suite.libav.org/h264/twofields_packet.mp4

$ ffmpeg -c:v h264 -re -i twofields_packet.mp4 -c:a copy -c:v libx264 -vframes 10 -f hls -hls_time 2 -hls_playlist_type event -master_pl_name index.m3u8 -hls_segment_filename stream_%v/data%06d.ts -use_localtime_mkdir 1 -var_stream_map 'v:0,a:0' stream_%v.m3u8
ffmpeg version N-97218-gdf4e008 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-39)
  configuration: --prefix=/home/dvrogozh/git/github/install --disable-doc --enable-vaapi --enable-libmfx --enable-gpl --enable-libx264
  libavutil      56. 42.102 / 56. 42.102
  libavcodec     58. 77.101 / 58. 77.101
  libavformat    58. 42.100 / 58. 42.100
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 77.101 /  7. 77.101
  libswscale      5.  6.101 /  5.  6.101
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2ca9780] st: 0 edit list: 1 Missing key frame while searching for timestamp: 3003
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2ca9780] st: 0 edit list 1 Cannot find an index entry before timestamp: 3003.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'twofields_packet.mp4':
  Metadata:
    major_brand     : avc1
    minor_version   : 0
    compatible_brands: avc1isom
    creation_time   : 2008-01-01T00:01:58.000000Z
  Duration: 00:00:04.90, start: 0.000000, bitrate: 10785 kb/s
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 10603 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 2008-01-01T00:01:58.000000Z
      handler_name    :  Ambarella AVC
      encoder         : Ambarella AVC encoder
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      creation_time   : 2008-01-01T00:01:58.000000Z
      handler_name    :  Ambarella AAC
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[libx264 @ 0x2cb0400] using SAR=1/1
[libx264 @ 0x2cb0400] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x2cb0400] profile High, level 4.0
[libx264 @ 0x2cb0400] 264 - core 142 r2495 6a301b6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, hls, to 'stream_%v.m3u8':
  Metadata:
    major_brand     : avc1
    minor_version   : 0
    compatible_brands: avc1isom
    encoder         : Lavf58.42.100
    Stream #0:0(eng): Video: h264 (libx264), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 29.97 fps, 90k tbn, 29.97 tbc (default)
    Metadata:
      creation_time   : 2008-01-01T00:01:58.000000Z
      handler_name    :  Ambarella AVC
      encoder         : Lavc58.77.101 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      creation_time   : 2008-01-01T00:01:58.000000Z
      handler_name    :  Ambarella AAC
[hls @ 0x2caefc0] Opening 'stream_0/data000000.ts' for writingA speed=1.02x
[hls @ 0x2caefc0] Opening 'stream_0.m3u8.tmp' for writing
[hls @ 0x2caefc0] Opening 'index.m3u8' for writing
frame=   10 fps=0.0 q=-1.0 Lsize=N/A time=00:00:00.59 bitrate=N/A speed=0.618x
video:272kB audio:10kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[libx264 @ 0x2cb0400] frame I:1     Avg QP:24.91  size: 80106
[libx264 @ 0x2cb0400] frame P:5     Avg QP:25.19  size: 27159
[libx264 @ 0x2cb0400] frame B:4     Avg QP:26.13  size: 15588
[libx264 @ 0x2cb0400] consecutive B-frames: 20.0% 80.0%  0.0%  0.0%
[libx264 @ 0x2cb0400] mb I  I16..4: 13.7% 69.5% 16.8%
[libx264 @ 0x2cb0400] mb P  I16..4:  2.9% 13.2%  1.1%  P16..4: 32.8%  5.7%  4.6%  0.0%  0.0%    skip:39.8%
[libx264 @ 0x2cb0400] mb B  I16..4:  0.3%  1.9%  0.1%  B16..8: 41.0%  2.9%  0.7%  direct: 3.3%  skip:49.8%  L0:41.3% L1:55.1% BI: 3.7%
[libx264 @ 0x2cb0400] 8x8 transform intra:73.4% inter:87.4%
[libx264 @ 0x2cb0400] coded y,uvDC,uvAC intra: 71.1% 43.9% 9.0% inter: 15.6% 14.1% 0.8%
[libx264 @ 0x2cb0400] i16 v,h,dc,p: 18% 40%  5% 37%
[libx264 @ 0x2cb0400] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 17% 25%  5%  4%  4%  7%  6%  5%
[libx264 @ 0x2cb0400] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 54%  8%  3%  3%  3%  3%  2%  2%
[libx264 @ 0x2cb0400] i8c dc,h,v,p: 58% 15% 25%  2%
[libx264 @ 0x2cb0400] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x2cb0400] ref P L0: 76.4%  9.9% 10.8%  2.9%
[libx264 @ 0x2cb0400] ref B L0: 90.8%  9.2%
[libx264 @ 0x2cb0400] kb/s:6671.42

$ cat index.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=140798,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
tream_0.m3u8

Issues is on the very last line: 'tream_0.m3u8' should actually be 'stream_0.m3u8'

[lmwang]

Sorry, I have submit a patch to fix it, please try it.
​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20200409015046.831-1-lance.lmwang@gmail.com/

[dvrogozh]

Patch works for me, thank you.

[Steven Liu]

fixed by Limin Wang
commit id: df08db07110dc00f7dc0ca8bc5162fffcb9008c7