Computed HLS duration is sometimes incorrect with short segment durations

[fthiery]

Summary of the bug:

When segmenting an mp4/H.264/AAC file, computed HLS playlist duration (by summing all EXTINF metadata) deviates from the proper duration, i.e. the original file's.

How to reproduce:

% ffmpeg -i video.mp4 -flags -global_header -f segment -segment_time 1 -segment_list playlist.m3u8 -segment_list_type m3u8 -segment_format mpegts -map 0 -bsf h264_mp4toannexb -vcodec copy -acodec copy -y media%05d.ts

Ffmpeg version compiled from git today:
ffmpeg version N-64012-g61df081 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jun 17 2014 08:50:14 with gcc 4.8 (Ubuntu 4.8.1-2ubuntu1~10.04.1)
  configuration: --disable-ffserver --enable-shared --enable-libx264 --enable-libfaac --enable-libtheora --enable-libvorbis --enable-libmp3lame --enable-libxvid --enable-gpl --enable-nonfree --enable-pthreads --enable-libvpx --extra-cflags='-I/tmp/codecs/include/ -I/tmp/codecs/include/ao -I/tmp/codecs/include/lame -I/tmp/codecs/include/ogg -I/tmp/codecs/include/vorbis -I/tmp/codecs/include/theora -I/tmp/codecs/include/vpx' --extra-ldflags='-L/tmp/codecs/lib -static'
  libavutil      52. 89.100 / 52. 89.100
  libavcodec     55. 67.100 / 55. 67.100
  libavformat    55. 43.100 / 55. 43.100
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  8.100 /  4.  8.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100

ffprobe reports some duration on the m3u8 resource (in my case, 1957.493063s) where the actual duration is 1935.033333s (obtained by ffprob'ing the original mp4 file or using the last frame timestamp + frame duration by analyzing the m3u8 resource with ffprobe). One of the issues is that the duration difference is large enough (~40s for one hour of content) for most players to seek to an unappropriate location (e.g. 10 seconds too late).

Btw original mp4 file has an average gop size of 30 (video is 30 fps), and the goal here is to have 1 second segments for precise seeking.

[fthiery]

Example: real duration is 1935.03333 seconds.

When raising the segment duration, the duration difference (and drift) is smaller.

segment_time (s)	hls reported dur (s)	duration difference (s)	per minute drift (ms)
1	1957.493063	22.46	696
5	1939.729325	4.7	146
10	1937.385608	2.35	73

[Carl Eugen Hoyos]

Is a specific input file necessary to reproduce the issue or does ffmpeg -f lavfi -i testsrc allow to reproduce?
Please provide the command line that produces the unexpected output files together with the complete, uncut console output to make this a valid ticket.

[fthiery]

I am indeed failing to reproduce it using a clean, generated test file.

ffmpeg -f lavfi -i testsrc -t 1800 -r 30 -g 30 -codec:v libx264 -pix_fmt yuv420p -b:v 1M -filter_complex "aevalsrc=sin(440*2*PI*t)" -t 1800 -c:a libfaac -b:a 128k test.mp4

Duration is quite precisely 1800s (30 min):

ffprobe test.mp4
[...]
  Duration: 00:30:00.02, start: 0.023220, bitrate: 313 kb/s

Then i segment:

mkdir ts
ffmpeg -i test.mp4 -flags -global_header -f segment -segment_time 1 -segment_list ts/playlist.m3u8 -segment_list_type m3u8 -segment_format mpegts -map 0 -bsf h264_mp4toannexb -vcodec copy -acodec copy -y ts/media%05d.ts

Duration is then correct

ffprobe ts/playlist.m3u
[...]
  Duration: 00:30:00.09, start: 0.043422, bitrate: 0 kb/s

The difference is that my content is the concatenation of two mkv files using mkvtoolnix's mkvmerge between a 5 seconds long "intro" and the actual video. Analyzing the gopsize using ffprobe shows that gops get "offset" after the intro video, where the file generated with ffmpeg has a perfect gop of 30 all the time.

Gopsize (i.e. timecode_s is the iframe timecode, gopsize is the number of consecutive frames + 1 until the next keyframe):

timecode_s	gopsize
0.0	30
1.0	30
2.0	30
3.0	30
4.0	30
5.0	7
5.233	30
6.233	30
...	...
94.233	30
95.233	2
95.3	28
96.233	30
...	...
220.833	30
221.833	1
221.866	29
686.833	30
687.833	1
687.866	29
688.833	30
...	...

While i am suspecting that this merging is the origin of the problem, is there a different way of generating a test video using ffmpeg so that it triggers scene changes and shorter gops (as gops of 1 or 2, then 28 in the real video) ?

[Carl Eugen Hoyos]

Replying to fthiery:

The difference is that my content is the concatenation of two mkv files using mkvtoolnix's mkvmerge

How did you convert the output of mkvmerge to the mp4 file used in your original report?

[fthiery]

After further investigation, it seems that the merging is not related, but the encoding.

After remuxing, reencoding audio, then video, it seems that the original h.264 encoding is
the problem (gstreamer's x264enc with pass=5 quantizer=22 subme=4 key-int-max=30) because the problem only appears if it keep the original video bitstream (vcodec copy).

The closest equivalent would be -r 30 -codec:v libx264 -crf 22 -x264opts keyint=30:subme=4, and it still does not reproduce the issue...

On the reencoded file, there are non-regular gops (on scenecuts), but the problem doesn't appear.

89.0	30
90.0	2
90.066667	28
91.0	30
92.0	30

If you are willing to take a look, the following video triggers the problem (it is 25 fps and not 30, but that's the same issue): ​http://d2hqzi4c71e9y2.cloudfront.net/moodlemoot/presentation-des-defis_83f8/presentation-des-defis_low.mp4

[Carl Eugen Hoyos]

Please post the command line that produces the unexpected output for the sample you provided together with the complete, uncut console output.

[fthiery]

Here it is:

wget http://d2hqzi4c71e9y2.cloudfront.net/moodlemoot/presentation-des-defis_83f8/presentation-des-defis_low.mp4

ffprobe presentation-des-defis_low.mp4

ffprobe version N-60597-g1e5cb42 Copyright (c) 2007-2014 the FFmpeg developers
  built on Feb 14 2014 10:22:18 with gcc 4.8 (Ubuntu 4.8.1-2ubuntu1~10.04.1)
  configuration: --disable-ffserver --enable-shared --enable-libx264 --enable-libfaac --enable-libtheora --enable-libvorbis --enable-libmp3lame --enable-libxvid --enable-gpl --enable-nonfree --enable-pthreads --enable-libvpx --extra-cflags='-I/tmp/codecs/include/ -I/tmp/codecs/include/ao -I/tmp/codecs/include/lame -I/tmp/codecs/include/ogg -I/tmp/codecs/include/vorbis -I/tmp/codecs/include/theora -I/tmp/codecs/include/vpx' --extra-ldflags='-L/tmp/codecs/lib -static'
  libavutil      52. 63.101 / 52. 63.101
  libavcodec     55. 52.101 / 55. 52.101
  libavformat    55. 32.101 / 55. 32.101
  libavdevice    55.  9.101 / 55.  9.101
  libavfilter     4.  1.102 /  4.  1.102
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'presentation-des-defis_low.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isomiso2
    creation_time   : 2014-06-17 20:14:44
  Duration: 00:05:11.96, start: 0.000000, bitrate: 769 kb/s
    Stream #0:0(eng): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 96 kb/s (default)
    Metadata:
      creation_time   : 2014-06-17 20:14:44
      handler_name    : SoundHandler
    Stream #0:1(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 669 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2014-06-17 20:14:44
      handler_name    : VideoHandler

mkdir ts

ffmpeg -i presentation-des-defis_low.mp4 -flags -global_header -f segment -segment_time 1 -segment_list ts/playlist.m3u8 -segment_list_type m3u8 -segment_format mpegts -map 0 -bsf h264_mp4toannexb -vcodec copy -acodec copy -y ts/media%05d.ts

ffmpeg version N-64012-g61df081 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jun 17 2014 08:50:14 with gcc 4.8 (Ubuntu 4.8.1-2ubuntu1~10.04.1)
  configuration: --disable-ffserver --enable-shared --enable-libx264 --enable-libfaac --enable-libtheora --enable-libvorbis --enable-libmp3lame --enable-libxvid --enable-gpl --enable-nonfree --enable-pthreads --enable-libvpx --extra-cflags='-I/tmp/codecs/include/ -I/tmp/codecs/include/ao -I/tmp/codecs/include/lame -I/tmp/codecs/include/ogg -I/tmp/codecs/include/vorbis -I/tmp/codecs/include/theora -I/tmp/codecs/include/vpx' --extra-ldflags='-L/tmp/codecs/lib -static'
  libavutil      52. 89.100 / 52. 89.100
  libavcodec     55. 67.100 / 55. 67.100
  libavformat    55. 43.100 / 55. 43.100
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  8.100 /  4.  8.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 19.100 /  0. 19.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'presentation-des-defis_low.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isomiso2
    creation_time   : 2014-06-17 20:14:44
  Duration: 00:05:11.96, start: 0.000000, bitrate: 769 kb/s
    Stream #0:0(eng): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 96 kb/s (default)
    Metadata:
      creation_time   : 2014-06-17 20:14:44
      handler_name    : SoundHandler
    Stream #0:1(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 669 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2014-06-17 20:14:44
      handler_name    : VideoHandler
[segment @ 0xa0deac0] Codec for stream 0 does not use global headers but container format requires global headers
[segment @ 0xa0deac0] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, segment, to 'ts/media%05d.ts':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41isomiso2
    encoder         : Lavf55.43.100
    Stream #0:0(eng): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, 96 kb/s (default)
    Metadata:
      creation_time   : 2014-06-17 20:14:44
      handler_name    : SoundHandler
    Stream #0:1(eng): Video: h264 (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], q=2-31, 669 kb/s, 25 fps, 90k tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2014-06-17 20:14:44
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 7799 fps=6601 q=-1.0 Lsize=N/A time=00:05:11.96 bitrate=N/A    
video:25496kB audio:3655kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

ffprobe ts/playlist.m3u

ffprobe version N-60597-g1e5cb42 Copyright (c) 2007-2014 the FFmpeg developers
  built on Feb 14 2014 10:22:18 with gcc 4.8 (Ubuntu 4.8.1-2ubuntu1~10.04.1)
  configuration: --disable-ffserver --enable-shared --enable-libx264 --enable-libfaac --enable-libtheora --enable-libvorbis --enable-libmp3lame --enable-libxvid --enable-gpl --enable-nonfree --enable-pthreads --enable-libvpx --extra-cflags='-I/tmp/codecs/include/ -I/tmp/codecs/include/ao -I/tmp/codecs/include/lame -I/tmp/codecs/include/ogg -I/tmp/codecs/include/vorbis -I/tmp/codecs/include/theora -I/tmp/codecs/include/vpx' --extra-ldflags='-L/tmp/codecs/lib -static'
  libavutil      52. 63.101 / 52. 63.101
  libavcodec     55. 52.101 / 55. 52.101
  libavformat    55. 32.101 / 55. 32.101
  libavdevice    55.  9.101 / 55.  9.101
  libavfilter     4.  1.102 /  4.  1.102
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, hls,applehttp, from 'ts/playlist.m3u8':
  Duration: 00:05:15.51, start: 0.000000, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: aac ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 137 kb/s
    Stream #0:1: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc

As you can see, duration of original video is 00:05:11.96, where the HLS segmented version is 00:05:15.51, a 4 seconds difference (around 13ms per second of added duration)

[fthiery]

Just to be sure, can you reproduce the issue with the sample provided ?

[Stefano Sabatini]

Replying to fthiery:
[...]

ffprobe reports some duration on the m3u8 resource (in my case, 1957.493063s) where the actual duration is 1935.033333s (obtained by ffprob'ing the original mp4 file or using the last frame timestamp + frame duration by analyzing the m3u8 resource with ffprobe). One of the issues is that the duration difference is large enough (~40s for one hour of content) for most players to seek to an unappropriate location (e.g. 10 seconds too late).

Btw original mp4 file has an average gop size of 30 (video is 30 fps), and the goal here is to have 1 second segments for precise seeking.

I think this is due to the way the segment muxer computes the duration of each segment.
We have this code to compute the ending time (libavformat/segment.c):

        seg->cur_entry.end_time =
            FFMAX(seg->cur_entry.end_time, (double)(pkt->pts + pkt->duration) * av_q2d(st->time_base));

The adopted semantics is: total segment duration = maximum ending time of contained packets minus initial time of the segment. If you have several streams it might be that the sum of the segments durations will be greater to the real global duration (for a single stream or for an A/V stream where A/V durations are aligned this shouldn't be an issue, so you won't be able to reproduce the issue with a synthetic testsrc stream).

Note that when computing the HLS duration (libavformat/hls.c) is simply performing the sum of the duration of each segment as read from the EXTINF tag.

A possible solution would be to compute the duration only considering the reference stream packets. Not sure if this is consistent with the specs though.

[Stefano Sabatini]

Should be fixed now in commit:

commit 5acad50056f653672a0e9b6b2986089c32046afb
Author: Stefano Sabatini <stefasab@gmail.com>
Date:   Thu Jul 17 14:49:20 2014 +0200

    lavf/segment: only use reference frames for computing the segment end time
    
    This avoids a systematic overestimate of the segments duration when there
    are several streams.
    
    Fix trac ticket #3724.