Accurate seek not possible with MPEG program stream and transport sream

[pkerling]

According to the documentation, the "-ss" ffmpeg option should be frame-accurate. Sadly, this is not the case for MPEG program and transport streams. It always skips to the closest keyframe after the specified timestamp.

How to reproduce:

% ffmpeg -loop 1 -i 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12NgYGAAAAAEAAEnNCcKAAAAAElFTkSuQmCC' -vf scale=640:480,drawtext="fontfile=/usr/share/fonts/DejaVuSans.ttf:text='%{n} %{pts}':x=5:y=5:fontsize=50:fontcolor=yellow:box=1:boxcolor=red" -r 25 -t 00:01:00 -codec:v mpeg2video -g 40 test.mpg
ffmpeg version 2.8.3 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.1.1 (GCC) 20150618 (Red Hat 5.1.1-4)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-frei0r --enable-gnutls --enable-ladspa --enable-libass --enable-libcdio --enable-libdc1394 --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, png_pipe, from 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12NgYGAAAAAEAAEnNCcKAAAAAElFTkSuQmCC':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: png, rgb24(pc), 1x1, 25 fps, 25 tbr, 25 tbn, 25 tbc
[mpeg @ 0x16f59e0] VBV buffer size not set, using default size of 130KB
If you want the mpeg file to be compliant to some specification
Like DVD, VCD or others, make sure you set the correct buffer size
Output #0, mpeg, to 'test.mpg':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: mpeg2video, yuv420p, 640x480, q=2-31, 200 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.60.100 mpeg2video
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
frame= 1500 fps=716 q=21.7 Lsize=    1606kB time=00:00:59.96 bitrate= 219.4kbits/s    
video:1586kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.251059%


% ffmpeg -ss 00:00:05 -i test.mpg -frames:v 1 test.png
ffmpeg version 2.8.3 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.1.1 (GCC) 20150618 (Red Hat 5.1.1-4)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-frei0r --enable-gnutls --enable-ladspa --enable-libass --enable-libcdio --enable-libdc1394 --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mpeg, from 'test.mpg':
  Duration: 00:01:00.00, start: 0.540000, bitrate: 219 kb/s
    Stream #0:0[0x1e0]: Video: mpeg2video (Main), yuv420p(tv), 640x480 [SAR 1:1 DAR 4:3], max. 104857 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
Output #0, image2, to 'test.png':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: png, rgb24, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.60.100 png
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video (native) -> png (native))
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:01.48 bitrate=N/A    
video:17kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

Result: Frame with PTS 6.4 seconds is shown instead of somewhere close to 5 seconds.

[Carl Eugen Hoyos]

The output option -ss is frame-accurate, the input option -ss (generally) can't be.

For future tickets, please remember to test current FFmpeg git head before reporting issues.

[pkerling]

Hey, thanks for your reply. Sorry that I did not write it, but I have tested the git head version and it shows the same behaviour.
If -ss is not supposed to be accurate on input, the documentation should be changed. It currently reads:
"When used as an input option (before -i), seeks in this input file to position. Note that in most formats it is not possible to seek exactly, so ffmpeg will seek to the closest seek point before position. When transcoding and -accurate_seek is enabled (the default), this extra segment between the seek point and position will be decoded and discarded."
This clearly sounds like it should be accurate as the option is even called "-accurate_seek".

[pkerling]

​https://trac.ffmpeg.org/wiki/Seeking also says that input seek is frame-accurate.

[pkerling]

I've looked into the reason for this behaviour, and it seems that with MPEG PS and TS the initial seek is done based on the AVIndexEntry entries which are not accurate. Because the PES header does not include whether the video frame is a key frame or not, the MPEG demuxer marks every frame as keyframe (See the call to av_add_index_entry in mpegps_read_pes_header, libavformat/mpeg.c). The seek code then thinks it's ok to seek there even if it's really a B frame and does not try to search for a preceding keyframe which would normally be necessary. The video decoder throws away everything up to the next keyframe so this is what gets written to the PNG file in the end.
After all, I do think this is a bug, so I'm reopening the ticket. Feel free to close it again if this is really the way it should be.

[Cigaes]

I think your explanation is on the right track but misses part of the problem.

In order to achieve frame-accurate fast-ish seek, ffmpeg fast-seeks to the requested position or before it. Then it slow-seeks to the requested position over non-keyframes.

Unfortunately, when "16b0c92 avconv: Add loop option." was merged as b994788, the setting for seeking backwards was lost.

You can probably get things working again by editing ffmpeg.c and adding AVSEEK_FLAG_BACKWARD to the av_seek_frame() call.

You can also test fcfb66b to see if it works for you. If it does, and I suspect it will, then this issue is a regression.

To really fix things, all the work that was done on seeking beyond the fork needs to be ported to the new location: see the avformat_seek_file() call in ffmpeg_opt.c and the commits b684f744 and b0322e4a.

[pkerling]

Yes, the problem is that ffmpeg seeks past the requested point and not before it. But it's probably not that easy - unfortunately, the suggested solutions do not work. Adding AVSEEK_FLAG_BACKWARD does not change anything. Going back to older versions only makes it worse, depending on the seek time an entirely black PNG file is sometimes written. I tried to go to fcfb66b but could not find this revision in the ffmpeg git repository. Instead, I went back to both one commit before 16b0c92 (11c5f43) and 0e0cefb.

On another note, I can find none of the mentioned commits except 16b0c92. Which repository should I be looking at?

[pkerling]

Also, I do not think that the call to av_seek_frame in ffmpeg.c has anything to do with the problem. It is only used to seek to the actual beginning of the input stream when looping and is not related to input/output seeking.

[pkerling]

As a workaround:

ffmpeg -ss 3.000 -itsoffset -2.000 -copyts -i test.mpg -frames:v 1 test.png

with the -ss value set to two seconds before the requested time gives the correct frame in almost all cases I've tested, the exception being a WMV file with duplicated frames at the beginning (the "normal" way is failing there, too). This then seeks to two seconds before the requested frame to make sure it is not accidentally skipped and then skips two seconds of unwanted frames by adjusting the input timestamps. The timestamp offset should be set bigger than the maximum expected keyframe interval.
That this works depends on a lot of ffmpeg internals like the trim filter automatically inserted and how copyts is handled, but I can recommend this command as the most exact way of extracting a frame known to me as of FFmpeg 2.8.5. Also, the -ss option is necessary even if set to 0.000, or the frame might be wrong.

[Carl Eugen Hoyos]

I see the same output with fcfb66b

[Chris Wiggins]

I've been stumbling over this for the past day or so and am still seeing the problem as of e4d5310a

Anything I can do to help fix this? It's causing problems for me seeking a live HLS stream accurately