HEVC fails to seek on keyframe

[SuRGeoNix]

I'm using av_seek_frame and even if it returns 0 pb will not be at an actual keyframe. The issues seems to happen with any HEVC/mpeg-ts format container.

[Format ] MPEG-TS (MPEG-2 Transport Stream)/mpegts
[Video #0] hevc yuv420p10le 3840x2160 @ 59,94
[Audio #1-eng] aac fltp@0 48KHz stereo

I've tested also with ffplay which fails as well with the following output

C:\Users\Owner\Downloads\ffmpegmb\bin>ffplay.exe -ss 00:01:00 -analyzeduration 100000000000 -probesize 100000000000 "c:\root\down\samples\hd\LG Chess 4K Demo.ts"
ffplay version N-103543-gc655a734b1-20210907 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 10-win32 (GCC) 20210408
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libvmaf --enable-vulkan --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libglslang --enable-libgme --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20210907
  libavutil      57.  4.101 / 57.  4.101
  libavcodec     59.  7.102 / 59.  7.102
  libavformat    59.  5.100 / 59.  5.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8.  7.101 /  8.  7.101
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[mpegts @ 00000282a0400480] stream 0 : no PTS found at end of file, duration not set
Input #0, mpegts, from 'c:\root\down\samples\hd\LG Chess 4K Demo.ts':
  Duration: 00:01:52.92, start: 1.066722, bitrate: 62572 kb/s
  Program 1
  Stream #0:0[0x101]: Video: hevc (Main 10) ([36][0][0][0] / 0x0024), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 59.94 fps, 59.94 tbr, 90k tbn
  Stream #0:1[0x102](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 383 kb/s
[aac @ 00000282a5beb040] channel element 0.0 is not allocated
[hevc @ 00000282a8b2d040] Could not find ref with POC 15 0B f=0/0
[hevc @ 00000282a8b2d040] Could not find ref with POC 13
[hevc @ 00000282a8b2d040] Could not find ref with POC 17
[hevc @ 00000282a8b2d040] Could not find ref with POC 21
[hevc @ 00000282a8b2d040] Could not find ref with POC 28 0B f=0/0
  62.26 A-V: -0.025 fd=   3 aq=   47KB vq=11446KB sq=    0B f=0/0

[Balling]

I can confirm that, please see: #7950

It happens due to 4 tiles in the file.

[SuRGeoNix]

Found another issue with HEVC (at least with the same LG Chess 4K Demo.ts) that might be related to this one.

I have implemented a reverse playback by decoding from keyframe until the next one and then seeking to the previous keyframe (Manually ensuring that I actually have keyframe to keyframe).

While decoding I'm draining the decoder properly but it seems that sometimes it misses some frames and I have at least 2-3 frames diff between the last draining frame and the next keyframe. I will do further testing and open a new issue if required.

[Balling]

I can give more samples... Most of them are fixed when copying to mp4! So that means the bug is in MPEG-TS part maybe in keyframes marking in TS itself.

Also you can try to revert fbb283cfefb1865375718c4a56ce608d96a4a8ed that marks Clean Random Access recovery points as keyframes. Recovery points are not mandated to be perfect! So this may be a bug of ATEME encoder, though -c copy fixing it is still strange.

[SuRGeoNix]

I tried reverting (fbb283cfefb1865375718c4a56ce608d96a4a8ed) but did not fix the issue. I did not notice any difference at all.

[Balling]

Such a problem in mpv. ​https://github.com/mpv-player/mpv/issues/9645

​https://github.com/mpv-player/mpv/issues/9639

[Balling]

Does this fix it? ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190507032623.80375-1-ffmpeg@tmm1.net/

[Balling]

Also apparently the whole thing is so complex, it is crazy. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220218232001.345826-5-u@pkh.me/
Patset was applied, did not help.

[Balling]

Another such patch touching the same code!!

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190831101039.2179-2-fumoboy007@me.com/

There are some comments as to how seeking for a real keyframe should work: ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190831101039.2179-1-fumoboy007@me.com/
The bug is that you need to continue searching for keyframes after finding one, because there may be one closer. In mp4 that all happens correctly.

[Carl Eugen Hoyos]

Is this issue only reproducible with ffplay or also with ffmpeg, the application?

[SuRGeoNix]

Maybe by trying exporting a specific frame (which will be a gray / broken as it will fail to seek on keyframe)

ffmpeg -ss 00:00:20 -analyzeduration 100000000000 -probesize 100000000000 -i "LG Chess 4K Demo.ts" -frames:v 1 out.jpg -loglevel debug

[Balling]

SuRGeoNix, I can confirm that command line produces bad image. I propose to use avif though to preserve 10 bit 420 samples losslessly (aside from top-left chroma siting flag, but that is just metadata): -crf 0 -cpu-used 0 out.avif

[Carl Eugen Hoyos]

Should be tested again once #4888 is fixed - for H.264 several development iterations were necessary.

[Balling]

This is a separate issue! It looks like the artifacts happen on the corners of 4 tiles! Because this is Ateme file with 4 tiles.

[Balling]

In other cases we do have separate issues for hevc and avc.

[Balling]

Still happens