h265: Total image distortion even on 1% packet loss

[Vasily Evseenko]

Summary of the bug:
I use rtp as h265 transport and found that even with 1% of packets loss image became totally distorted (static background became gray and only moving parts are shown). It seems that avdec_h265 (based on libav) has a bug because if I use NVidia HW decoder nvh265dec it decode that same stream without any visible artifacts. I've tried to play such file (1% loss) with ffplay and found that the problem is inside ffmpeg.

How to reproduce:

% ffplay h265_0.01_loss.mkv
ffplay version 5.1.3-1 Copyright (c) 2003-2022 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)

If I do the same (1% loss) for h264 codec no such problem occured.
Gstreamer developers redirect me to ffmpeg: ​https://gitlab.freedesktop.org/gstreamer/gst-libav/-/issues/120#note_2089071

[Vasily Evseenko]

h265 test with with 1% packet loss

[Vasily Evseenko]

h264 test with with 1% packet loss

[Vasily Evseenko]

Gstreamer pipeline (for tests):
gst-launch-1.0 videotestsrc is-live=true do-timestamp=true ! 'video/x-raw,width=640,height=480,framerate=30/1,format=I420' ! x265enc tune=zerolatency bitrate=2000 ! rtph265pay mtu=1400 aggregate-mode=zero-latency ! identity drop-probability=0.01 ! queue ! rtph265depay ! h265parse ! avdec_h265 ! videoconvert ! xvimagesink sync=false

[Balling]

based on libav

libav is gone. There is only ffmpeg now.

nvh265dec

We do have -c:v h264_cuvid for legacy cuvid. In you case:

ffplay.exe -vcodec hevc_cuvid -i h265_0.01_loss.mkv

[Vasily Evseenko]

I known about libav so I reproduce the bug using ffplay

Replying to Balling:

based on libav

libav is gone. There is only ffmpeg now.

nvh265dec

Example with NVidia HW decoder was for proof that it is not bug in H265 standard and there are implementations that can decode such stream correctly.

We do have -c:v h264_cuvid for legacy cuvid. In you case:

ffplay.exe -vcodec hevc_cuvid -i h265_0.01_loss.mkv

With "-vcodec hevc_cuvid" and nvidia card stream displayed correctly without gray background

[Balling]

With "-vcodec hevc_cuvid" and nvidia card stream displayed correctly without gray background

I did see that, yes

[Michael Niedermayer]

./ffplay ~/trac/10572/h265_0.01_loss.mkv

shows no artifacts, so i cannot replicate, please reopen with a testcase showing the issue

[Vasily Evseenko]

Hi,
What ffmpeg version, video card and linux distro do you use?
I've tested it now and it still shows grey screen for me
(lenovo x230 (Intel HD Graphics 4000 (IVB GT2)) Debian 12 x86_64, ffmpeg version 5.1.7-0+deb12u1

Replying to Michael Niedermayer:

./ffplay ~/trac/10572/h265_0.01_loss.mkv

shows no artifacts, so i cannot replicate, please reopen with a testcase showing the issue

[quinkblack]

FFmpeg master doesn't have artifacts since

bc1a3bfd2cbc01ffa386312662af8a014890d861

To show the corrupted frames:

ffplay -flags output_corrupt h265_0.01_loss.mkv

Better error concealment is still on my TODO list.

[Vasily Evseenko]

LD_LIBRARY_PATH=/opt/ffmpeg/lib/ /opt/ffmpeg/bin/ffplay /tmp/h265_0.01_loss.mkv using commit f4a87d8ca4118b5130ff6ef332a5f20feb4bb9a7

[Vasily Evseenko]

LD_LIBRARY_PATH=/opt/ffmpeg/lib/ /opt/ffmpeg/bin/ffplay -flags output_corrupt /tmp/h265_0.01_loss.mkv using commit f4a87d8ca4118b5130ff6ef332a5f20feb4bb9a7

[Vasily Evseenko]

I've build latest master of ffmpeg (f4a87d8ca4118b5130ff6ef332a5f20feb4bb9a7) but problem is still here. See attached screen casts above

[svpcom@localhost ~]$ LD_LIBRARY_PATH=/opt/ffmpeg/lib/ /opt/ffmpeg/bin/ffplay -version
ffplay version N-121437-gf4a87d8ca4 Copyright (c) 2003-2025 the FFmpeg developers
built with gcc 12 (Debian 12.2.0-14+deb12u1)
configuration: --prefix=/opt/ffmpeg --enable-shared --enable-gpl --enable-nonfree --enable-version3
libavutil      60. 13.100 / 60. 13.100
libavcodec     62. 16.100 / 62. 16.100
libavformat    62.  6.101 / 62.  6.101
libavdevice    62.  2.100 / 62.  2.100
libavfilter    11.  9.100 / 11.  9.100
libswscale      9.  3.100 /  9.  3.100
libswresample   6.  2.100 /  6.  2.100

[quinkblack]

Replying to Vasily Evseenko:

I've build latest master of ffmpeg (f4a87d8ca4118b5130ff6ef332a5f20feb4bb9a7) but problem is still here. See attached screen casts above

[svpcom@localhost ~]$ LD_LIBRARY_PATH=/opt/ffmpeg/lib/ /opt/ffmpeg/bin/ffplay -version
ffplay version N-121437-gf4a87d8ca4 Copyright (c) 2003-2025 the FFmpeg developers
built with gcc 12 (Debian 12.2.0-14+deb12u1)
configuration: --prefix=/opt/ffmpeg --enable-shared --enable-gpl --enable-nonfree --enable-version3
libavutil      60. 13.100 / 60. 13.100
libavcodec     62. 16.100 / 62. 16.100
libavformat    62.  6.101 / 62.  6.101
libavdevice    62.  2.100 / 62.  2.100
libavfilter    11.  9.100 / 11.  9.100
libswscale      9.  3.100 /  9.  3.100
libswresample   6.  2.100 /  6.  2.100

Yes. It works as expected. If there is a single reference frame is missing/corrupted, all frames affected are dropped by default. The playback is freezed for those frames than a grey screen.

Apple quicktime player use the same logic.

VLC is special, it enable output_corrupted flags by default, unless it use hardware decoder, most of which drop corrupted frames by default.

hevc_cuvid error recovery works very well for this video, but may not be as effective for real-world video footage.

[Balling]

Do we know what ec algorithm is used in hevc_cuvid?

[Vasily Evseenko]

The problem is that in case of h264 there are no any grey screen in such case. Also nvidia and rockchip (mppvideodec) hw decoders decode h265 video streams with partially missing frames without total image corruption

Replying to quinkblack:

Yes. It works as expected. If there is a single reference frame is missing/corrupted, all frames affected are dropped by default. The playback is freezed for those frames than a grey screen.

Apple quicktime player use the same logic.

VLC is special, it enable output_corrupted flags by default, unless it use hardware decoder, most of which drop corrupted frames by default.

hevc_cuvid error recovery works very well for this video, but may not be as effective for real-world video footage.

[quinkblack]

Replying to Vasily Evseenko:

The problem is that in case of h264 there are no any grey screen in such case.

It depends on where the data is lossed (slice header or slice data). H.264 do error canceling for mv.

Also nvidia and rockchip (mppvideodec) hw decoders decode h265 video streams with partially missing frames without total image corruption

There is no image grey screen or image corruption in the video ffplay_default.mp4. It match what reference software does.

It easy to replace the missing reference frame with a nearby frame, and it works very well for this special sample, but unlikely to get stable results for real life videos.

Replying to quinkblack:

Yes. It works as expected. If there is a single reference frame is missing/corrupted, all frames affected are dropped by default. The playback is freezed for those frames than a grey screen.

Apple quicktime player use the same logic.

VLC is special, it enable output_corrupted flags by default, unless it use hardware decoder, most of which drop corrupted frames by default.

hevc_cuvid error recovery works very well for this video, but may not be as effective for real-world video footage.