62bd8deef causes a regression in HEVC decoding / hevc needs support to skip initial corrupt frames

[Ståle Kristoffersen]

Summary of the bug:
I'm trying to extract a thumb from the middle of a live TS, containing one
HEVC-stream.
Before this patch
I would always get a good looking thumb, but with this patch I would for
the most part get a mostly gray frame.

If I generates a few more thumbnails i see that it eventually spits out the
same frame that ffmpeg before this patch produced as the first thumb, and
it looks OK.

However it is much more blocky than what the previous ffmpeg produced.

How to reproduce (b7baebb75 is the commit just before the patch):

% ./ffmpeg.b7baebb75 -an -i hevc_fail.ts -vframes 1 before_patch%d.jpg
ffmpeg version N-74781-gb7baebb Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (Debian 4.7.2-5)
  configuration: 
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
[NULL @ 0x1f74780] PPS id out of range: 0
[hevc @ 0x1f74780] PPS id out of range: 0
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
[hevc @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1f74780] Error parsing NAL unit #2.
Input #0, mpegts, from 'hevc_fail.ts':
  Duration: 00:00:02.50, start: 2237.360033, bitrate: 7740 kb/s
    Stream #0:0[0x1e1]: Video: hevc (Main 10), yuv420p10le(tv), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 23.98 tbc
[swscaler @ 0x1f9a140] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to 'before_patch%d.jpg':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: mjpeg, yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc
    Metadata:
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[NULL @ 0x1f74780] PPS id out of range: 0
[hevc @ 0x2052600] PPS id out of range: 0
[hevc @ 0x2052600] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x20609a0] PPS id out of range: 0
[hevc @ 0x20609a0] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
[hevc @ 0x2072400] PPS id out of range: 0
[hevc @ 0x2072400] Error parsing NAL unit #2.
[hevc @ 0x2083e60] PPS id out of range: 0
[hevc @ 0x2083e60] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x20958c0] PPS id out of range: 0
[hevc @ 0x20958c0] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
[hevc @ 0x20a7320] PPS id out of range: 0
[hevc @ 0x20a7320] Error parsing NAL unit #2.
[hevc @ 0x20b8d80] PPS id out of range: 0
[hevc @ 0x20b8d80] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x20ca7e0] PPS id out of range: 0
[hevc @ 0x20ca7e0] Error parsing NAL unit #2.
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x20dc240] PPS id out of range: 0
[hevc @ 0x20dc240] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x2052600] PPS id out of range: 0
[hevc @ 0x2052600] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x20609a0] PPS id out of range: 0
[hevc @ 0x20609a0] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x2072400] PPS id out of range: 0
[hevc @ 0x2072400] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x2083e60] PPS id out of range: 0
[hevc @ 0x2083e60] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x20958c0] PPS id out of range: 0
[hevc @ 0x20958c0] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x20a7320] PPS id out of range: 0
[hevc @ 0x20a7320] Error parsing NAL unit #2.
[NULL @ 0x1f74780] PPS id out of range: 0
Error while decoding stream #0:0: Invalid data found when processing input
[hevc @ 0x20b8d80] PPS id out of range: 0
[hevc @ 0x20b8d80] Error parsing NAL unit #2.
Error while decoding stream #0:0: Invalid data found when processing input
    Last message repeated 7 times
frame=    1 fps=0.0 q=7.5 Lsize=N/A time=00:00:00.75 bitrate=N/A    
video:108kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

This gives me one image, before_patch1.jpg that looks completly fine.
​http://kolbu.ws/~chiller/ffmpeg_bug/before_patch1.jpg

Now I do the same with the patch:

ffmpeg version N-74782-g62bd8de Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.7 (Debian 4.7.2-5)
  configuration: 
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
[NULL @ 0x1dc2780] PPS id out of range: 0
[hevc @ 0x1dc2780] PPS id out of range: 0
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
[hevc @ 0x1dc2780] PPS id out of range: 0
    Last message repeated 1 times
[hevc @ 0x1dc2780] Error parsing NAL unit #2.
Input #0, mpegts, from 'hevc_fail.ts':
  Duration: 00:00:02.50, start: 2237.360033, bitrate: 7740 kb/s
    Stream #0:0[0x1e1]: Video: hevc (Main 10), yuv420p10le(tv), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 23.98 tbc
[swscaler @ 0x1dec4e0] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to 'with_patch%d.jpg':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: mjpeg, yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc
    Metadata:
      encoder         : Lavc56.60.100 mjpeg
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[hevc @ 0x1ea0760] Could not find ref with POC 12
[hevc @ 0x1ea0760] Could not find ref with POC 15
frame=    1 fps=0.0 q=5.0 Lsize=N/A time=00:00:00.04 bitrate=N/A    
video:69kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

This gives me one file with_patch1.jpg that looks garbaged.
​http://kolbu.ws/~chiller/ffmpeg_bug/with_patch1.jpg

If I re-run the last command with -vframes 100 it generates 60 images, where image 18 is almost identical to the before_patch1.jpg, except a bit more blocky.

​http://kolbu.ws/~chiller/ffmpeg_bug/with_patch18.jpg

Here is the link to the .ts-file:
​http://kolbu.ws/~chiller/ffmpeg_bug/hevc_fail.ts

and all files involved can be viewed here:
​http://kolbu.ws/~chiller/ffmpeg_bug/
(The most important ones are attached)

[Hendrik]

Thinking about this some, I think this is an unfortunate side-effect of fixing the parser bug that we had before.

Without the patch, the demuxer would be unable to extract SPS/PPS/VPS from the stream, and thus the decoder would be incapable of actually decoding early frames in the stream - it would only start decoding on an IDR frame, which also includes the SPS/PPS/VPS again.

Now, the demuxer can extract SPS/PPS/VPS, and as such, the decoder can start decoding right away - even if the first frames are missing references, and as such will decode corrupted.

Unfortunately, there is no quick fix, as having the SPS/PPS/VPS is quite essential for many other workflows. The proper fix would be to teach the HEVC decoder to discard corrupted frames.

Anyway, I'm not 100% sure this is the full explanation, but its the most reasonable one that comes to mind.

[Carl Eugen Hoyos]

Replying to heleppkes:

Unfortunately, there is no quick fix, as having the SPS/PPS/VPS is quite essential for many other workflows. The proper fix would be to teach the HEVC decoder to discard corrupted frames.

So this is just a feature request to implement discarding corrupted initial frames if -flags2 +showall was not specified?

[Hendrik]

Thats my interpretation of the problem. The HEVC decoder quite certainly doesn't have that yet.
It worked "by accident" for him before, due to the missing extradata (as that caused any frames to get discarded)

[Carl Eugen Hoyos]

Seeking backwards with older FFplay confirms your interpretation afaict.

[Ståle Kristoffersen]

What about the difference between before_patch1.jpg and with_patch18.jpg? Why is the second one lacking very much details, even if they are the same frame?

What would one have to implement to have it skip the initial frames? Could I do it based on how it is done in h264 or is it more involving?

[Carl Eugen Hoyos]

Replying to staalebk:

What about the difference between before_patch1.jpg and with_patch18.jpg? Why is the second one lacking very much details, even if they are the same frame?

I don't think it is the same frame, as said please test older ffplay with seeking back to the beginning of the given stream to see the deeper issue.

What would one have to implement to have it skip the initial frames? Could I do it based on how it is done in h264 or is it more involving?

It was extremely involving to implement skipping initial corrupt frames for h264 iirc (I don't know if the same concept of recovery points exists in hevc).