Regression recovering broken H264 stream

[Alex Agranovsky]

Summary of the bug:

With v3.1.7, the attached H264 input encounters an error, but recovers almost immediately. With v3.3.3 (and current master), it still will recover if AV_CODEC_FLAG2_CHUNKS flag isn't set (which is why ffmpeg works), but will never decode another frame, if the flag is provided.

How to reproduce:

Change ffmpeg.c by adding

ist->dec_ctx->flags2 |= AV_CODEC_FLAG2_CHUNKS;

before avcodec_open2 (not sure if there's a command line option to set it?)

% ffmpeg -threads 1 -i brokenStream.h264 -vcodec mpeg4 foo.mp4

n3.1.7 will recover after encountering an error
concealing 1580 DC, 1580 AC, 1580 MV errors in P frame

n3.3.3 will not

[Carl Eugen Hoyos]

Please provide the input stream.

[Alex Agranovsky]

Here goes. For some reason it did not go through the first time.

​https://www.dropbox.com/s/g28nnu5vksq1q1r/brokenStream.h264?dl=0

[Carl Eugen Hoyos]

Regression since 78c7197ea0e5c53393849a32dd6f49e3b89f7815

$ ffmpeg -flags2 +chunks -threads 1 -i brokenStream_cut.h264 out.mp4
ffmpeg version N-88612-gce52e43 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-gpl
  libavutil      56.  0.100 / 56.  0.100
  libavcodec     58.  1.100 / 58.  1.100
  libavformat    58.  2.100 / 58.  2.100
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  0.101 /  7.  0.101
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, h264, from 'brokenStream_cut.h264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg4 (native))
Press [q] to stop, [?] for help
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf58.2.100
    Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 1280x720, q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.1.100 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
[h264 @ 0x39b7f20] concealing 1580 DC, 1580 AC, 1580 MV errors in P frame
[h264 @ 0x39b7f20] Frame num change from 13 to 14
[h264 @ 0x39b7f20] decode_slice_header error
[h264 @ 0x39b7f20] concealing 1580 DC, 1580 AC, 1580 MV errors in P frame
[h264 @ 0x39b7f20] Frame num change from 13 to 15
[h264 @ 0x39b7f20] decode_slice_header error

...

[h264 @ 0x39b7f20] concealing 1580 DC, 1580 AC, 1580 MV errors in P frame
[h264 @ 0x39b7f20] Frame num change from 13 to 1
[h264 @ 0x39b7f20] decode_slice_header error
[h264 @ 0x39b7f20] concealing 1580 DC, 1580 AC, 1580 MV errors in P frame
[h264 @ 0x39b7f20] Frame num change from 13 to 0
[h264 @ 0x39b7f20] decode_slice_header error
[h264 @ 0x39b7f20] concealing 1580 DC, 1580 AC, 1580 MV errors in P frame
frame=   45 fps=0.0 q=31.0 Lsize=     506kB time=00:00:01.76 bitrate=2356.2kbits/s speed=2.21x
video:505kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.203155%

[Alex Agranovsky]

Hi,

I assume this is a regression in the error recovery, rather than a straight ffmpeg bug. With that assumption, is there a way to explain to the source equipment manufacturer what exactly is broken in the stream they generate, and how they could fix it?

Thanks!

[Alex Agranovsky]

Another instance of broken input

[Simon]

This issue is also causing me problems. The symptoms are exactly the same. In our case the video data is being sent over a network connection that seems to be causing the issues with the stream rather than it being a bad stream generated by the source and as such we have to be as resilient to errors as we possibly can.

[Simon]

H,

Having tested with multiple version while trying to find a build that would work for our code I have narrowed the change down to be between 3.1.11 and 3.2 versions on git and also is in the latest master. I have built using various options and settings using ​https://github.com/jb-alvarado/media-autobuild_suite.git on windows 7. We for now will use 3.1.11 but this bug effectively blocks us from ever updating to a later version of ffmpeg.

Hope this helps and progress can be made towards a fix for this issue.

Thanks!

[Hendrik]

I would strongly recommend to in fact not use the chunked decoding mode, and instead make use of the h264 parser to re-assemble the bitstream. The result should be the same, just using a more thoroughly tested code path.

[Simon]

Replying to heleppkes:

I would strongly recommend to in fact not use the chunked decoding mode, and instead make use of the h264 parser to re-assemble the bitstream. The result should be the same, just using a more thoroughly tested code path.

Hi,

I'm not entirely sure how I would go about making use of the h264 parser to re-assemble the bitstream?

Currently I am reading NAL unit packets using av_read_frame and then passing these packets to avcodec_send_packet and pulling the decoded frames out using avcodec_receive_frame. Is this re-assembing done by adding another call to ffmpeg to request it re-assemble the nal-unit packets back into larger nal-unit packets before passing to avcodec_send_packet or an option we pass in when creating the codec_context to request that ffmpeg does it internally?

If you need any more details please can we discuss directly over emails? My email address is "staircase AT sector DASH alpha DOT net".

Thanks,
Simon

[Alex Agranovsky]

Replying to heleppkes:

I would strongly recommend to in fact not use the chunked decoding mode, and instead make use of the h264 parser to re-assemble the bitstream. The result should be the same, just using a more thoroughly tested code path.

While I appreciate the recommendation, I'm not confident it applies in the cases where a 3rd party/non-FFMPEG network component is being utilized to receive H264 bitstream, and FFMPEG makes first appearance with the decoder.
If I'm wrong, and there's a way to insert a codec parser between receiver and decoder using public FFMPEG API, I'd be thankful for any pointers on how to accomplish that.

[Hendrik]

Replying to staircase27:

I'm not entirely sure how I would go about making use of the h264 parser to re-assemble the bitstream?

Currently I am reading NAL unit packets using av_read_frame

If you are using avformat through av_read_frame already, then you don't need to do anything. Just don't set the chunk decoding flag.

Replying to w3sip:

If I'm wrong, and there's a way to insert a codec parser between receiver and decoder using public FFMPEG API, I'd be thankful for any pointers on how to accomplish that.

Look into av_parser_parse2 and the related functions in avcodec.h, which are used to do this. If you use an internal demuxer it'll do that automatically, if you use an external one (like you seem to be), its best to use that to create a fully reconstructed bitstream using a parser.

[Simon]

Replying to heleppkes:

Replying to staircase27:

I'm not entirely sure how I would go about making use of the h264 parser to re-assemble the bitstream?

Currently I am reading NAL unit packets using av_read_frame

If you are using avformat through av_read_frame already, then you don't need to do anything. Just don't set the chunk decoding flag.

Replying to w3sip:

If I'm wrong, and there's a way to insert a codec parser between receiver and decoder using public FFMPEG API, I'd be thankful for any pointers on how to accomplish that.

Look into av_parser_parse2 and the related functions in avcodec.h, which are used to do this. If you use an internal demuxer it'll do that automatically, if you use an external one (like you seem to be), its best to use that to create a fully reconstructed bitstream using a parser.

I am already using av_read_frame but it does still need the CHUNKS decoding flag for some of the streams we work with. Any suggestions for how to fix this? The streams that need the CHUNKS flag use slices rather than fragmentation of large nal-units.

Any suggestions?

Thanks!

[Balling]

Still a problem.