Opened 5 years ago
Last modified 5 years ago
#8465 new enhancement
avformat_find_stream_info does not fill MPEG2-TS/HEVC resolution with HEVC decoder disabled
Reported by: | Damian Dyńdo | Owned by: | |
---|---|---|---|
Priority: | wish | Component: | avcodec |
Version: | git-master | Keywords: | hevc |
Cc: | ddyndo@vewd.com | Blocked By: | |
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
Summary of the bug:
I've found that if FFmpeg is built with HEVC decoder disabled (or does not have "hevc" decoder on whitelist in CLI), it will fail to acquire basic metadata (e.g. resolution, color characteristics, etc.) even though internal parsers are enabled and do parse NAL units like SPS or VUI from it (and during the call too).
It seems that in most cases it works because at the end of this logic if FFmpeg detects that the parameters are not set, it tries to decode the first sample which then correctly sets resolution and other stream metadata.
How to reproduce:
Either:
- Build FFmpeg without HEVC decoder
- Use FFmpeg API (even example code is fine) - avformat_find_stream_info() function on opened MPEG2-TS/HEVC file
- Watch errors logged and check out that video stream in AVFormatContext does not have basic metadata like resolution, color space, etc. filled correctly.
- If you enable logs on very high level, you will actually see that it does parse SPS NAL unit correctly.
or:
% ffprobe -v 50 -codec_whitelist h265 h1.ts % % ... % [NULL @ 0x3ed7a528ca00] Could not find codec parameters for stream 0 (Video: hevc, 1 reference frame (HEVC / 0x43564548), none): unspecified size % ...
I've used h1.ts file from #3487 (http://trac.ffmpeg.org/ticket/3487) and it reproduces just as well.
Some analysis
I've found out that SPS NAL unit (and others) are parsed on such stack trace:
ff_hevc_decode_nal_sps()
hevc_parse()
av_parser_parse2()
parse_packet()
read_frame_internal()
avformat_find_stream_info()
The problem is that this information parsed there is not propagated higher (outside of the parser (AVCodecParserContext) to the codec (AVCodecContext)). This can be seen in av_parser_parse2()
where there is code like:
% if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) { % FILL(field_order); % }
This is one place in code I've found that does not depend on specific parser/internal data but can rewrite metadata from the parser to AVCodecContext. Adding there something like:
% FILL(width); % FILL(height);
correctly propagates video resolution further. Unfortunately, this seems to be only part of missing metadata because format/pix_fmt is not correctly set/propagated either, and neither are other fields there like AVCodecContext's colorspace, color_trc or color_primaries members.
I wasn't able to find a good place to do this in a clean way. For HEVC I see that such metadata is accessible and partially set in hevc_parser.c
file in hevc_parse_slice_header()
function which looks currently like:
% s->coded_width = ps->sps->width; % s->coded_height = ps->sps->height; % s->width = ps->sps->width - ow->left_offset - ow->right_offset; % s->height = ps->sps->height - ow->top_offset - ow->bottom_offset; % s->format = ps->sps->pix_fmt; % avctx->profile = ps->sps->ptl.general_ptl.profile_idc; % avctx->level = ps->sps->ptl.general_ptl.level_idc;
while it sets the parser's metadata, it is not propagated forward (only profile and level here seems to be propagated forward).
In the end, I think that this can affect negatively performance (fallback to decoding just to get metadata from the stream) and actually makes it impossible to obtain stream information (metadata such as resolution, color space, etc.) through parsers and avformat_find_stream_info()
function and as such should have high priority for fixing it.
Change History (6)
comment:2 by , 5 years ago
And this is an output (with resolution, etc.) if decoder is enabled:
~/Desktop/ffmpeg-git-20190521-amd64-static$ ./ffprobe -v 50 ../h1.ts ffprobe version N-48905-ge45e6005ce-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2007-2019 the FFmpeg developers built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516 configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg libavutil 56. 28.100 / 56. 28.100 libavcodec 58. 52.102 / 58. 52.102 libavformat 58. 27.103 / 58. 27.103 libavdevice 58. 7.100 / 58. 7.100 libavfilter 7. 53.101 / 7. 53.101 libswscale 5. 4.101 / 5. 4.101 libswresample 3. 4.100 / 3. 4.100 libpostproc 55. 4.100 / 55. 4.100 [NULL @ 0x5b88c00] Opening '../h1.ts' for reading [file @ 0x5b89500] Setting default whitelist 'file,crypto' [mpegts @ 0x5b88c00] Format mpegts probed with size=2048 and score=50 [mpegts @ 0x5b88c00] stream=0 stream_type=6 pid=12d prog_reg_desc= [mpegts @ 0x5b88c00] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1 [hevc @ 0x5b8d740] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] Decoding VPS [hevc @ 0x5b8d740] Unknown HEVC profile: 0 [hevc @ 0x5b8d740] Decoding SPS [hevc @ 0x5b8d740] Unknown HEVC profile: 0 [hevc @ 0x5b8d740] Decoding PPS [AVBSFContext @ 0x5bf7e40] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0 [AVBSFContext @ 0x5bf7e40] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0 [AVBSFContext @ 0x5bf7e40] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0 [AVBSFContext @ 0x5bf7e40] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] Decoding VPS [hevc @ 0x5b8d740] Unknown HEVC profile: 0 [hevc @ 0x5b8d740] Decoding SPS [hevc @ 0x5b8d740] Unknown HEVC profile: 0 [hevc @ 0x5b8d740] Decoding PPS [hevc @ 0x5b8d740] Format yuv420p chosen by get_format(). [hevc @ 0x5b8d740] Decoded frame with POC 0. [hevc @ 0x5b8d740] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0 [hevc @ 0x5b8d740] Decoding VPS [hevc @ 0x5b8d740] Unknown HEVC profile: 0 [hevc @ 0x5b8d740] Decoding SPS [hevc @ 0x5b8d740] Unknown HEVC profile: 0 [hevc @ 0x5b8d740] Decoding PPS [hevc @ 0x5b8d740] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0 Last message repeated 121 times [mpegts @ 0x5b88c00] max_analyze_duration 5000000 reached at 5006256 microseconds st:0 [mpegts @ 0x5b88c00] rfps: 23.916667 0.009446 [mpegts @ 0x5b88c00] rfps: 24.000000 0.005342 [mpegts @ 0x5b88c00] rfps: 23.976024 0.003501 [mpegts @ 0x5b88c00] rfps: 47.952048 0.014003 [mpegts @ 0x5b88c00] After avformat_find_stream_info() pos: 0 bytes read:479376 seeks:2 frames:122 Input #0, mpegts, from '../h1.ts': Duration: 00:00:12.64, start: 0.080000, bitrate: 355 kb/s Program 1 Stream #0:0[0x12d], 122, 1/90000: Video: hevc, 1 reference frame (HEVC / 0x43564548), yuv420p(tv), 320x240 (320x256), 0/1, 23.98 tbr, 90k tbn, 90k tbc
comment:3 by , 5 years ago
Component: | undetermined → avcodec |
---|---|
Keywords: | mpeg2ts mpegts metadata removed |
Priority: | normal → wish |
Type: | defect → enhancement |
This is the expected behaviour, a change is not completely out of the question though.
comment:4 by , 5 years ago
Cc: | added |
---|
Hi @cehoyos,
Thanks for the answer. Could you explain two things:
- Why is this considered "intended behavior"? For me, it seems that demuxers and parsers should be enough to parse high-level metadata such as color-related metadata and stream resolution just from the parsers. Furthermore, this information *is* already parsed, just not passed higher (if not present from the other sources). Lastly, it is possibly a performance degradation to require to decode samples just to get information during the
avformat_find_stream_info()
call (when you don't really need to decode anything, the information is just there for "parsing").
- What would be the intended way to get such metadata (having just demuxers and parsers) then?
What is even stranger and makes it totally non-consistent is that when I disable H264/AVC decoder (and leave only demuxer and parser) I get perfectly well metadata from avformat_find_stream_info()
. This can be observed for example on this stream (http://www.avalpa.com/assets/video/OC3.demo.ts) (randomly found on the internet) where it correctly gives me metadata for H264 tracks like:
Stream 7: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), none(progressive), 720x576 (0x0) Stream 8: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), none(progressive), 1920x1080 (0x0)
and the color-related metadata are also correctly filled (although not printed in this log here). As such this really looks like a bug/lack of implementation from my point of view.
Best regards,
Damian Dyńdo.
comment:5 by , 5 years ago
Priority: | wish → normal |
---|---|
Type: | enhancement → defect |
Hi all,
In lieu of the above, I'm changing the type to "defect" again as this has rather high priority for us and we really believe this to be a "defect" rather then "intended behavior". Could you please take another look (especially with regard to what H264 parser does vs H265)?
Best regards,
Damian Dyńdo.
comment:6 by , 5 years ago
Priority: | normal → wish |
---|---|
Type: | defect → enhancement |
*Please note that this reproduces with: demuxer enabled, parser enabled, decoder disabled* which I think is a bug because having demuxers and parsers enabled should be enough to get basic stream information/metadata.
I've also collected logs from another built-from-scratch (this time standalone) ffmpeg/ffprobe:
This shows that a lot of metadata is not parsed/propagated correctly if FFmpeg's HEVC decoder is disabled/compiled-out.