av_parser_parse2 fails on H264 elementary stream

[lordAtticus]

Latest, including 2.x releases of ffmpeg have updated the function h264_find_frame_end in file h264_parser.c. The optimised assembler functions called from above function return incorrect values on x64 and ARM platforms.
Older, c code implementations found in 1.2.6 version of ffmpeg behave correctly. With latest ffmpeg I was unable to parse correctly H264 elementary streams, that contained multiple nalus in the input buffer. I use only avcodec part of ffmpeg programatically, so I don't know how could I trigger ffmpeg to use av_parser to build a test case. I can glady supply a test stream should it be necessary. Also I have tried replacing the asm part with the code from 1.2.6 release and that fixes the problem, so I suspect a bug in the asm.

[Carl Eugen Hoyos]

Please provide a sample.

[lordAtticus]

Uploaded to ​ftp://upload.ffmpeg.org/incoming as ticket_3449.264 as recommended in 'Submitting a Bug report' page.
Please let me know if I can be of further assistance, with a function that calls av_parser perhaps...

[Carl Eugen Hoyos]

(I don't know much about H.264 but I wonder how one raw frame can allow to see a parser problem.)

Please either explain how the problem can be reproduced or point to the change introducing the regression.

[lordAtticus]

It's more about parsing here, rather then actual decoding. The stream I added contains 4 nalus(h264 packet), a PPS, SPS a SEI and an I frame. This makes it valid video and can be played back even in vlc. Parser is meant to be used when we have in same buffer multiple frames. The parser identifies the boundaries between multiple nalus so we can split the buffer and feed it step by step to decoder. So the generic approach in streaming video is to first parse the incoming buffer to identify individual nalus, then feed them individually to the decoder, something like this:

        // parse buffer until complete
        while(inLength) { 
            parsedLength = av_parser_parse2(videoInParserContext, videoInCodecContext,
                                            &outData, &outLength,
                                            inData, inLength,
                                            0, 0, 0);
            inData += parsedLength;
            inLength -= parsedLength;
            if(outLength) {
                av_init_packet(&videoInPacket);
                videoInPacket.data = outData;
                videoInPacket.size = outLength;
                avcodec_decode_video2(videoInCodecContext, videoInFrame, &gotData, &videoInPacket);
                if (gotData) {
                    // display videoInFrame
                }
            }

The problem since 2.x releases is that the parser sets outData to NULL, outSize to 0 and parsedLength to inLength, forcing code to exit the loop and discard data.
Running a debug build I traced the root of the problem to line 67 to 70 in h264_parser.c, if we are to restore the state "7" with code from ffmpeg v1.2.x the problem is gone. However, reverting to old code may not be the solution, since I reckon the asm was introduced to improve performance...

[Carl Eugen Hoyos]

Do I understand you correctly that the problem is not reproducible if you configure with --disable-asm?

[lordAtticus]

After extensive testing, the problem seems to be with me misinterpreting how av_parser works, for example for the sample I submitted, either I input more frames or call the parser with inLength 0, to trigger it to output the previous frame. I have validated many releases v1.2.6 to v2.1.4 all working fine on the ARM platform (both with or without asm). Please don't close the ticket yet I'll get some testing on x64 platform tomorrow.

[lordAtticus]

Ran some tests on x64 platform as well. Ffmpeg works just fine, my bad, sorry. Please close ticket as invalid.

[Carl Eugen Hoyos]

Thank you for testing again!