dump_extra Bitstream Filter doesn't work with H.264 in mp4-format

[mkver]

When using the dump_extra bitstream filter on a H.264 track inside a Matroska file (i.e. the H.264 isn't annex b, but in mp4-format), the CodecPrivate of the track gets simply dumped to the beginning of the Matroska packets. Instead of this every SPS and PPS inside the CodecPrivate should be extracted from the CodecPrivate and prefixed with the length of the respective NAL unit and then dumped to the beginning of the packet.
Given that the first bytes of the packet's payload will be considered the size of a NAL unit, the stream is unreadable afterwards.
How to reproduce:

ffmpeg started on 2017-11-25 at 12:47:16
Report written to "ffmpeg-20171125-124716.log"
ffmpeg version N-89212-ga60b2425c3 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx
  libavutil      56.  2.100 / 56.  2.100
  libavcodec     58.  3.105 / 58.  3.105
  libavformat    58.  2.102 / 58.  2.102
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  2.102 /  7.  2.102
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
Input #0, matroska,webm, from 'No.Inband.Header.mkv':
  Metadata:
    encoder         : Haali Matroska Writer b0
  Duration: 00:00:10.00, start: 0.000000, bitrate: 6 kb/s
    Stream #0:0(eng): Video: h264 (High), yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 20k tbn, 48 tbc (default)
Output #0, matroska, to 'Inband.Header.mkv':
  Metadata:
    encoder         : Lavf58.2.102
    Stream #0:0(eng): Video: h264 (High) (H264 / 0x34363248), yuv420p(progressive), 640x480 [SAR 1:1 DAR 4:3], q=2-31, 24 fps, 24 tbr, 1k tbn, 20k tbc (default)
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=  240 fps=0.0 q=-1.0 Lsize=       9kB time=00:00:09.87 bitrate=   7.2kbits/s speed=2.47e+003x    
video:6kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 33.679615%

And this shows that the created file is indeed defective:

ffmpeg started on 2017-11-25 at 12:48:17
Report written to "ffmpeg-20171125-124817.log"
ffmpeg version N-89212-ga60b2425c3 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx
  libavutil      56.  2.100 / 56.  2.100
  libavcodec     58.  3.105 / 58.  3.105
  libavformat    58.  2.102 / 58.  2.102
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  2.102 /  7.  2.102
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.101 /  3.  0.101
  libpostproc    55.  0.100 / 55.  0.100
[h264 @ 0000000000159be0] Invalid NAL unit size (23330846 > 834).
[h264 @ 0000000000159be0] missing picture in access unit with size 838
[h264 @ 0000000000159be0] Invalid NAL unit size (23330846 > 144).
[h264 @ 0000000000159be0] missing picture in access unit with size 148
[h264 @ 0000000000159be0] Invalid NAL unit size (23330846 > 144).
[h264 @ 0000000000159be0] missing picture in access unit with size 148
[h264 @ 0000000000159be0] Invalid NAL unit size (23330846 > 144).
[h264 @ 0000000000159be0] missing picture in access unit with size 148
Input #0, matroska,webm, from 'Inband.Header.mkv':
  Metadata:
    ENCODER         : Lavf58.2.102
  Duration: 00:00:10.00, start: 0.000000, bitrate: 7 kb/s
    Stream #0:0(eng): Video: h264 (High), yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 48 tbc (default)
    Metadata:
      DURATION        : 00:00:10.000000000
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Finishing stream 0:0 without any data written to it.
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.2.102
    Stream #0:0(eng): Video: wrapped_avframe, yuv420p, 640x480, q=2-31, 200 kb/s, 24 fps, 24 tbn, 24 tbc (default)
    Metadata:
      DURATION        : 00:00:10.000000000
      encoder         : Lavc58.3.105 wrapped_avframe
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

[James]

You should use the h264_mp4toannexb bitstream filter for this, not dump_extradata

[mkver]

Replying to jamrial:

You should use the h264_mp4toannexb bitstream filter for this, not dump_extradata

This was actually the first thing I checked and it didn't work (with a different sample). But now I did this with the sample I uploaded here and it worked. After creating a different sample I found out that the h264_mp4toannexb bitstream filter seems to insert the SPS/PPS only in front of IDR frames, not I frames with a recovery point SEI message (these are real keyframes and some streams contain only those). But this would be a different ticket. Returning to this one: Is the behaviour I encountered intended behaviour? Or does this count as a bug?

[James]

It is the intended behavior of dump_extradata. As the name suggests, it simply dumps the raw contents of the extradata at the beginning of every keyframe or every packet, depending on the value passed to the freq AVOption.

And yes, open a new ticket for what you suggest h264_mp4toannexb should be doing instead.