Opened 9 days ago

#6899 new enhancement

Create a Bitstream Filter to remove filler material

Reported by: mkver Owned by:
Priority: wish Component: avcodec
Version: git-master Keywords: H264
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

H.264 knows several ways to include filler material to achieve a minimum/constant bitrate:
a) With filler data NAL units (nal_unit_type 12; see section 7.3.2.7 of the standard).
b) With filler payload SEI messages (SEI payloadType 3; see D.1.4 of the standard).
c) The annex B format allows the insertion of 0x00 in between two NAL units (called trailing_zero_8bits; see B.1.1 of the standard). Given that these trailing_zero_8bits actually aren't part of the NAL units (but only of what is called a byte_stream_nal_unit in annex B) and given that ISO/IEC 14496-15 only speaks of NAL units that should be stored in mp4 (and hence matroska) these trailing_zero_8bits are probably illegal in mp4/matroska (although decoders don't seem to care).

It would be beneficial if there were a bitstream filter to strip these filler bytes away. Notice that the trailing_zero_8bits may actually be present between two NAL units of the same access unit so that the chomp bitstream filter is not sufficient for stripping it away. I further add that filler data NAL units (case a)) mustn't be present in an mp4 when using the sample entry "avc1" or "avc2" and is discouraged in any case.

My sample contains both cases a) and c): I remuxed only the video track with ffmpeg, with ffmpeg with the chomp bitstream filter and with mkvmerge 18.0. Target format is Matroska. Here is mkvinfo's output for the first remux with ffmpeg without chomp:

Track 1: video, codec ID: V_MPEG4/ISO/AVC (h.264 profile: High @L4.0), mkvmerge/mkvextract track ID: 0, language: und, default duration: 20.000ms (50.000 frames/fields per second for a video track), pixel width: 1280, pixel height: 720
I frame, track 1, timestamp 400 (00:00:00.400), size 101483, adler 0x6b64579f
P frame, track 1, timestamp 320 (00:00:00.320), size 31064, adler 0xa61049b5
P frame, track 1, timestamp 260 (00:00:00.260), size 2953, adler 0x3ffda73e
P frame, track 1, timestamp 280 (00:00:00.280), size 6987, adler 0x31d09004
P frame, track 1, timestamp 300 (00:00:00.300), size 7304, adler 0xd79b426f
P frame, track 1, timestamp 340 (00:00:00.340), size 2456, adler 0xec1ab18c
P frame, track 1, timestamp 360 (00:00:00.360), size 5651, adler 0x6fc328a6
P frame, track 1, timestamp 380 (00:00:00.380), size 106940, adler 0x1820cd2b
P frame, track 1, timestamp 520 (00:00:00.520), size 44470, adler 0x117a6aee
P frame, track 1, timestamp 420 (00:00:00.420), size 1514, adler 0x9acbece0
P frame, track 1, timestamp 440 (00:00:00.440), size 6259, adler 0x39551c3e
P frame, track 1, timestamp 460 (00:00:00.460), size 6382, adler 0x585b700e
P frame, track 1, timestamp 480 (00:00:00.480), size 6460, adler 0x7ca4a027
P frame, track 1, timestamp 500 (00:00:00.500), size 96555, adler 0x57831fa8
P frame, track 1, timestamp 640 (00:00:00.640), size 48782, adler 0xafa42b60
P frame, track 1, timestamp 540 (00:00:00.540), size 2519, adler 0xbc5edb96
P frame, track 1, timestamp 560 (00:00:00.560), size 7187, adler 0x2d86ddb2
P frame, track 1, timestamp 580 (00:00:00.580), size 7522, adler 0x23c5b5b0
P frame, track 1, timestamp 600 (00:00:00.600), size 7560, adler 0x5b32ee66
P frame, track 1, timestamp 620 (00:00:00.620), size 112757, adler 0x4b2de930
P frame, track 1, timestamp 760 (00:00:00.760), size 49073, adler 0x1bd8788a
P frame, track 1, timestamp 660 (00:00:00.660), size 2647, adler 0x1a242642
P frame, track 1, timestamp 680 (00:00:00.680), size 6884, adler 0xc1db4151
P frame, track 1, timestamp 700 (00:00:00.700), size 7899, adler 0x38575d09
P frame, track 1, timestamp 720 (00:00:00.720), size 7121, adler 0x97820327
P frame, track 1, timestamp 740 (00:00:00.740), size 82902, adler 0xf7fdb745
P frame, track 1, timestamp 880 (00:00:00.880), size 53367, adler 0xaa5794ac
P frame, track 1, timestamp 780 (00:00:00.780), size 2781, adler 0xbcd56f48
P frame, track 1, timestamp 800 (00:00:00.800), size 8008, adler 0xc745b3f0
P frame, track 1, timestamp 820 (00:00:00.820), size 8178, adler 0xd8040a07
P frame, track 1, timestamp 840 (00:00:00.840), size 8275, adler 0xa25a06cf
P frame, track 1, timestamp 860 (00:00:00.860), size 96050, adler 0x3fe87a58
I frame, track 1, timestamp 1040 (00:00:01.040), size 105371, adler 0x3078dd18
P frame, track 1, timestamp 960 (00:00:00.960), size 23562, adler 0xf4e29885
P frame, track 1, timestamp 900 (00:00:00.900), size 3536, adler 0x820bca7c
P frame, track 1, timestamp 920 (00:00:00.920), size 6821, adler 0x82be2225
P frame, track 1, timestamp 940 (00:00:00.940), size 7833, adler 0xd9535524
P frame, track 1, timestamp 980 (00:00:00.980), size 101682, adler 0xc312e7c5
P frame, track 1, timestamp 1000 (00:00:01.000), size 5608, adler 0xba71fd6c
P frame, track 1, timestamp 1020 (00:00:01.020), size 6591, adler 0x42edd90f
P frame, track 1, timestamp 1160 (00:00:01.160), size 46410, adler 0x273f3480
P frame, track 1, timestamp 1060 (00:00:01.060), size 1847, adler 0x21f681f1
P frame, track 1, timestamp 1080 (00:00:01.080), size 7032, adler 0x825ecedd
P frame, track 1, timestamp 1100 (00:00:01.100), size 4761, adler 0xe5bd1295

Here is the output for the second case (with the chomp bitstream filter):

Track 1: video, codec ID: V_MPEG4/ISO/AVC (h.264 profile: High @L4.0), mkvmerge/mkvextract track ID: 0, language: und, default duration: 20.000ms (50.000 frames/fields per second for a video track), pixel width: 1280, pixel height: 720
I frame, track 1, timestamp 400 (00:00:00.400), size 101479, adler 0xdd64579b
P frame, track 1, timestamp 320 (00:00:00.320), size 31064, adler 0xa61049b5
P frame, track 1, timestamp 260 (00:00:00.260), size 2946, adler 0x5cf0a737
P frame, track 1, timestamp 280 (00:00:00.280), size 6982, adler 0xd97c8fff
P frame, track 1, timestamp 300 (00:00:00.300), size 7299, adler 0xfd1d426a
P frame, track 1, timestamp 340 (00:00:00.340), size 2451, adler 0x44acb187
P frame, track 1, timestamp 360 (00:00:00.360), size 5646, adler 0x1dd729a0
P frame, track 1, timestamp 380 (00:00:00.380), size 106940, adler 0x1820cd2b
P frame, track 1, timestamp 520 (00:00:00.520), size 44466, adler 0xaef76aea
P frame, track 1, timestamp 420 (00:00:00.420), size 1509, adler 0xdd01ecdb
P frame, track 1, timestamp 440 (00:00:00.440), size 6259, adler 0x39551c3e
P frame, track 1, timestamp 460 (00:00:00.460), size 6375, adler 0x99ea7007
P frame, track 1, timestamp 480 (00:00:00.480), size 6454, adler 0x24a6a021
P frame, track 1, timestamp 500 (00:00:00.500), size 96555, adler 0x57831fa8
P frame, track 1, timestamp 640 (00:00:00.640), size 48775, adler 0x4a772b59
P frame, track 1, timestamp 540 (00:00:00.540), size 2514, adler 0x4174db91
P frame, track 1, timestamp 560 (00:00:00.560), size 7183, adler 0x2934dead
P frame, track 1, timestamp 580 (00:00:00.580), size 7515, adler 0x5e9cb5a9
P frame, track 1, timestamp 600 (00:00:00.600), size 7556, adler 0x2b96ee62
P frame, track 1, timestamp 620 (00:00:00.620), size 112757, adler 0x4b2de930
P frame, track 1, timestamp 760 (00:00:00.760), size 49067, adler 0xca8e7884
P frame, track 1, timestamp 660 (00:00:00.660), size 2639, adler 0x960e263a
P frame, track 1, timestamp 680 (00:00:00.680), size 6884, adler 0xc1db4151
P frame, track 1, timestamp 700 (00:00:00.700), size 7899, adler 0x38575d09
P frame, track 1, timestamp 720 (00:00:00.720), size 7114, adler 0xbf5a0320
P frame, track 1, timestamp 740 (00:00:00.740), size 82902, adler 0xf7fdb745
P frame, track 1, timestamp 880 (00:00:00.880), size 53367, adler 0xaa5794ac
P frame, track 1, timestamp 780 (00:00:00.780), size 2777, adler 0xd47b6f44
P frame, track 1, timestamp 800 (00:00:00.800), size 8004, adler 0x7a90b3ec
P frame, track 1, timestamp 820 (00:00:00.820), size 8178, adler 0xd8040a07
P frame, track 1, timestamp 840 (00:00:00.840), size 8271, adler 0x062a06cb
P frame, track 1, timestamp 860 (00:00:00.860), size 96050, adler 0x3fe87a58
I frame, track 1, timestamp 1040 (00:00:01.040), size 105367, adler 0x4fb6dd14
P frame, track 1, timestamp 960 (00:00:00.960), size 23562, adler 0xf4e29885
P frame, track 1, timestamp 900 (00:00:00.900), size 3532, adler 0x2106ca78
P frame, track 1, timestamp 920 (00:00:00.920), size 6821, adler 0x82be2225
P frame, track 1, timestamp 940 (00:00:00.940), size 7833, adler 0xd9535524
P frame, track 1, timestamp 980 (00:00:00.980), size 101682, adler 0xc312e7c5
P frame, track 1, timestamp 1000 (00:00:01.000), size 5604, adler 0x6d3dfd68
P frame, track 1, timestamp 1020 (00:00:01.020), size 6591, adler 0x42edd90f
P frame, track 1, timestamp 1160 (00:00:01.160), size 46410, adler 0x273f3480
P frame, track 1, timestamp 1060 (00:00:01.060), size 1847, adler 0x21f681f1
P frame, track 1, timestamp 1080 (00:00:01.080), size 7027, adler 0xeeddced8
P frame, track 1, timestamp 1100 (00:00:01.100), size 4761, adler 0xe5bd1295

And here is the output for the file remuxed with mkvmerge (mkvmerge already strips a) and c) away); the timestamps are different, but the underlying frames are the same:

Track 1: video, codec ID: V_MPEG4/ISO/AVC (h.264 profile: High @L4.0), mkvmerge/mkvextract track ID: 0, language: und, pixel width: 1280, pixel height: 720, display width: 1280, display height: 720, default duration: 20.000ms (50.000 frames/fields per second for a video track)
I frame, track 1, timestamp 1716 (00:00:01.716), size 101330, adler 0x5c562bc4
P frame, track 1, timestamp 1636 (00:00:01.636), size 22958, adler 0xad78caca
B frame, track 1, timestamp 1576 (00:00:01.576), size 2940, adler 0x4732a6dc
B frame, track 1, timestamp 1596 (00:00:01.596), size 6976, adler 0x28c78fa4
B frame, track 1, timestamp 1616 (00:00:01.616), size 7293, adler 0xdbaa420f
B frame, track 1, timestamp 1656 (00:00:01.656), size 2445, adler 0xdee3b12c
B frame, track 1, timestamp 1676 (00:00:01.676), size 5640, adler 0x48192945
B frame, track 1, timestamp 1696 (00:00:01.696), size 5790, adler 0x2d3d34a8
P frame, track 1, timestamp 1836 (00:00:01.836), size 44460, adler 0xa5ad6aaf
B frame, track 1, timestamp 1736 (00:00:01.736), size 1503, adler 0xc630ec80
B frame, track 1, timestamp 1756 (00:00:01.756), size 6253, adler 0x89a11be3
B frame, track 1, timestamp 1776 (00:00:01.776), size 6369, adler 0xc0fa6fac
B frame, track 1, timestamp 1796 (00:00:01.796), size 6448, adler 0x2fa19fc6
B frame, track 1, timestamp 1816 (00:00:01.816), size 6452, adler 0x35637dce
P frame, track 1, timestamp 1956 (00:00:01.956), size 48769, adler 0x5fda2b1e
B frame, track 1, timestamp 1856 (00:00:01.856), size 2508, adler 0xc546db36
B frame, track 1, timestamp 1876 (00:00:01.876), size 7177, adler 0x30fdde52
B frame, track 1, timestamp 1896 (00:00:01.896), size 7509, adler 0xf052b54e
B frame, track 1, timestamp 1916 (00:00:01.916), size 7550, adler 0xaeb9ee07
B frame, track 1, timestamp 1936 (00:00:01.936), size 7598, adler 0xca03aeb5
P frame, track 1, timestamp 2076 (00:00:02.076), size 49061, adler 0x9ca57849
B frame, track 1, timestamp 1976 (00:00:01.976), size 2633, adler 0xed7125df
B frame, track 1, timestamp 1996 (00:00:01.996), size 6878, adler 0x33fc40f6
B frame, track 1, timestamp 2016 (00:00:02.016), size 7893, adler 0x418d5cae
B frame, track 1, timestamp 2036 (00:00:02.036), size 7108, adler 0xdfaa02c5
B frame, track 1, timestamp 2056 (00:00:02.056), size 7146, adler 0x9962f0e6
P frame, track 1, timestamp 2196 (00:00:02.196), size 53361, adler 0x9d2e9471
B frame, track 1, timestamp 2096 (00:00:02.096), size 2771, adler 0xfad06ee9
B frame, track 1, timestamp 2116 (00:00:02.116), size 7998, adler 0x5e73b391
B frame, track 1, timestamp 2136 (00:00:02.136), size 8172, adler 0x7e0d09ac
B frame, track 1, timestamp 2156 (00:00:02.156), size 8265, adler 0x8b150670
B frame, track 1, timestamp 2176 (00:00:02.176), size 8920, adler 0x56cc6e3a
I frame, track 1, timestamp 2356 (00:00:02.356), size 105361, adler 0xe4a9dcf9
P frame, track 1, timestamp 2276 (00:00:02.276), size 23556, adler 0x3d28982a
B frame, track 1, timestamp 2216 (00:00:02.216), size 3526, adler 0x3aebca1d
B frame, track 1, timestamp 2236 (00:00:02.236), size 6815, adler 0x0b4421ca
B frame, track 1, timestamp 2256 (00:00:02.256), size 7827, adler 0xf9ff54c9
B frame, track 1, timestamp 2296 (00:00:02.296), size 2931, adler 0x90c19dda
B frame, track 1, timestamp 2316 (00:00:02.316), size 5598, adler 0xa66dfd0d
B frame, track 1, timestamp 2336 (00:00:02.336), size 6585, adler 0x1d35d8b4
P frame, track 1, timestamp 2476 (00:00:02.476), size 46404, adler 0x5dcf3445
B frame, track 1, timestamp 2376 (00:00:02.376), size 1841, adler 0x92f08196
B frame, track 1, timestamp 2396 (00:00:02.396), size 7021, adler 0x2e29ce7d
B frame, track 1, timestamp 2416 (00:00:02.416), duration 60.000, size 4755, adler 0x4ab4123a

The size of the files are 1.268.229 B, 1.268.099 B and 706.616 B, respectively. As one sees, there are several packets where the mkvmerge version is much smaller than the other two; these are the packets with filler data NAL units (case a)). For the frames containing filler data NAL units the application of the chomp bitstream filter was useless -- because the trailing_zero_8bits are between the video NAL unit and the filler data NAL unit and are threrefore not affected by it. This confirms what I said about the insufficiency of the chomp bitstream filter for dealing with c). The remaining difference is due to mkvmerge stripping the Access Unit Delimiter NAL unit away (each amounts to 6B) and mkvmerge stripping the SPS and PPS of the first keyframe away and putting it into the CodecPrivate?.
While this difference of more than 40% may seem gigantic, it isn't in my experience. (I am comparing it to the CBR satellite streams I know, not to means of transportation that don't have a cbr requirement and therefore don't use filler data.)

Attachments (3)

Filler.Bytes.ts (2.1 MB) - added by mkver 9 days ago.
ffmpeg-20171207-070926.log (38.9 KB) - added by mkver 9 days ago.
FFmpeg-log of remuxing Filler.Bytes.ts without chomp.
ffmpeg-20171207-070944.log (39.2 KB) - added by mkver 9 days ago.
FFmpeg-log of remuxing Filler.Bytes.ts with the chomp bitstream filter.

Change History (3)

Changed 9 days ago by mkver

Changed 9 days ago by mkver

FFmpeg-log of remuxing Filler.Bytes.ts without chomp.

Changed 9 days ago by mkver

FFmpeg-log of remuxing Filler.Bytes.ts with the chomp bitstream filter.

Note: See TracTickets for help on using tickets.