Adding a FATE Test

There are two main kinds of FATE test: using existing samples (primarily useful with decoder testing), and using an artificially generated sample for testing.

The recipes for the tests written in GNU Make are located under tests/fate in the FFmpeg source, and the tests themselves are executed through a shell wrapper script tests/fate-run.sh. For the most part you do not need to know anything about the shell script for this wiki page, but we assume moderate knowledge and experience with GNU Make for this wiki page.

Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests. Then, after that, we'll have some notes on testing specific component like filters or decoders.

Three Parts of a Test

A typical test usually consists of four parts:

1. Registration: registering your new test to make fate target;
2. Dependencies: which are usually samples for a format, a generated sample, or a program specifically written to test this feature;
3. Configuration: this part is the most important of all, because it controls how the test will be run.

Registration

One Test for One Feature

The most common case is that only one test is allocated for one feature (e.g. decoder, muxer, hashing algorithm, etc.). Let's open tests/fate/audio.mak, which contains tests for audio codecs, and scroll down to a test named fate-dts. For convenience, you can also see a copy of the recipe below.

FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts
fate-dts: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts.ts
fate-dts: CMP = oneoff
fate-dts: REF = $(SAMPLES)/dts/dts.pcm

This is one of the simplest tests for FFmpeg. Look at the first quoted line. It adds fate-dts to a weird variable containing a call to DEMDEC. The DEMDEC function returns "yes" if the required DEMuxer and DECoder are enabled, in this case the mpegts demuxer and dca decoder. So fate-dts, the target name of the test, gets added into FATE_SAMPLES_AUDIO-yes variable.

Then, if you look at the end of the file:

FATE_SAMPLES_AUDIO += $(FATE_SAMPLES_AUDIO-yes)

FATE_SAMPLES_FFMPEG += $(FATE_SAMPLES_AUDIO)

The enabled tests are added into FATE_SAMPLES_AUDIO and subsequently FATE_SAMPLES_FFMPEG variable, which then of course gets called when doing make fate.

Using this kind of system guarantees that if a person builds FFmpeg with certain parts disabled, the FATE test will skip the disabled parts.

Keep in mind that different tests may have different variable prefixes. Not all tests can be categorized into FATE_SAMPLES_FFMPEG variable.

Notes:

• If only one component for the test you want to add is required, you can replace $(call DEMDEC, MPEGTS, DCA) with $(CONFIG_*).
• There are many functions similar to DEMDEC but offers other checks like encoder and muxer (MUXENC). Check tests/Makefile for more info.

<= 3 Tests for One Feature

Now knowing how to add one test for one feature, it is easy to add multiple tests. For example, you can just add multiple tests like this:

FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-1
fate-dts-1: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-1.ts
fate-dts-1: CMP = oneoff
fate-dts-1: REF = $(SAMPLES)/dts/dts-1.pcm

FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-2
fate-dts-2: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-2.ts
fate-dts-2: CMP = oneoff
fate-dts-2: REF = $(SAMPLES)/dts/dts-2.pcm

However, this approach has some disadvantages:

1. The long FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) is repeated for each test, which reduces readability.
2. It is not possible to do make fate-dts which runs these two tests (of course you can add another target that depends on the two tests, but that is not pretty for code readers).

The two disadvantages become more severe with more tests for this approach.

Lots of Tests for One Feature -- Irregular File Names

Therefore, you can use an additional variable that contains the targets related to a feature, and then use it.

Let's open tests/fate/amrnb.mk, which is a great example of multiple tests with irregular file names. (A part of the file is omitted for clarity below.)

FATE_AMRNB += fate-amrnb-4k75
fate-amrnb-4k75: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/4.75k.amr
fate-amrnb-4k75: REF = $(SAMPLES)/amrnb/4.75k.pcm

FATE_AMRNB += fate-amrnb-5k15
fate-amrnb-5k15: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.15k.amr
fate-amrnb-5k15: REF = $(SAMPLES)/amrnb/5.15k.pcm

FATE_AMRNB += fate-amrnb-5k9
fate-amrnb-5k9: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.9k.amr
fate-amrnb-5k9: REF = $(SAMPLES)/amrnb/5.9k.pcm

[...]

$(FATE_AMRNB): CMP = stddev

FATE_SAMPLES_AVCONV-$(call DEMDEC, AMR, AMRNB) += $(FATE_AMRNB)
fate-amrnb: $(FATE_AMRNB)

So here, first the target names of three tests are added to a variable FATE_AMRNB, and then the

Dependencies

It is very important for you to declare the dependencies of a test if it uses any external files, like special samples or synthetic files.

Configuration

This controls how the test will be run, the parameters of each test and so on. There exist many commands, the basic commands are:

enc_dec_pcm

This command first encodes an input file and then decodes the encoded file. It’s useful to test a audio encoder. An example of this command is fate-vorbis-encode test, which can be found in tests/fate/vorbis.mak. The code of this command is the following:

FATE_VORBIS += fate-vorbis-encode
fate-vorbis-encode: CMD = enc_dec_pcm ogg wav s16le $(TARGET_SAMPLES)/audio-reference/luckynight_2ch_44kHz_s16.wav -c:a vorbis -strict experimental
fate-vorbis-encode: REF = $(SAMPLES)/audio-reference/luckynight_2ch_44kHz_s16.wav
fate-vorbis-encode: CMP_SHIFT = 0
fate-vorbis-encode: CMP_TARGET = 2000
fate-vorbis-encode: SIZE_TOLERANCE = 12123
fate-vorbis-encode: FUZZ = 0
[...]
fate-vorbis-encode: CMP = stddev

• CMD
  • enc_dec_pcm: indicates the command to be executed.
  • ogg: indicates the format in which to encode the input file
  • wav: indicates the format in which to decode the encoded file.
  • s16le: indicates the codec_name.
  • file: the input file
  • @: the other commands to be included

• REF: indicates the reference file, the decoded file will be compared with this file.
• CMP_SHIFT: to calculate the CMP_SHIFT value you can use the tests/tini_psnr command. A guide how to use it is ​explained here.
• CMP_TARGET: it depends on what “CMP= “ is set.
• FUZZ: is the difference that is considered ok between the CMP_TARGET and the actual value.
• SIZE_TOLERANCE is the difference between the input and output sizes. The value of this parameter is calculated by subtracting the size of the two files.

Notes:

The CMP_TARGET and FUZZ may differ to one architecture to another. You may need to adjust these values depending on the architecture.

To do a new test do the following:

1. Compute CMP_SHIFT value with tiny_psnr. The program needs the encoder input and decoder output (that is 2 pcm wav or raw files).
2. Copy the fate test to the .mak file and adjust the CMD options to match your needs.
3. Run the new test, it should show errors like:

   stddev:  296.37 PSNR: 46.89 MAXDIFF: 4810 bytes:  1675800/  1679360
   stddev: |296.37 - 2000| >= 0
   

4. The CMP_TARGET for your test is 296 and your FUZZ is 0.
5. The SIZE_TOLERANCE is 1679360 - 1675800 = 3560.

Filter Tests

Decoder Tests

libavformat Tests

libswresample Tests

This page is licensed under Creative Commons Attribution-ShareAlike 2.0 Generic, 2.5 Generic, 3.0 Unported, or 4.0 International license.