Adding a FATE Test

There are two main kinds of FATE test: using existing samples (primarily useful with decoder testing), and using an artificially generated sample for testing.

The recipes for the tests written in GNU Make are located under tests/fate in the FFmpeg source, and the tests themselves are executed through a shell wrapper script tests/fate-run.sh. For the most part you do not need to know anything about the shell script for this wiki page, but we assume moderate knowledge and experience with GNU Make for this wiki page.

Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests. Then, after that, we'll have some notes on testing specific component like filters or decoders.

Three Parts of a Test

A typical test usually consists of four parts:

1. Registration: registering your new test to make fate target;
2. Dependencies: which are usually samples for a format, a generated sample, or a program specifically written to test this feature;
3. Configuration: this part is the most important of all, because it controls how the test will be run.

Registration

One Test for One Feature

The most common case is that only one test is allocated for one feature (e.g. decoder, muxer, hashing algorithm, etc.). Let's open tests/fate/audio.mak, which contains tests for audio codecs, and scroll down to a test named fate-dts. For convenience, you can also see a copy of the recipe below.

FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts
fate-dts: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts.ts
fate-dts: CMP = oneoff
fate-dts: REF = $(SAMPLES)/dts/dts.pcm

This is one of the simplest tests for FFmpeg. Look at the first quoted line. It adds fate-dts to a weird variable containing a call to DEMDEC. The DEMDEC function returns "yes" if the required DEMuxer and DECoder are enabled, in this case the mpegts demuxer and dca decoder. So fate-dts, the target name of the test, gets added into FATE_SAMPLES_AUDIO-yes variable.

Then, if you look at the end of the file:

FATE_SAMPLES_AUDIO += $(FATE_SAMPLES_AUDIO-yes)

FATE_SAMPLES_FFMPEG += $(FATE_SAMPLES_AUDIO)

The enabled tests are added into FATE_SAMPLES_AUDIO and subsequently FATE_SAMPLES_FFMPEG variable, which then of course gets called when doing make fate.

Using this kind of system guarantees that if a person builds FFmpeg with certain parts disabled, the FATE test will skip the disabled parts.

Keep in mind that different tests may have different variable prefixes. Not all tests can be categorized into FATE_SAMPLES_FFMPEG variable.

Notes:

• If only one component for the test you want to add is required, you can replace $(call DEMDEC, MPEGTS, DCA) with $(CONFIG_*).
• There are many functions similar to DEMDEC but offers other checks like encoder and muxer (MUXENC). Check tests/Makefile for more info.

<= 3 Tests for One Feature

Now knowing how to add one test for one feature, it is easy to add multiple tests. For example, you can just add multiple tests like this:

FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-1
fate-dts-1: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-1.ts
fate-dts-1: CMP = oneoff
fate-dts-1: REF = $(SAMPLES)/dts/dts-1.pcm

FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-2
fate-dts-2: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-2.ts
fate-dts-2: CMP = oneoff
fate-dts-2: REF = $(SAMPLES)/dts/dts-2.pcm

However, this approach has some disadvantages:

1. The long FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) is repeated for each test, which reduces readability.
2. It is not possible to do make fate-dts which runs these two tests (of course you can add another target that depends on the two tests, but that is not pretty for code readers).

The two disadvantages become more severe with more tests for this approach.

Lots of Tests for One Feature -- Irregular File Names

Therefore, you can use an additional variable that contains the targets related to a feature, and then use it.

Let's open tests/fate/amrnb.mk, which is a great example of multiple tests with irregular file names. (A part of the file is omitted for clarity below.)

FATE_AMRNB += fate-amrnb-4k75
fate-amrnb-4k75: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/4.75k.amr
fate-amrnb-4k75: REF = $(SAMPLES)/amrnb/4.75k.pcm

FATE_AMRNB += fate-amrnb-5k15
fate-amrnb-5k15: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.15k.amr
fate-amrnb-5k15: REF = $(SAMPLES)/amrnb/5.15k.pcm

FATE_AMRNB += fate-amrnb-5k9
fate-amrnb-5k9: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.9k.amr
fate-amrnb-5k9: REF = $(SAMPLES)/amrnb/5.9k.pcm

[...]

$(FATE_AMRNB): CMP = stddev

FATE_SAMPLES_AVCONV-$(call DEMDEC, AMR, AMRNB) += $(FATE_AMRNB)
fate-amrnb: $(FATE_AMRNB)

So here, first the target names of three tests are added to a variable FATE_AMRNB, and then the

Dependencies

It is very important for you to declare the dependencies of a test if it uses any external files, like special samples or synthetic files.

Filter Tests

Decoder Tests

libavformat Tests

libswresample Tests

This page is licensed under Creative Commons Attribution-ShareAlike 2.0 Generic, 2.5 Generic, 3.0 Unported, or 4.0 International license.