Changes between Version 6 and Version 7 of FATE/AddingATest

Timestamp:

Apr 14, 2014, 3:22:47 AM (12 years ago)

Author:

Timothy Gu

Comment:

--

FATE/AddingATest

@@ -3 +3 @@
 There are two main kinds of FATE test: using existing samples (primarily useful with decoder testing), and using an artificially generated sample for testing.
 
-The recipes for the tests written in GNU Make are located under `tests/fate` in the FFmpeg source. Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests.
+The recipes for the tests written in GNU Make are located under `tests/fate` in the FFmpeg source, and the tests themselves are executed through a shell wrapper script `tests/fate-run.sh`. For the most part you do not need to know anything about the shell script for this wiki page, but '''we assume moderate knowledge and experience with GNU Make''' for this wiki page.
 
-== Four Parts of a Test ==
+Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests. Then, after that, we'll have some notes on testing specific component like filters or decoders.
+
+== Three Parts of a Test ==
 
 A typical test usually consists of four parts:
 
-1. Registration: meaning that `make fate` automatically execute the test you want to add;
+1. Registration: registering your new test to `make fate` target;
 2. Dependencies: which are usually samples for a format, a generated sample, or a program specifically written to test this feature;
 3. Configuration: this part is the most important of all, because it controls how the test will be run.
 
-Let us focus on the separate parts of a test.
-
 === Registration ===
 
-==== One Test for one Feature ====
+==== One Test for One Feature ====
 
 The most common case is that only one test is allocated for one feature (e.g. decoder, muxer, hashing algorithm, etc.). Let's open `tests/fate/audio.mak`, which contains tests for audio codecs, and scroll down to a test named `fate-dts`. For convenience, you can also see a copy of the recipe below.
@@ -52 +52 @@
 }}}
 
+==== Multiple Tests for One Feature ====
+
+Now knowing how to add one test for one feature, it is easy to add multiple tests. For example, you can just add multiple tests like this:
+
+{{{
+FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-1
+fate-dts-1: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-1.ts
+fate-dts-1: CMP = oneoff
+fate-dts-1: REF = $(SAMPLES)/dts/dts-1.pcm
+
+FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-2
+fate-dts-2: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-2.ts
+fate-dts-2: CMP = oneoff
+fate-dts-2: REF = $(SAMPLES)/dts/dts-2.pcm
+}}}
+
+However, this approach has some quirks:
+
+1. The long `FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA)` is repeated for each test, which reduces readability.
+2. It is not possible to do `make fate-dts` which runs these two tests (of course you can add another target that depends on the two tests, but that is not pretty for code readers).
+
+Therefore, you can use an additional variable that contains the targets related to a feature, and then use it.
+
+'''tl;dr:''' Let's open `tests/fate/amrnb.mk`, which is a great example of multiple tests. (A part of the file is omitted for clarity below.)
+
+{{{
+FATE_AMRNB += fate-amrnb-4k75
+fate-amrnb-4k75: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/4.75k.amr
+fate-amrnb-4k75: REF = $(SAMPLES)/amrnb/4.75k.pcm
+
+FATE_AMRNB += fate-amrnb-5k15
+fate-amrnb-5k15: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.15k.amr
+fate-amrnb-5k15: REF = $(SAMPLES)/amrnb/5.15k.pcm
+
+FATE_AMRNB += fate-amrnb-5k9
+fate-amrnb-5k9: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.9k.amr
+fate-amrnb-5k9: REF = $(SAMPLES)/amrnb/5.9k.pcm
+
+[...]
+
+$(FATE_AMRNB): CMP = stddev
+
+FATE_SAMPLES_AVCONV-$(call DEMDEC, AMR, AMRNB) += $(FATE_AMRNB)
+fate-amrnb: $(FATE_AMRNB)
+}}}
+
+So here, first the target names of three tests are added to a variable `FATE_AMRNB`, and then the
+
+== Dependencies ==
+
+It is very important for you to declare the dependencies of a test if it uses any external files, like special samples or synthetic files.
+
 == Filter Tests ==