Changes between Version 6 and Version 7 of FATE/AddingATest


Ignore:
Timestamp:
Apr 14, 2014, 6:22:47 AM (5 years ago)
Author:
Timothy_Gu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • FATE/AddingATest

    v6 v7  
    33There are two main kinds of FATE test: using existing samples (primarily useful with decoder testing), and using an artificially generated sample for testing.
    44
    5 The recipes for the tests written in GNU Make are located under `tests/fate` in the FFmpeg source. Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests.
     5The recipes for the tests written in GNU Make are located under `tests/fate` in the FFmpeg source, and the tests themselves are executed through a shell wrapper script `tests/fate-run.sh`. For the most part you do not need to know anything about the shell script for this wiki page, but '''we assume moderate knowledge and experience with GNU Make''' for this wiki page.
    66
    7 == Four Parts of a Test ==
     7Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests. Then, after that, we'll have some notes on testing specific component like filters or decoders.
     8
     9== Three Parts of a Test ==
    810
    911A typical test usually consists of four parts:
    1012
    11 1. Registration: meaning that `make fate` automatically execute the test you want to add;
     131. Registration: registering your new test to `make fate` target;
    12142. Dependencies: which are usually samples for a format, a generated sample, or a program specifically written to test this feature;
    13153. Configuration: this part is the most important of all, because it controls how the test will be run.
    1416
    15 Let us focus on the separate parts of a test.
    16 
    1717=== Registration ===
    1818
    19 ==== One Test for one Feature ====
     19==== One Test for One Feature ====
    2020
    2121The most common case is that only one test is allocated for one feature (e.g. decoder, muxer, hashing algorithm, etc.). Let's open `tests/fate/audio.mak`, which contains tests for audio codecs, and scroll down to a test named `fate-dts`. For convenience, you can also see a copy of the recipe below.
     
    5252}}}
    5353
     54==== Multiple Tests for One Feature ====
     55
     56Now knowing how to add one test for one feature, it is easy to add multiple tests. For example, you can just add multiple tests like this:
     57
     58{{{
     59FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-1
     60fate-dts-1: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-1.ts
     61fate-dts-1: CMP = oneoff
     62fate-dts-1: REF = $(SAMPLES)/dts/dts-1.pcm
     63
     64FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-2
     65fate-dts-2: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-2.ts
     66fate-dts-2: CMP = oneoff
     67fate-dts-2: REF = $(SAMPLES)/dts/dts-2.pcm
     68}}}
     69
     70However, this approach has some quirks:
     71
     721. The long `FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA)` is repeated for each test, which reduces readability.
     732. It is not possible to do `make fate-dts` which runs these two tests (of course you can add another target that depends on the two tests, but that is not pretty for code readers).
     74
     75Therefore, you can use an additional variable that contains the targets related to a feature, and then use it.
     76
     77'''tl;dr:''' Let's open `tests/fate/amrnb.mk`, which is a great example of multiple tests. (A part of the file is omitted for clarity below.)
     78
     79{{{
     80FATE_AMRNB += fate-amrnb-4k75
     81fate-amrnb-4k75: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/4.75k.amr
     82fate-amrnb-4k75: REF = $(SAMPLES)/amrnb/4.75k.pcm
     83
     84FATE_AMRNB += fate-amrnb-5k15
     85fate-amrnb-5k15: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.15k.amr
     86fate-amrnb-5k15: REF = $(SAMPLES)/amrnb/5.15k.pcm
     87
     88FATE_AMRNB += fate-amrnb-5k9
     89fate-amrnb-5k9: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.9k.amr
     90fate-amrnb-5k9: REF = $(SAMPLES)/amrnb/5.9k.pcm
     91
     92[...]
     93
     94$(FATE_AMRNB): CMP = stddev
     95
     96FATE_SAMPLES_AVCONV-$(call DEMDEC, AMR, AMRNB) += $(FATE_AMRNB)
     97fate-amrnb: $(FATE_AMRNB)
     98}}}
     99
     100So here, first the target names of three tests are added to a variable `FATE_AMRNB`, and then the
     101
     102== Dependencies ==
     103
     104It is very important for you to declare the dependencies of a test if it uses any external files, like special samples or synthetic files.
     105
    54106== Filter Tests ==
    55107