FATE Automated Testing Environment

Table of Contents

1. Introduction

FATE provides a regression test suite embedded within the Libav build system. It can be run locally and optionally configured to send reports to a web aggregator and viewer http://fate.libav.org.

It is advised to run FATE before submitting patches to the current codebase and provide new tests when submitting patches to add additional features.

2. Running FATE

2.1 Samples and References

In order to run, FATE needs a large amount of data (samples and references) that is provided separately from the actual source distribution.

To inform the build system about the test suite location, pass ‘--samples=<path to the samples>’ to configure or set the SAMPLES Make variable or the LIBAV_SAMPLES environment variable to a suitable value.

To use a custom wrapper to run the test, pass ‘--target-exec’ to configure or set the TARGET_EXEC Make variable.

The dataset is available through rsync, is possible to fetch the current sample using the straight rsync command or through a specific Makefile target.

# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
# make fate-rsync SAMPLES=fate-suite

3. Manual Run

FATE regression test can be run through make. Specific Makefile targets and Makefile variables are available:

3.1 FATE Makefile targets


List all fate/regression test targets.


Shortcut to download the fate test samples to the specified test suite location.


Run the FATE test suite (requires the fate-suite dataset).

3.2 FATE Makefile variables


Verbosity level, can be set to 0, 1 or 2.


show just the test arguments


show just the command used in the test


show everything


Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.


Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.


Specify which threading strategy test, either slice or frame, by default slice+frame


Specify a mask to be applied to autodetected CPU flags.


Specify or override the wrapper used to run the tests.


Set to 1 to generate the missing or mismatched references.


Specify which hardware acceleration to use while running regression tests, by default none is used.

    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate

4. Automated Tests

In order to automatically testing specific configurations, e.g. multiple compilers, tests/fate.sh is provided.

This shell script builds Libav, runs the regression tests and prepares a report that can be sent to http://fate.libav.org/ or directly examined locally.

4.1 Testing Profiles

The configuration file passed to fate.sh is shell scripts as well.

It must provide at least a slot identifier, the repo from which fetch the sources, the samples directory, a workdir with enough space to build and run all the tests. Optional submit command fate_recv and a comment to describe the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process can be passed.

slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
#branch=release/10                      # the branch to test
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
comment=                                # optional description
build_only=     # set to "yes" for a compile-only instance that skips tests

# the following are optional and map to configure options
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#makeopts_fate= # extra options passed to 'make' when running tests,
                # defaulting to makeopts above if this is not set
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'

4.2 Special Instances

The TARGET_EXEC option provides a way to run FATE wrapped in valgrind, qemu-user or wine or on remote targets through ssh.

4.3 Submitting Reports

In order to send reports you need to create an ssh key and send it to root@libav.org.