FATE Automated Testing Environment

Table of Contents

1. Introduction

FATE provides a regression testsuite embedded within the Libav build system. It can be run locally and optionally configured to send reports to a web aggregator and viewer http://fate.libav.org.

It is advised to run FATE before submitting patches to the current codebase and provide new tests when submitting patches to add additional features.

2. Running FATE

2.1 Samples and References

In order to run, FATE needs a large amount of data (samples and references) that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass ‘--samples=<path to the samples>’ to configure or set the SAMPLES Make variable or the LIBAV_SAMPLES environment variable to a suitable value.

To use a custom wrapper to run the test, pass ‘--target-exec’ to configure or set the TARGET_EXEC Make variable.

The dataset is available through rsync, is possible to fetch the current sample using the straight rsync command or through a specific Makefile target.

 
# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
 
# make fate-rsync SAMPLES=fate-suite

3. Manual Run

FATE regression test can be run through make. Specific Makefile targets and Makefile variables are available:

3.1 FATE Makefile targets

fate-list

List all fate/regression test targets.

fate-rsync

Shortcut to download the fate test samples to the specified testsuite location.

fate

Run the FATE test suite (requires the fate-suite dataset).

3.2 FATE Makefile variables

V

Verbosity level, can be set to 0, 1 or 2.

0

show just the test arguments

1

show just the command used in the test

2

show everything

SAMPLES

Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.

THREADS

Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.

THREAD_TYPE

Specify which threading strategy test, either slice or frame, by default slice+frame

CPUFLAGS

Specify a mask to be applied to autodetected CPU flags.

TARGET_EXEC

Specify or override the wrapper used to run the tests.

GEN

Set to 1 to generate the missing or mismatched references.

 
    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate

4. Automated Tests

In order to automatically testing specific configurations, e.g. multiple compilers, tests/fate.sh is provided.

This shell script builds Libav, runs the regression tests and prepares a report that can be sent to http://fate.libav.org/ or directly examined locally.

4.1 Testing Profiles

The configuration file passed to fate.sh is shell scripts as well.

It must provide at least a slot identifier, the repo from which fetch the sources, the samples directory, a workdir with enough space to build and run all the tests. Optional submit command fate_recv and a comment to describe the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process can be passed.

 
slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
#branch=release/10                      # the branch to test
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
comment=                                # optional description
build_only=     # set to "yes" for a compile-only instance that skips tests

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
as=
cc=
ld=
target_os=
sysroot=
target_exec=
target_path=
target_samples=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'

4.2 Special Instances

The TARGET_EXEC option provides a way to run FATE wrapped in valgrind, qemu-user or wine or on remote targets through ssh.

4.3 Submitting Reports

In order to send reports you need to create an ssh key and send it to root@libav.org.