L4Re - L4 Runtime Environment
test.mk - Test Application Role

The test role is very similar to the application role, it also builds an executable binary.

The difference is that is also builds for each target a test script that executes the test target either on the host (MODE=host) or a target platform (currently only qemu).

The role accepts all make variables that are accepted by the application role. The only difference is that the TARGET variable is not required. If it is missing, the source directory will be scanned for source files that fit the pattern test_*.c[c] and create one target for each of them.

It is possible to still use SRC_C[C] when targets are determined automatically. In that case the specified sources will be used in addition* to the main test_*.c[c] source.

In addition to the variables above, there are a number of variables that control how the test is executed. All these variables may be used as a global variable that applies to all test or, if the target name is added as a suffix, set for a specific target only.

Name of binary containing the test (default: same as TARGET).
When TARGET is undefined, these targets are added to the list of targets for the specified architecture. For all targets SRC_C[C] files must be defined separately.
File containing expected output. By default the variable is empty, which means the test binary is expected to produce TAP test output, that can be directly processed.
Number of times the expected output should be repeated, by default 1. When set to 0 then output is expected to repeat forever. This is particularly useful to make sure that stress tests that are meant to run in an endless loop are still alive. Note that such endless tests can only be run by directly executing the test script. They will be skipped when run in a test harness like prove.
Non-standard timeout after which the test run is aborted (useful for tests involving sleep).
LUA configuration file for startup to give to Ned
Additional modules needed to run the test.
Additional parameters to supply to QEMU.

Additional parameters to supply to moe.

Features the Fiasco kernel must have been compiled with. A space-separated list of config options as used by Kconfig. run_test looks for a globalconfig.out file in the same directory as the kernel and checks that all options are enabled. If not, the test is skipped. Has only an effect if the globalconfig.out file is present.
Features the L4Linux kernel must have been compiled with. Similar to KERNEL_CONF but checks for a .config file in the directory of the L4Linux kernel.
Command to execute before the test is run. The test will only be executed if the command returns 0. If the exit code is 2, the test is marked as skipped with the reason provided in the final line of stdout.
Append output of test execution to the given file unless TEST_WORKDIR is given.
Create logs, temp and other files below the given directory. That directory is taken as base dir for more automatically created subdir levels using the current test path, in order to guarantee conflict-free usage when running many different tests with a common workdir. When TEST_WORKDIR is provided then TEST_LOGFILE is ignored as it is organized below workdir.
Comma separated list of tags which have to be specified in order to run the test. Test will be skipped without error otherwise. Useful to mark tests, e.g. broken or slow. Tag names can basically contain any character except comma. Whitespace characters surrounding a tag name will be ignored. A tag starting with exclamation mark is checked to not appear in the specified run tags, otherwise the test is also skipped.

In addition to compiled tests, it is also possible to create tests where the test binary or script comes from a different source. These tests must be listed in EXTRA_TARGET and for each target a custom TEST_TARGET must be provided.

Running Tests

The make role creates a test script which can be found in <builddir>/test/t/<arch>/<api>. It is possible to organise the tests further in subdirectories below by specifying a TEST_GROUP.

To be able to execute the test, a minimal test environment needs to be set up by exporting the following environment variables:

Fiasco kernel binary to use. The test runner is able to check if the kernel has all features necessary for the test and skip tests accordingly. In order for this to work, the globalconfig.out config file from the build directory needs to be available in the same directory as the kernel.
L4Linux binary to use. This is only required to run tests in mode=l4linux. If no L4Linux kernel is set then these tests will simply be skipped. The test runner is also able to check if the kernel has all features compiled in that are required to run the test successfully (see make variable L4LINUX_CONF above). For this to work, the .config configuration file from the build directory needs to be available in the same directory as the kernel.
Ramdisk to mount as root in L4Linux. This is only required to run tests in mode=l4linux. If not supplied, L4Linux tests will be skipped. The ramdisk must be set up to start the test directly after the initial startup is finished. The name of the test binary is supplied via the kernel command line option l4re_testprog. The tool/test directory contains an example script launch-l4linux-test. which can be copied onto the ramdisk and started by the init script.

In addition to these variables, the following BID variables can be overwritten at runtime: PT (for the plaform type) and TEST_TIMEOUT. You may also supply QEMU_ARGS and MOE_ARGS which will be appended to the parameters specified in the BID test make file.

Once the environment is set up, the tests can be run either by simply executing all of them from the build directory with

make test

or executing them directly, like


or running one or more tests through the test harness prove, like

prove test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t
prove -r test/t/amd64_amdfam10/l4f/l4re-core/
prove -rv test/t/amd64_amdfam10/l4f/l4re-core/

To run tests with tags (see TEST_TAGS), these tags need to be specified either in the environment variable TEST_RUN_TAGS or as arguments to the test.t files:

$ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t --run-tags slow,gtest-shuffle
$ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t -T slow,gtest-shuffle
$ prove -r test/t/amd64_amdfam10/l4f/l4re-core/ :: -T slow,gtest-shuffle
$ TEST_RUN_TAGS=slow,gtest-shuffle prove -r test/t/amd64_amdfam10/l4f/l4re-core/

Without providing run tags, tests requiring certain tags are skipped:

$ make test
test/t/amd64_amdfam10/test_one.t .... ok
test/t/amd64_amdfam10/test_two.t .... skipped: requires additional run tags: slow
test/t/amd64_amdfam10/test_three.t .. ok

When tags are provided, the tests requiring those tags are now also executed, the tests that forbid them are skipped:

$ TEST_RUN_TAGS=slow,gtest-shuffle
$ make test
test/t/amd64_amdfam10/test_one.t .... ok
test/t/amd64_amdfam10/test_two.t .... ok
test/t/amd64_amdfam10/test_three.t .. skipped: not supported with run tags: gtest-shuffle

Tests without any tags are always executed.

Running Tests in External Programs

You can hand-over test execution to an external program by setting the environment variable EXTERNAL_TEST_STARTER to the full path of that program:

export EXTERNAL_TEST_STARTER=/path/to/external/test-starter
make test

This variable is evaluated by tool/bin/run_test (the backend behind make test) and contains the full path to the tool which actually starts the test instead of the test itself.

The EXTERNAL_TEST_STARTER can be any program instead of the default execution via make qemu E=maketest. Its output is taken by run_test as the test output.

Usually it is just a bridge to prepare the test execution, e.g., it could create the test as image and start that image via a simulator.

Running Tests in a Simulator

Based on above mechanism there is a dedicated external test starter tool/bin/teststarter-image-telnet.pl shipped in BID which assumes an image to be started with another program which provides test execution output on a network port.

This can be used to execute tests in a simulator, like this:

export EXTERNAL_TEST_STARTER=$L4RE_SRC/tool/bin/teststarter-image-telnet.pl
export SIMULATOR_START=/path/to/configured/simulator-exe
make test

After building the image and starting the simulator it contacts the simulator via a network port (sometimes called "telnet" port) to pass-through its execution output as its own output so it gets captured by run_test as usual.

The following variables control teststarter-image-telnet.pl:

This points to the full path of the program that actually starts the prepared test image. Most often this is the frontend script of your simulator environment which is pre-configured so that it actually works in the way that teststarter-image-telnet.pl expects from the following settings.
The image type to be generated via make $SIMULATOR_IMAGETYPE E=maketest. Default is elfimage.
The simulator will be contacted via socket on that host to read its output. Default is localhost.
The simulator will be contacted via socket on that port to read its output. Default is 11111.
After starting the simulator it waits that many seconds before reading from the port. Default is 1 (second).

Running tests without tapper-wrapper

In case you want to replace the tapper-wrapper test starter, you can replace the default one by setting the environment variable TEST_STARTER to the path of your test starter. Then your test starter can use the same environment which is normally set up for the default starter, which includes environment variables provided by the build system as well as the test itself. Among these are SEARCHPATH, MODE, ARCH, MOE_CFG, MOE_ARGS, TEST_TIMEOUT, TEST_TARGET, TEST_EXPECTED, QEMU_ARGS and many more.

Debugging Tests

The test script is only a thin wrapper that sets up the test environment as it was defined in the make file and then executes two scripts: tapper-wrapper and run_test.

The main work horse of the two is tool/bin/run_test. It collects the necessary files and starts qemu to execute the test. This script is always required.

There is then a second script wrapped around the test runner: tool/bin/tapper-wrapper. This tool inspects the output of the test runner and reformats it, so that it can be read by tools like prove. If the test produces tap output, then the script scans for this output and filters away all the debug output. If TEST_EXPECTED was defined, then the script scans the output for the expected lines and prints a suitable TAP message with success or failure. It also makes sure that qemu is killed as soon as the test is finished.

There are a number of command-line parameters that allow to quickly change test parameters for debugging purposes. Run the test with '–help' for more information about available parameters.