L4Re - L4 Runtime Environment
test.mk - Test Application Role

The test role is very similar to the application role, it also builds an executable binary.

The difference is that is also builds for each target a test script that executes the test target either on the host (MODE=host) or a target platform (currently only qemu).

The role accepts all make variables that are accepted by the application role. The only difference is that the TARGET variable is not required. If it is missing, the source directory will be scanned for source files that fit the pattern test_*.c[c] and create one target for each of them.

Note
It is possible to still use SRC_C[C] when targets are determined automatically. In that case the specified sources will be used in addition* to the main test_*.c[c] source.

In addition to the variables above, there are a number of variables that control how the test is executed. All these variables may be used as a global variable that applies to all test or, if the target name is added as a suffix, set for a specific target only.

TEST_TARGET
Name of binary containing the test (default: same as TARGET).
TARGET_$(ARCH)
When TARGET is undefined, these targets are added to the list of targets for the specified architecture. For all targets SRC_C[C] files must be defined separately.
TEST_EXPECTED
File containing expected output. By default the variable is empty, which means the test binary is expected to produce TAP test output, that can be directly processed.
TEST_EXPECTED_REPEAT
Number of times the expected output should be repeated, by default 1. When set to 0 then output is expected to repeat forever. This is particularly useful to make sure that stress tests that are meant to run in an endless loop are still alive. Note that such endless tests can only be run by directly executing the test script. They will be skipped when run in a test harness like prove.
TEST_TIMEOUT
Non-standard timeout after which the test run is aborted (useful for tests involving sleep).
MOE_CFG
LUA configuration file for startup to give to moe
REQUIRED_MODULES
Additional modules needed to run the test.
QEMU_ARGS
Additional parameters to supply to QEMU.
MOE_ARGS

Additional parameters to supply to moe.

KERNEL_CONF
Features the Fiasco kernel must have been compiled with. A space-separated list of config options as used by Kconfig. run_test looks for a globalconfig.out file in the same directory as the kernel and checks that all options are enabled. If not, the test is skipped. Has only an effect if the globalconfig.out file is present.
L4LINUX_CONF
Features the L4Linux kernel must have been compiled with. Similar to KERNEL_CONF but checks for a .config file in the directory of the L4Linux kernel.
TEST_SETUP
Command to execute before the test is run. The test will only be executed if the command returns 0. If the exit code is 2, the test is marked as skipped with the reason provided in the final line of stdout.
TEST_LOGFILE
Append output of test execution to the given file unless TEST_WORKDIR is given.
TEST_WORKDIR
Create logs, temp and other files below the given directory. That directory is taken as base dir for more automatically created subdir levels using the current test path, in order to guarantee conflict-free usage when running many different tests with a common workdir. When TEST_WORKDIR is provided then TEST_LOGFILE is ignored as it is organized below workdir.

In addition to compiled tests, it is also possible to create tests where the test binary or script comes from a different source. These tests must be listed in EXTRA_TARGET and for each target a custom TEST_TARGET must be provided.

Running Tests

The make role creates a test script which can be found in <builddir>/test/t/<arch>/<api>. It is possible to organise the tests further in subdirectories below by specifying a TEST_GROUP.

To be able to execute the test, a minimal test environment needs to be set up by exporting the following environment variables:

KERNEL_<arch>, KERNEL
Fiasco kernel binary to use. The test runner is able to check if the kernel has all features necessary for the test and skip tests accordingly. In order for this to work, the globalconfig.out config file from the build directory needs to be available in the same directory as the kernel.
L4LX_KERNEL_<arch>, L4_LX_KERNEL
L4Linux binary to use. This is only required to run tests in mode=l4linux. If no L4Linux kernel is set then these tests will simply be skipped. The test runner is also able to check if the kernel has all features compiled in that are required to run the test successfully (see make variable L4LINUX_CONF above). For this to work, the .config configuration file from the build directory needs to be available in the same directory as the kernel.
LINUX_RAMDISK_<arch>, LINUX_RAMDISK
Ramdisk to mount as root in L4Linux. This is only required to run tests in mode=l4linux. If not supplied, L4Linux tests will be skipped. The ramdisk must be set up to start the test directly after the initial startup is finished. The name of the test binary is supplied via the kernel command line option l4re_testprog. The tool/test directory contains an example script launch-l4linux-test. which can be copied onto the ramdisk and started by the init script.

In addition to these variables, the following BID variables can be overwritten at runtime: PT (for the plaform type) and TEST_TIMEOUT. You may also supply QEMU_ARGS and MOE_ARGS which will be appended to the parameters specified in the BID test make file.

Once the environment is set up, the tests can be run either by simply executing all of them from the build directory with

make test

or executing them directly, like

test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t

or running one or more tests through the test harness prove, like

prove test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t
prove -r test/t/amd64_amdfam10/l4f/l4re-core/
prove -rv test/t/amd64_amdfam10/l4f/l4re-core/

Running Tests in External Programs

You can hand-over test execution to an external program by setting the environment variable EXTERNAL_TEST_STARTER to the full path of that program:

export EXTERNAL_TEST_STARTER=/path/to/external/test-starter
make test
EXTERNAL_TEST_STARTER

This variable is evaluated by tool/bin/run_test (the backend behind make test) and contains the full path to the tool which actually starts the test instead of the test itself.

The EXTERNAL_TEST_STARTER can be any program instead of the default execution via make qemu E=maketest. Its output is taken by run_test as the test output.

Usually it is just a bridge to prepare the test execution, e.g., it could create the test as image and start that image via a simulator.

Running Tests in a Simulator

Based on above mechanism there is a dedicated external test starter tool/bin/teststarter-image-telnet.pl shipped in BID which assumes an image to be started with another program which provides test execution output on a network port.

This can be used to execute tests in a simulator, like this:

export EXTERNAL_TEST_STARTER=$L4RE_SRC/tool/bin/teststarter-image-telnet.pl
export SIMULATOR_START=/path/to/configured/simulator-exe
make test

After building the image and starting the simulator it contacts the simulator via a network port (sometimes called "telnet" port) to pass-through its execution output as its own output so it gets captured by run_test as usual.

The following variables control teststarter-image-telnet.pl:

SIMULATOR_START
This points to the full path of the program that actually starts the prepared test image. Most often this is the frontend script of your simulator environment which is pre-configured so that it actually works in the way that teststarter-image-telnet.pl expects from the following settings.
SIMULATOR_IMAGETYPE
The image type to be generated via make $SIMULATOR_IMAGETYPE E=maketest. Default is elfimage.
SIMULATOR_HOST
The simulator will be contacted via socket on that host to read its output. Default is localhost.
SIMULATOR_PORT
The simulator will be contacted via socket on that port to read its output. Default is 11111.
SIMULATOR_START_SLEEPTIME
After starting the simulator it waits that many seconds before reading from the port. Default is 1 (second).

Debugging Tests

The test script is only a thin wrapper that sets up the test environment as it was defined in the make file and then executes two scripts: tapper-wrapper and run_test.

The main work horse of the two is tool/bin/run_test. It collects the necessary files and starts qemu to execute the test. This script is always required.

There is then a second script wrapped around the test runner: tool/bin/tapper-wrapper. This tool inspects the output of the test runner and reformats it, so that it can be read by tools like prove. If the test produces tap output, then the script scans for this output and filters away all the debug output. If TEST_EXPECTED was defined, then the script scans the output for the expected lines and prints a suitable TAP message with success or failure. It also makes sure that qemu is killed as soon as the test is finished.

There are a number of command-line parameters that allow to quickly change test parameters for debugging purposes. Run the test with '–help' for more information about available parameters.