L4Re – L4 Runtime Environment
|
The test role is very similar to the application role, it also builds an executable binary.
The difference is that is also builds for each target a test script that executes the test target either on the host (MODE=host) or a target platform (currently only qemu).
The role accepts all make variables that are accepted by the application role. The only difference is that the TARGET
variable is not required. If it is missing, the source directory will be scanned for source files that fit the pattern test_*.c[c]
and create one target for each of them.
test_*.c[c]
source.In addition to the variables above, there are a number of variables that control how the test is executed. All these variables may be used as a global variable that applies to all test or, if the target name is added as a suffix, set for a specific target only.
TARGET
).SRC_C[C]
files must be defined separately.prove
.TEST_STARTER
(tapper-wrapper per default).TEST_ROOT_TASK
if TEST_ROOT_TASK
is different from moe.run_test
looks for a globalconfig.out
file in the same directory as the kernel and checks that all options are enabled. If not, the test is skipped. Has only an effect if the globalconfig.out
file is present.KERNEL_CONF
but checks for a .config
file in the directory of the L4Linux kernel.List of conditions for tags provided during execution of a test. A tag can be set to 1, set to 0 or be unspecified via TEST_RUN_TAGS during execution. Therefore there are 4 possible conditions for a tag that can be specified in TEST_TAGS: tag, !tag, +tag and -tag. The following table shows the conditions they represent.
TEST_RUN_TAGS \ TEST_TAGS | tag | !tag | +tag | -tag |
---|---|---|---|---|
tag or tag=1 | y | y | ||
unspecified | y | y | ||
tag=0 | y | y |
Example usage:
The tag long-running
is used by tests which take a long time and should be skipped by default. These tests are marked with the tag long-running unprefixed.
The tag hardware
is set to 1 at runtime when the tests will run on real hardware. Tests that must not run on real hardware are marked with !hardware
.
The tag impl-def
is used by tests that test implementation details. They are run by default but can be excluded from execution by settings TEST_RUN_TAGS to impl-def=0. These tests are marked using +impl-def
.
If you want to specify multiple tag conditions they need to be separated with a comma.
In addition to compiled tests, it is also possible to create tests where the test binary or script comes from a different source. These tests must be listed in EXTRA_TARGET
and for each target a custom TEST_TARGET
must be provided.
The make role creates a test script which can be found in <builddir>/test/t/<arch>/<api>
. It is possible to organise the tests further in subdirectories below by specifying a TEST_GROUP.
To be able to execute the test, a minimal test environment needs to be set up by exporting the following environment variables:
globalconfig.out
config file from the build directory needs to be available in the same directory as the kernel.mode=l4linux
. If no L4Linux kernel is set then these tests will simply be skipped. The test runner is also able to check if the kernel has all features compiled in that are required to run the test successfully (see make variable L4LINUX_CONF
above). For this to work, the .config
configuration file from the build directory needs to be available in the same directory as the kernel.mode=l4linux
. If not supplied, L4Linux tests will be skipped. The ramdisk must be set up to start the test directly after the initial startup is finished. The name of the test binary is supplied via the kernel command line option l4re_testprog
. The tool/test
directory contains an example script launch-l4linux-test
. which can be copied onto the ramdisk and started by the init script.In addition to these variables, the following BID variables can be overwritten at runtime: PT
(for the plaform type) and TEST_TIMEOUT
. You may also supply QEMU_ARGS
and MOE_ARGS
which will be appended to the parameters specified in the BID test make file.
Once the environment is set up, the tests can be run either by simply executing all of them from the build directory with
make test
or executing them directly, like
test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t
or running one or more tests through the test harness prove, like
prove test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t prove -r test/t/amd64_amdfam10/l4f/l4re-core/ prove -rv test/t/amd64_amdfam10/l4f/l4re-core/
TEST_TAGS allow for a way to include or exclude whole groups of tests during execution, primarily with prove. You can specify which tests to run at runtime using one of the following ways:
$ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t --run-tags slow,gtest-shuffle=0 $ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t -T slow,gtest-shuffle=0 $ prove -r test/t/amd64_amdfam10/l4f/l4re-core/ :: -T slow,gtest-shuffle=0 $ TEST_RUN_TAGS=slow,gtest-shuffle=0 prove -r test/t/amd64_amdfam10/l4f/l4re-core/
For each test tag requirements defined in the corresponding TEST_TAGS Makefile variable are tested. If the requirements for tags do not match the test is skipped. The SKIP message will provide insight why the test was skipped:
$ make test ... test/t/amd64_amdfam10/test_one.t .... ok test/t/amd64_amdfam10/test_two.t .... skipped: Running this test requires tag slow to be set to 1. test/t/amd64_amdfam10/test_three.t .. ok
When tags are provided, the tests requiring those tags are now also executed while the tests that forbid them are skipped:
$ TEST_RUN_TAGS=slow,gtest-shuffle $ make test ... test/t/amd64_amdfam10/test_one.t .... ok test/t/amd64_amdfam10/test_two.t .... ok test/t/amd64_amdfam10/test_three.t .. skipped: Running this test requires tag gtest-shuffle to be set to 0 or not specified.
For further details on how values in TEST_TAGS and TEST_RUN_TAGS interact, see the help text for TEST_TAGS.
You can hand-over test execution to an external program by setting the environment variable EXTERNAL_TEST_STARTER
to the full path of that program:
export EXTERNAL_TEST_STARTER=/path/to/external/test-starter make test
This variable is evaluated by tool/bin/run_test
(the backend behind make test
) and contains the full path to the tool which actually starts the test instead of the test itself.
The EXTERNAL_TEST_STARTER
can be any program instead of the default execution via make qemu E=maketest
. Its output is taken by run_test
as the test output.
Usually it is just a bridge to prepare the test execution, e.g., it could create the test as image and start that image via a simulator.
Based on above mechanism there is a dedicated external test starter tool/bin/teststarter-image-telnet.pl
shipped in BID which assumes an image to be started with another program which provides test execution output on a network port.
This can be used to execute tests in a simulator, like this:
export EXTERNAL_TEST_STARTER=$L4RE_SRC/tool/bin/teststarter-image-telnet.pl export SIMULATOR_START=/path/to/configured/simulator-exe make test
After building the image and starting the simulator it contacts the simulator via a network port (sometimes called "telnet" port) to pass-through its execution output as its own output so it gets captured by run_test
as usual.
The following variables control teststarter-image-telnet.pl
:
teststarter-image-telnet.pl
expects from the following settings.make $SIMULATOR_IMAGETYPE E=maketest
. Default is elfimage
.localhost
.11111
.1
(second).In case you want to replace the tapper-wrapper test starter, you can replace the default one by setting the environment variable TEST_STARTER
to the path of your test starter. Then your test starter can use the same environment which is normally set up for the default starter, which includes environment variables provided by the build system as well as the test itself. Among these are SEARCHPATH
, MODE
, ARCH
, MOE_CFG
, MOE_ARGS
, TEST_TIMEOUT
, TEST_TARGET
, TEST_EXPECTED
, QEMU_ARGS
and many more.
The test script is only a thin wrapper that sets up the test environment as it was defined in the make file and then executes two scripts: tapper-wrapper
and run_test
.
The main work horse of the two is tool/bin/run_test
. It collects the necessary files and starts qemu to execute the test. This script is always required.
There is then a second script wrapped around the test runner: tool/bin/tapper-wrapper
. This tool inspects the output of the test runner and reformats it, so that it can be read by tools like prove
. If the test produces tap output, then the script scans for this output and filters away all the debug output. If TEST_EXPECTED
was defined, then the script scans the output for the expected lines and prints a suitable TAP message with success or failure. It also makes sure that qemu is killed as soon as the test is finished.
There are a number of command-line parameters that allow to quickly change test parameters for debugging purposes. Run the test with '–help' for more information about available parameters.