L4Re - L4 Runtime Environment
|
The test role is very similar to the application role, it also builds an executable binary.
The difference is that is also builds for each target a test script that executes the test target either on the host (MODE=host) or a target platform (currently only qemu).
The role accepts all make variables that are accepted by the application role. The only difference is that the TARGET
variable is not required. If it is missing, the source directory will be scanned for source files that fit the pattern test_*.c[c]
and create one target for each of them.
test_*.c[c]
source.In addition to the variables above, there are a number of variables that control how the test is executed. All these variables may be used as a global variable that applies to all test or, if the target name is added as a suffix, set for a specific target only.
TARGET
).SRC_C[C]
files must be defined separately.prove
.Additional parameters to supply to moe.
run_test
looks for a globalconfig.out
file in the same directory as the kernel and checks that all options are enabled. If not, the test is skipped. Has only an effect if the globalconfig.out
file is present.KERNEL_CONF
but checks for a .config
file in the directory of the L4Linux kernel.In addition to compiled tests, it is also possible to create tests where the test binary or script comes from a different source. These tests must be listed in EXTRA_TARGET
and for each target a custom TEST_TARGET
must be provided.
The make role creates a test script which can be found in <builddir>/test/t/<arch>/<api>
. It is possible to organise the tests further in subdirectories below by specifying a TEST_GROUP.
To be able to execute the test, a minimal test environment needs to be set up by exporting the following environment variables:
globalconfig.out
config file from the build directory needs to be available in the same directory as the kernel.mode=l4linux
. If no L4Linux kernel is set then these tests will simply be skipped. The test runner is also able to check if the kernel has all features compiled in that are required to run the test successfully (see make variable L4LINUX_CONF
above). For this to work, the .config
configuration file from the build directory needs to be available in the same directory as the kernel.mode=l4linux
. If not supplied, L4Linux tests will be skipped. The ramdisk must be set up to start the test directly after the initial startup is finished. The name of the test binary is supplied via the kernel command line option l4re_testprog
. The tool/test
directory contains an example script launch-l4linux-test
. which can be copied onto the ramdisk and started by the init script.In addition to these variables, the following BID variables can be overwritten at runtime: PT
(for the plaform type) and TEST_TIMEOUT
. You may also supply QEMU_ARGS
and MOE_ARGS
which will be appended to the parameters specified in the BID test make file.
Once the environment is set up, the tests can be run either by simply executing all of them from the build directory with
make test
or executing them directly, like
test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t
or running one or more tests through the test harness prove, like
prove test/t/amd64_amdfam10/l4f/l4re-core/moe/test_namespace.t prove -r test/t/amd64_amdfam10/l4f/l4re-core/ prove -rv test/t/amd64_amdfam10/l4f/l4re-core/
To run tests with tags (see TEST_TAGS
), these tags need to be specified either in the environment variable TEST_RUN_TAGS
or as arguments to the test.t files:
$ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t --run-tags slow,gtest-shuffle $ test/t/amd64_amdfam10/l4f/l4re-core/test_one.t -T slow,gtest-shuffle $ prove -r test/t/amd64_amdfam10/l4f/l4re-core/ :: -T slow,gtest-shuffle $ TEST_RUN_TAGS=slow,gtest-shuffle prove -r test/t/amd64_amdfam10/l4f/l4re-core/
Without providing run tags, tests requiring certain tags are skipped:
$ make test ... test/t/amd64_amdfam10/test_one.t .... ok test/t/amd64_amdfam10/test_two.t .... skipped: requires additional run tags: slow test/t/amd64_amdfam10/test_three.t .. ok
When tags are provided, the tests requiring those tags are now also executed, the tests that forbid them are skipped:
$ TEST_RUN_TAGS=slow,gtest-shuffle $ make test ... test/t/amd64_amdfam10/test_one.t .... ok test/t/amd64_amdfam10/test_two.t .... ok test/t/amd64_amdfam10/test_three.t .. skipped: not supported with run tags: gtest-shuffle
Tests without any tags are always executed.
You can hand-over test execution to an external program by setting the environment variable EXTERNAL_TEST_STARTER
to the full path of that program:
export EXTERNAL_TEST_STARTER=/path/to/external/test-starter make test
This variable is evaluated by tool/bin/run_test
(the backend behind make test
) and contains the full path to the tool which actually starts the test instead of the test itself.
The EXTERNAL_TEST_STARTER
can be any program instead of the default execution via make qemu E=maketest
. Its output is taken by run_test
as the test output.
Usually it is just a bridge to prepare the test execution, e.g., it could create the test as image and start that image via a simulator.
Based on above mechanism there is a dedicated external test starter tool/bin/teststarter-image-telnet.pl
shipped in BID which assumes an image to be started with another program which provides test execution output on a network port.
This can be used to execute tests in a simulator, like this:
export EXTERNAL_TEST_STARTER=$L4RE_SRC/tool/bin/teststarter-image-telnet.pl export SIMULATOR_START=/path/to/configured/simulator-exe make test
After building the image and starting the simulator it contacts the simulator via a network port (sometimes called "telnet" port) to pass-through its execution output as its own output so it gets captured by run_test
as usual.
The following variables control teststarter-image-telnet.pl
:
teststarter-image-telnet.pl
expects from the following settings.make $SIMULATOR_IMAGETYPE E=maketest
. Default is elfimage
.localhost
.11111
.1
(second).In case you want to replace the tapper-wrapper test starter, you can replace the default one by setting the environment variable TEST_STARTER
to the path of your test starter. Then your test starter can use the same environment which is normally set up for the default starter, which includes environment variables provided by the build system as well as the test itself. Among these are SEARCHPATH
, MODE
, ARCH
, MOE_CFG
, MOE_ARGS
, TEST_TIMEOUT
, TEST_TARGET
, TEST_EXPECTED
, QEMU_ARGS
and many more.
The test script is only a thin wrapper that sets up the test environment as it was defined in the make file and then executes two scripts: tapper-wrapper
and run_test
.
The main work horse of the two is tool/bin/run_test
. It collects the necessary files and starts qemu to execute the test. This script is always required.
There is then a second script wrapped around the test runner: tool/bin/tapper-wrapper
. This tool inspects the output of the test runner and reformats it, so that it can be read by tools like prove
. If the test produces tap output, then the script scans for this output and filters away all the debug output. If TEST_EXPECTED
was defined, then the script scans the output for the expected lines and prints a suitable TAP message with success or failure. It also makes sure that qemu is killed as soon as the test is finished.
There are a number of command-line parameters that allow to quickly change test parameters for debugging purposes. Run the test with '–help' for more information about available parameters.