mirror of
https://github.com/RIOT-OS/RIOT.git
synced 2024-12-29 04:50:03 +01:00
tests/doc: reorganization and improvements to tests doc
This commit is contained in:
krzysztof-cabaj
parent
897abd09f2
commit
f13794fdcd
136
tests/README.md
136
tests/README.md
@ -1,14 +1,90 @@
|
||||
Running and creating tests {#running-and-creating-tests}
|
||||
==========================
|
||||
# Running and creating tests {#running-and-creating-tests}
|
||||
|
||||
There are a number of tests included in RIOT. They are located in the
|
||||
[tests folder](https://github.com/RIOT-OS/RIOT/tree/master/tests). These tests
|
||||
allow basic functionality to be verified as well as provide an example of
|
||||
usage.
|
||||
|
||||
# Running automated tests
|
||||
|
||||
Implementing automated tests
|
||||
----------------------------
|
||||
Some tests can be performed automatically. The test automation scripts are
|
||||
defined in the `<test_application>/tests/` folder. They are written in python
|
||||
and interact through the serial (typically UART) with the test application code running on a
|
||||
board to do the validation. It is recommended to flash the board with the
|
||||
test just before running it because some platforms cannot be reset while
|
||||
testing.
|
||||
|
||||
## Running single test
|
||||
|
||||
From the test application directory run:
|
||||
|
||||
BOARD=<board_of_your_choice> make flash test
|
||||
|
||||
|
||||
An automated way of knowing if a test is available is to execute the
|
||||
'test/available' target from the test application directory.
|
||||
It executes without error if tests run by 'make test' are present.
|
||||
|
||||
make test/available
|
||||
|
||||
## Running all test for particular board
|
||||
|
||||
If you would like execute all tests for given board, you could use dedicated
|
||||
script `compile_and_test_for_board.py`
|
||||
|
||||
Go to main RIOT directory and execute command:
|
||||
|
||||
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <board_of_your_choice> --with-test-only --jobs=4
|
||||
|
||||
More details concerning other available parameters provided by this tool can be found in
|
||||
[README.md file](https://github.com/RIOT-OS/RIOT/tree/master/dist/tools/compile_and_test_for_board)
|
||||
and directly in [compile_and_test_for_board.py](https://github.com/RIOT-OS/RIOT/tree/master/dist/tools/compile_and_test_for_board/compile_and_test_for_board.py) script.
|
||||
|
||||
## Running tests that require a preliminary manual configuration
|
||||
|
||||
Some tests need active monitoring or manual setup steps but still have some
|
||||
automated scripts. The test automation scripts are defined in the
|
||||
`<test_application>/tests-with-config/` folder.
|
||||
For running them, follow the setup or analysis documentation and use the
|
||||
`test-with-config` target.
|
||||
|
||||
## Running tests that require root privileges
|
||||
|
||||
Some tests require root privileges to launch their automated script. In this
|
||||
case, the test automation scripts are defined in the
|
||||
`<test_application>/tests-as-root/` folder.
|
||||
For running them, follow the setup or analysis documentation and use the
|
||||
`test-as-root` target.
|
||||
|
||||
## Cleaning intermediate files
|
||||
|
||||
After test execution intermediate files are not automatically deleted.
|
||||
Execution of multiple tests, especially all for particular board could generate
|
||||
many files. For example, after execution of all test for stm32f469i-disco board
|
||||
(more than 230 tests) around 7.5 GB of intermediate files are created.
|
||||
|
||||
There are few methods for cleaning intermediate files.
|
||||
|
||||
If you would like to clean intermediate file only for particular board you should
|
||||
go to main RIOT directory and execute one from these commands:
|
||||
|
||||
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <board_of_your_choice> --compile-targets clean
|
||||
|
||||
or
|
||||
|
||||
make BOARD=<board_of_your_choice> clean
|
||||
|
||||
|
||||
If you would like to clean intermediate files for all boards go to main RIOT
|
||||
directory and use this command.
|
||||
|
||||
@warning This command cleans all local files, for example, pkg downloads and
|
||||
locally generared docs.
|
||||
|
||||
|
||||
make distclean
|
||||
|
||||
# Implementing automated tests
|
||||
|
||||
The goal is to be able to run all tests in a sequential way for as many targets
|
||||
as possible.
|
||||
@ -34,32 +110,7 @@ script and use `test_utils_interactive_sync();` function (e.g.:
|
||||
you can disable `test_utils_interactive_sync_shell` module in the application
|
||||
`Makefile`: `DISABLE_MODULE += test_utils_interactive_sync_shell`.
|
||||
|
||||
|
||||
Running automated tests
|
||||
-----------------------
|
||||
|
||||
Some tests can be performed automatically. The test automation scripts are
|
||||
defined in the `<test_application>/tests/` folder. They are written in python
|
||||
and interact through the uart with the test application code running on a
|
||||
board to do the validation. It is recommended to flash the board with the
|
||||
test just before running it because some platforms cannot be reset while
|
||||
testing.
|
||||
|
||||
From the test application directory run:
|
||||
|
||||
BOARD=<board_of_your_choice> make flash test
|
||||
|
||||
|
||||
An automated way of knowing if a test is available is to execute the
|
||||
'test/available' target from the test application directory.
|
||||
It executes without error if tests run by 'make test' are present.
|
||||
|
||||
make test/available
|
||||
|
||||
|
||||
Automated Tests Guidelines
|
||||
--------------------------
|
||||
|
||||
## Automated Tests Guidelines
|
||||
|
||||
When using `pexpect` `$` is useless for matching the end of a line, instead use
|
||||
`\r\n`([pexpect end-of-line](https://pexpect.readthedocs.io/en/stable/overview.html#find-the-end-of-line-cr-lf-conventions)).
|
||||
@ -93,8 +144,7 @@ pattern like `\r\n`, `\s`, `\)`, etc..
|
||||
child.expect(r'some string: (\d+) ,')
|
||||
~~~
|
||||
|
||||
Use expect() instead of assert()
|
||||
--------------------------------
|
||||
## Use expect() instead of assert()
|
||||
|
||||
In order to make a test application functional in all cases, use `expect()`
|
||||
instead of `assert()`. The former works like the latter, but will still be
|
||||
@ -104,8 +154,7 @@ be compiled with that flag. Otherwise, the application would force compiling
|
||||
all tested code with assertions enabled.
|
||||
`expect()` is defined in the header `test_utils/expect.h`.
|
||||
|
||||
Interaction through the uart
|
||||
----------------------------
|
||||
## Interaction through the uart
|
||||
|
||||
Tests implemented with `testrunner` use the `cleanterm` target that
|
||||
provides an interaction without adding extra text output or input handling.
|
||||
@ -117,22 +166,3 @@ The expected behavior is verified with the test in `tests/test_tools`.
|
||||
Tests cannot rely on having on all boards and terminal programs:
|
||||
* unbuffered input
|
||||
* allowing sending special characters like `ctrl+c/ctrl+d`
|
||||
|
||||
|
||||
Running tests that require a preliminary manual configuration
|
||||
-------------------------------------------------------------
|
||||
|
||||
Some tests need active monitoring or manual setup steps but still have some
|
||||
automated scripts. The test automation scripts are defined in the
|
||||
`<test_application>/tests-with-config/` folder.
|
||||
For running them, follow the setup or analysis documentation and use the
|
||||
`test-with-config` target.
|
||||
|
||||
Running tests that require root privileges
|
||||
------------------------------------------
|
||||
|
||||
Some tests require root privileges to launch their automated script. In this
|
||||
case, the test automation scripts are defined in the
|
||||
`<test_application>/tests-as-root/` folder.
|
||||
For running them, follow the setup or analysis documentation and use the
|
||||
`test-as-root` target.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user