diff --git a/tests/README.md b/tests/README.md index 2db0ac5faa..43eaaaa903 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,14 +1,90 @@ -Running and creating tests {#running-and-creating-tests} -========================== +# Running and creating tests {#running-and-creating-tests} There are a number of tests included in RIOT. They are located in the [tests folder](https://github.com/RIOT-OS/RIOT/tree/master/tests). These tests allow basic functionality to be verified as well as provide an example of usage. +# Running automated tests -Implementing automated tests ----------------------------- +Some tests can be performed automatically. The test automation scripts are +defined in the `/tests/` folder. They are written in python +and interact through the serial (typically UART) with the test application code running on a +board to do the validation. It is recommended to flash the board with the +test just before running it because some platforms cannot be reset while +testing. + +## Running single test + +From the test application directory run: + + BOARD= make flash test + + +An automated way of knowing if a test is available is to execute the +'test/available' target from the test application directory. +It executes without error if tests run by 'make test' are present. + + make test/available + +## Running all test for particular board + +If you would like execute all tests for given board, you could use dedicated +script `compile_and_test_for_board.py` + +Go to main RIOT directory and execute command: + + ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . --with-test-only --jobs=4 + +More details concerning other available parameters provided by this tool can be found in +[README.md file](https://github.com/RIOT-OS/RIOT/tree/master/dist/tools/compile_and_test_for_board) +and directly in [compile_and_test_for_board.py](https://github.com/RIOT-OS/RIOT/tree/master/dist/tools/compile_and_test_for_board/compile_and_test_for_board.py) script. + +## Running tests that require a preliminary manual configuration + +Some tests need active monitoring or manual setup steps but still have some +automated scripts. The test automation scripts are defined in the +`/tests-with-config/` folder. +For running them, follow the setup or analysis documentation and use the +`test-with-config` target. + +## Running tests that require root privileges + +Some tests require root privileges to launch their automated script. In this +case, the test automation scripts are defined in the +`/tests-as-root/` folder. +For running them, follow the setup or analysis documentation and use the +`test-as-root` target. + +## Cleaning intermediate files + +After test execution intermediate files are not automatically deleted. +Execution of multiple tests, especially all for particular board could generate +many files. For example, after execution of all test for stm32f469i-disco board +(more than 230 tests) around 7.5 GB of intermediate files are created. + +There are few methods for cleaning intermediate files. + +If you would like to clean intermediate file only for particular board you should +go to main RIOT directory and execute one from these commands: + + ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . --compile-targets clean + +or + + make BOARD= clean + + +If you would like to clean intermediate files for all boards go to main RIOT +directory and use this command. + +@warning This command cleans all local files, for example, pkg downloads and +locally generared docs. + + + make distclean + +# Implementing automated tests The goal is to be able to run all tests in a sequential way for as many targets as possible. @@ -34,32 +110,7 @@ script and use `test_utils_interactive_sync();` function (e.g.: you can disable `test_utils_interactive_sync_shell` module in the application `Makefile`: `DISABLE_MODULE += test_utils_interactive_sync_shell`. - -Running automated tests ------------------------ - -Some tests can be performed automatically. The test automation scripts are -defined in the `/tests/` folder. They are written in python -and interact through the uart with the test application code running on a -board to do the validation. It is recommended to flash the board with the -test just before running it because some platforms cannot be reset while -testing. - -From the test application directory run: - - BOARD= make flash test - - -An automated way of knowing if a test is available is to execute the -'test/available' target from the test application directory. -It executes without error if tests run by 'make test' are present. - - make test/available - - -Automated Tests Guidelines --------------------------- - +## Automated Tests Guidelines When using `pexpect` `$` is useless for matching the end of a line, instead use `\r\n`([pexpect end-of-line](https://pexpect.readthedocs.io/en/stable/overview.html#find-the-end-of-line-cr-lf-conventions)). @@ -93,8 +144,7 @@ pattern like `\r\n`, `\s`, `\)`, etc.. child.expect(r'some string: (\d+) ,') ~~~ -Use expect() instead of assert() --------------------------------- +## Use expect() instead of assert() In order to make a test application functional in all cases, use `expect()` instead of `assert()`. The former works like the latter, but will still be @@ -104,8 +154,7 @@ be compiled with that flag. Otherwise, the application would force compiling all tested code with assertions enabled. `expect()` is defined in the header `test_utils/expect.h`. -Interaction through the uart ----------------------------- +## Interaction through the uart Tests implemented with `testrunner` use the `cleanterm` target that provides an interaction without adding extra text output or input handling. @@ -117,22 +166,3 @@ The expected behavior is verified with the test in `tests/test_tools`. Tests cannot rely on having on all boards and terminal programs: * unbuffered input * allowing sending special characters like `ctrl+c/ctrl+d` - - -Running tests that require a preliminary manual configuration -------------------------------------------------------------- - -Some tests need active monitoring or manual setup steps but still have some -automated scripts. The test automation scripts are defined in the -`/tests-with-config/` folder. -For running them, follow the setup or analysis documentation and use the -`test-with-config` target. - -Running tests that require root privileges ------------------------------------------- - -Some tests require root privileges to launch their automated script. In this -case, the test automation scripts are defined in the -`/tests-as-root/` folder. -For running them, follow the setup or analysis documentation and use the -`test-as-root` target.