tests: add test-with-config and test-as-root targets #15771
Conversation
aabadie
added
Type: enhancement
aabadie
requested review from
fjmolinas,
leandrolanzieri,
MichelRottleuthner,
miri64 and
smlng
as code owners
Is there an explanation why that is? |
aabadie
force-pushed
the
review_test_manual
branch
from
32122e7 to
e6a4756
Compare
#15772 gives an answer and a temporary workaround for |
|
Murdock is reporting failures (see this one as an example) but I don't understand the problem. Maybe @leandrolanzieri has an idea ? |
It looks that with this changes the test-related modules are not pulled in anymore in the Makefile dependency resolution (e.g. env BOARD=samr21-xpro make info-modulesenv BOARD=samr21-xpro TEST_KCONFIG=1 make info-modulesThis causes the binaries to differ. |
Thanks, that makes sense. Let's see if my latest changes fix the problem. |
This is the case. Thanks for the analysis @leandrolanzieri |
aabadie
force-pushed
the
review_test_manual
branch
from
11ca63a to
048ecd8
Compare
aabadie
force-pushed
the
review_test_manual
branch
from
048ecd8 to
f996e3a
Compare
I added commits to adapt the mentioned tests. For some of them the Python test script is also adapted because it's not supposed to fail because of a missing device (since the configuration should provide it). |
I'm ok with the change, with a wrong configuration the initialization still causes a hard-fault in many cases so IMO this is an improvement. |
|
@MrKevinWeiss does this have any impact on the HIL setup? |
|
We should also add the |
It should be fine, no blockers here. |
fjmolinas
added
CI: run tests
If the configuration is correct, the test should succeed, so the test script is adapted
aabadie
force-pushed
the
review_test_manual
branch
from
8021f2b to
d0f3cef
Compare
There was a problem hiding this comment.
ACK, this is quite an important change but maintainers involved in testing @miri64 @leandrolanzieri @MrKevinWeiss have been aware or have taken a look at this PR. This will help with cleaning our testing output of false positives. Murdock does its own thing so it still works as before.
|
Thanks everyone for reviewing and testing! Thanks @cladmi for the original work :) |
|
I'm not able to run tests with sudo :( |
Do you have an error message ? What is your environment ? |
|
it doesn't find testrunner. |
|
For me, that works $ sudo -E PYTHONPATH=testValue bash -c 'echo $PYTHONPATH'
testValueCan you please provide the |
sudo: |
|
So mostly the same as mine 😕 |
Do you have any |
|
hi @cladmi!! :) No, I'm not using NOPASSWD :/ |
|
What would be the next steps for this PR? How should we list when some config is provided? |
Contribution description
Some tests are always failing unless there's a prior local and manual configuration (configure a tap, configure pins, etc) or the test script is launched with root permissions (the ones that use scapy for instance).
This PR is taking over and enhancing the preliminary work started in #11954 by adding 2 new test targets:
test-with-configandtest-as-root.test-with-configexpects a preliminary configuration, like creating a tap or setting up the right pins.test-as-rootallows to startmake testusingsudo. Some affected gnrc tests would need both (configuring a tap and running as root), for them, thetest-as-rootis kept, but a manual preliminary step is still required (configuring a tap interface or starting ethos).This PR is mainly targetting the
test-on-iotlabGH action where a lot of tests are failing because of missing configuration. With this PR most of them could be fixed.This PR is also useful with using
compile_and_test_for_boardwithout root permission or preliminary configurations.Testing procedure
./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <board> --with-test-only --jobs=4. Verify that there are no unrelated failing teststest-on-iotlabGH action should only havetests/riotboot,examples/suit_updateandtests/pkg_libfixmath-unitteststests that failexamples/suit_update,examples/lorawan,tests/pkg_semtech-loramac,tests/pkg_fatfs_vfs,tests/driver_at86rf2xx_aes,tests/driver_ds1307,tests/driver_hd44780,tests/driver_my9221andtests/driver_grove_ledbarshould pass with the following command (none tested):test-with-configtarget withtests/gnrc_sock_dns:test-as-roottests (only tested on native andtests/gnrc_dhcpv6_clientis untested):make -C tests/emcute flash test-as-root
make -C tests/gnrc_dhcpv6_client_6lbr flash test-as-root
make -C tests/ flash test-as-root
make -C tests/gnrc_ipv6_ext_frag flash test-as-root
make -C tests/gnrc_ipv6_ext_opt flash test-as-root
make -C tests/gnrc_ipv6_nib_dns flash test-as-root
make -C tests/gnrc_rpl_srh flash test-as-root
make -C tests/gnrc_tcp flash test-as-root
Issues/PRs references
take-over of #11954