Release 2021.01 - RC1 #202
Labels
Comments
10. Task 01 - Destination unreachable - no route to destinationPASSEDCaptured stdout callCaptured log call |
10. Task 02 - Destination unreachable - Beyond scope of source addressPASSEDCaptured stdout callCaptured log call |
10. Task 03 - Destination unreachable - address unreachable (target node address)PASSEDCaptured stdout call |
10. Task 04 - Destination unreachable - address unreachable (neighbor cache miss)PASSEDCaptured stdout callCaptured log call |
10. Task 05 - Destination unreachable - port unreachablePASSEDCaptured stdout call |
10. Task 06 - Destination unreachable - port unreachable (large payload)PASSEDCaptured stdout callCaptured log call |
10. Task 07 - Packet too bigPASSEDCaptured stdout callCaptured log call |
10. Task 08 - Time exceeded - hop limit exceeded in transitPASSEDCaptured stdout callCaptured log call |
10. Task 09 - Parameter problem - erroneous header field encounteredPASSEDCaptured stdout callCaptured log call |
10. Task 10 - IPv6-in-IPv6 encapsulationPASSEDCaptured stdout call |
03. Task 01 - ICMPv6 multicast echo on nativePASSEDCaptured stdout call |
04. Task 01 - ICMPv6 link-local echo with iotlab-m3PASSEDCaptured stdout call |
03. Task 02 - ICMPv6 link-local echo on nativePASSEDCaptured stdout call |
04. Task 02 - ICMPv6 multicast echo with iotlab-m3/samr21-xproPASSEDCaptured stdout call |
04. Task 03 - ICMPv6 echo with large payloadPASSEDCaptured stdout call |
04. Task 09 - ICMPv6 stress test on iotlab-m3PASSEDCaptured stdout call |
04. Task 10 (Exprimental) - ICMPv6 echo with large payload (IPv6 fragmentation)PASSEDCaptured stdout call |
05. Task 02 - ICMPv6 echo unicast addresess on iotlab-m3 (default route)PASSEDCaptured stdout call |
06. Task 01 - UDP on iotlab-m3PASSEDCaptured stdout call |
06. Task 02 - UDP on iotlab-m3 (UDP port compression)PASSEDCaptured stdout call |
06. Task 04 - UDP on iotlab-m3 (non-existent neighbor)PASSEDCaptured stdout call |
06. Task 06 - Empty UDP on iotlab-m3PASSEDCaptured stdout call |
07. Task 01 - ICMPv6 echo on iotlab-m3 with three hops (static route)PASSEDCaptured stdout setupCaptured stdout call |
07. Task 03 - ICMPv6 echo on iotlab-m3 with three hops (RPL route)PASSEDCaptured stdout setupCaptured stdout call |
03. Task 04 - ICMPv6 stress test on native (1 hour)PASSEDCaptured stdout call |
07. Task 04 - UDP on iotlab-m3 with three hops (RPL route)PASSEDCaptured stdout setupCaptured stdout call |
03. Task 06 - ICMPv6 link-local echo on native (IPv6 fragmentation)PASSEDCaptured stdout call |
05. Task 01 - ICMPv6 echo unicast addresess on native (default route)PASSEDCaptured stdout call |
05. Task 03 - ICMPv6 echo unicast addresess on native (specific route)PASSEDCaptured stdout call |
03. Task 03 - UDP exchange between iotlab-m3 and ContikiPASSED
|
Perfect! Then it's clear it was the RTC of those IoT LAB nodes |
04. Task 07 (Experimental) - ICMPv6 multicast echo with samr21-xpro/zero + xbeePASSEDCaptured stdout call |
|
I've notices xbee still sometimes fails to initialize correctly. |
|
This are the tests results for 02-tests/01 (native). Most of them are false positives (missing tools or ran without sudo). I will investigate
FAILED
FAILED (known bug, fixed with RIOT-OS/RIOT#15771)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (when executed manually)
PASS (floating point seems to be rounded differently depending of the compiler)
PASS (floating point seems to be rounded differently depending of the compiler)EDIT: I added checkboxes to mark when a test.fail is clarified. |
|
btw I noticed there are some tests that shouldn't be executed on native (e.g How to proceed with this? In the short term we could blacklist native for those tests, but I think in the long run we should find a better mechanism... |
02. Task #02 - Unittests on nativePASSED |
01. Task #03 - Unittests on native separatedPASSED |
01. Task #04 - Unittests on iotlab-m3PASSED |
02. compile_and_test_for_board (iotlab-m3)EDIT: I ran the tests with a local board instead of IoT-LAB and got rid of all flashing errors.
PASS (ran by @fjmolinas)
FAILED (as expected)
PASS (ran by @miri64)
FAILED (see RIOT-OS/RIOT#15856)
FAILED
FAILED (as expected)
FAILED (as expected)
FAILED |
02. compile_and_test_for_board (samr21-xpro)Most of them are false positives, I will check them one by one EDIT: I run these tests this time without iotlab and got rid of the flash errors. This list is more accurate:
PASS (ran by @fjmolinas)
FAILED (expected to fail)
PASSED (ran by @miri64)
FAILED
FAILED (it seems it never passed :/)
FAILED (expected to fail)
FAILED (expected to fail)
FAILED
FAILED |
04. Task #11 (Experimental) - ICMPv6 stress test on nrf802154PASSED04. Task #12 (Experimental) - ICMPv6 multicast echo with iotlab-m3/nrf802154PASSED04. Task #13 (Experimental) - ICMPv6 link-local echo with iotlab-m3/nrf802154PASSED (the first reported FAILED was because the test script calls `pktbuf` immediately. The pinger was still receiving the reply. A small delay fixed it)I suspect the failure in the last one just because of a race condition when calling pktbuf. I ran this test before and didn't see this usse. I will repeat the test just to be sure |
I can confirm this. I added a small delay in the test and now it passes. Maybe we should update that in the test scripts |
Task #11 - UDP exchange between iotlab-m3 and ZephyrPASSEDRIOT nodeZephyr nodeNote: Tested against current master of Zephyr (zephyrproject-rtos/zephyr@6ae2374). Noticed some random crashes on the zephyr node after echoing the UDP messages, but it was not always the case. |
99. Task 01 - - Run tests on different hardwaresFAILEDdwm1001/failuresummaryFailures during test: pba-d-01-kw2x/failuresummaryFailures during test:
nucleo-f207zg/failuresummaryFailures during test: nucleo-f334r8/failuresummaryFailures during test:
hifive1b/failuresummaryFailures during test:
nrf52dk/failuresummaryFailures during test: frdm-k64f/failuresummaryFailures during test:
nucleo-l152re/failuresummaryFailures during test: arduino-uno/failuresummarynucleo-f767zi/failuresummaryFailures during test: p-nucleo-wb55/failuresummaryFailures during test: nucleo-l496zg/failuresummaryFailures during test: nucleo-l452re/failuresummaryFailures during test: frdm-kw41z/failuresummaryFailures during test:
nucleo-l433rc/failuresummaryFailures during test: arduino-mega2560/failuresummaryFailures during test:
ek-lm4f120xl/failuresummaryFailures during test: i-nucleo-lrwan1/failuresummaryFailures during test:
usb-kw41z/failuresummaryFailures during test:
z1/failuresummaryFailures during test: nucleo-f091rc/failuresummaryFailures during test: Failures during test.flash: arduino-zero/failuresummaryFailures during test: nucleo-f303k8/failuresummaryFailures during test: nucleo-f103rb/failuresummaryFailures during test: b-l072z-lrwan1/failuresummaryFailures during test: atmega256rfr2-xpro/failuresummaryFailures during test:
nucleo-f303re/failuresummaryFailures during test: slstk3402a/failuresummaryFailures during test: nucleo-l073rz/failuresummaryFailures during test: Failures during test.flash: I ran the test on multiple BOARD's, all results are here, I filtered out what where expected to fail or false positives, some I already fixed in some PR but there are still other tests that should be looked at: |
05. Task #05 - ICMPv6 stress test on native (neighbor cache stress)XPASS |
08. Task #05 - ICMPv6 echo between iotlab-m3 and Internet host through RIOT border routerPASSED |
08. Task #02 - ICMPv6 echo between iotlab-m3 and Linux with 6LowPANPASSED |
08. Task #06 - UDP between iotlab-m3 and Internet host through RIOT border routerPASSED08. Task #07 - UDP between iotlab-m3 and Internet host through RIOT border router (200b payload)PASSED |
08. Task 12 - GNRC Border Router with WiFi uplink to the InternetPASSED08. Task 13 - GNRC NetworkingPASSEDI had to manually change the channel for this one on a Linux host |
02. compile_and_test_for_board (samr21-xpro)
SUCCESSI don;'t have an m3 with me ATM, will run the test when home. |
02. compile_and_test_for_board (iotlab-m3)
SUCCESS |
08. Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPANPASSED |
|
Ok, with RIOT-OS/RIOT#15869 finished, we officially have 100% release tests coverage for samr21-xpro, iotlab-m3 and native (although these compile tests points to the upcoming RC-2 ;) There are some XFAIL regarding tests with configurations (all of them solved with RIOT-OS/RIOT#15771)
Some that were fixed before RC2:
Some open bugs:
I will simply push the RC-2 and try to focus on these bugs. I will make sure that these at least get properly documented in case we don't find the bugs. |
|
Closing in favor of #206 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue lists the status of all tests for the Release Candidate 1 of the 2021.01 release.
Specs tested:
Task Release 2015.12 - pre-RC1 #9 - deprecatedTask Release 2015.12 - RC1 #10 - deprecatedThe text was updated successfully, but these errors were encountered: