Unit test always passes even if there is failure when first delay() in setup() is less than about 7500ms

[hideakitai]

HI, I have unit tested teensy 3.2 and 4.0, but the tests always pass even if _there is failure.
The environment is:

PlatformIO Core version 4.3.4
Teensy Platform version 4.9.0

[env:teensy31]
platform = teensy
board = teensy31
framework = arduino

For example, based on Blink unit test project, I have changed only this line to show failure.

TEST_ASSERT_EQUAL(12, LED_BUILTIN); // -> always fail

but the result is:

$ pio test
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

Processing * in teensy31 environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

0 Ignored
FAIL
-----------------------
11 Tests 1 Failures 0 Ignored
==================================================================================== [PASSED] Took 12.47 seconds ====================================================================================

Test    Environment    Status    Duration
------  -------------  --------  ------------
*       teensy31       PASSED    00:00:12.474
==================================================================================== 1 succeeded in 00:00:12.474 ====================================================================================

There is 1 Failures but test result is PASSED.
And when I change the first delay(2000) in setup() to delay(7500), I got test "failed correctly".

$ pio test
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

Processing * in teensy31 environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_main.cpp:13:test_led_builtin_pin_number:FAIL: Expected 12 Was 13      [FAILED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
-----------------------
11 Tests 1 Failures 0 Ignored
==================================================================================== [FAILED] Took 17.23 seconds ====================================================================================

Test    Environment    Status    Duration
------  -------------  --------  ------------
*       teensy31       FAILED    00:00:17.227
=============================================================================== 1 failed, 0 succeeded in 00:00:17.227 ===============================================================================

And when I use delay(7000), first two tests are missing (not shown on console).

$ pio test
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

Processing * in teensy31 environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
test/test_main.cpp:46:test_led_state_high       [PASSED]
test/test_main.cpp:48:test_led_state_low        [PASSED]
-----------------------
11 Tests 1 Failures 0 Ignored
==================================================================================== [PASSED] Took 17.03 seconds ====================================================================================

Test    Environment    Status    Duration
------  -------------  --------  ------------
*       teensy31       PASSED    00:00:17.026
==================================================================================== 1 succeeded in 00:00:17.026 ====================================================================================

So maybe the first delay() time is too short compared to other boards.
Can you fix it? Or It is difficult to fix and I should use more delay() than usual?

Thanks, in advance.

[hideakitai]

The test code (test/test_main.cpp) is here.

#include <Arduino.h>
#include <unity.h>

// void setUp(void) {
// // set stuff up here
// }

// void tearDown(void) {
// // clean stuff up here
// }

void test_led_builtin_pin_number(void) {
    TEST_ASSERT_EQUAL(12, LED_BUILTIN); // -> this test fails
}

void test_led_state_high(void) {
    digitalWrite(LED_BUILTIN, HIGH);
    TEST_ASSERT_EQUAL(HIGH, digitalRead(LED_BUILTIN));
}

void test_led_state_low(void) {
    digitalWrite(LED_BUILTIN, LOW);
    TEST_ASSERT_EQUAL(LOW, digitalRead(LED_BUILTIN));
}

void setup() {
    // NOTE!!! Wait for >2 secs
    // if board doesn't support software reset via Serial.DTR/RTS

    delay(2000);    // -> 1 unit test fails but pass (no test results)
    // delay(7000); // -> 1 unit test fails but pass (1 test result missing)
    // delay(7500); // -> test fails correctly

    UNITY_BEGIN();    // IMPORTANT LINE!
    RUN_TEST(test_led_builtin_pin_number);

    pinMode(LED_BUILTIN, OUTPUT);
}

uint8_t i = 0;
uint8_t max_blinks = 5;

void loop() {
    if (i < max_blinks)
    {
        RUN_TEST(test_led_state_high);
        delay(500);
        RUN_TEST(test_led_state_low);
        delay(500);
        i++;
    }
    else if (i == max_blinks) {
      UNITY_END(); // stop unit testing
    }
}

[hideakitai]

@ivankravets Hi, I have tested this issue on v4.11.0, now it should wait for more than 10seconds.