Unit Testing for embedded #408

ivankravets · 2015-12-28T11:09:01Z

The issue is an addition to #258. Requested by @Oaz, @kireevco.

How will it work? Nothing new here, general Unit testing design

Developer writes tests (using assert function/macro TBD) and places them into test folder
Run tests pio test

PlatformIO will use each test as source code for the new project and the source code of existing project will be used as input library. The developer writes tests for the specific part of code, PlatformIO uploads it on-the-fly, tests, gathers results, shows reports to the developer.

Comments are welcome.

The text was updated successfully, but these errors were encountered:

ivankravets · 2015-12-28T11:40:59Z

Tip: @ThrowTheSwitch https://github.com/ThrowTheSwitch/Unity

eerimoq · 2015-12-30T12:34:39Z

This is how I normally writes tests: https://github.com/eerimoq/simba/blob/master/tst/kernel/event/main.c. It is very basic and works very well for me. There are plenty of examples in the tst-folder in that repository.

Then I build, upload and run on on target with make test, and I get the result. For arduino, the test report is written to the serial port. On native linux, stdout.

This example run is on native linux. I don't have an Embedded device nearby. I can supply the output from that as well later.

erik@erik-VirtualBox ~/workspace/simba/tst/kernel/event 
> make -s test
Generating settings.bin from ../../../make/settings.ini
Generating settings.h from ../../../make/settings.ini
Compiling main.c
Compiling ../../../src/boards/linux/board.c
Compiling ../../../src/mcus/linux/mcu.c
Compiling ../../../src/kernel/chan.c
Compiling ../../../src/kernel/event.c
Compiling ../../../src/kernel/fs.c
Compiling ../../../src/kernel/log.c
Compiling ../../../src/kernel/queue.c
Compiling ../../../src/kernel/sem.c
Compiling ../../../src/kernel/setting.c
Compiling ../../../src/kernel/shell.c
Compiling ../../../src/kernel/std.c
Compiling ../../../src/kernel/sys.c
Compiling ../../../src/kernel/thrd.c
Compiling ../../../src/kernel/time.c
Compiling ../../../src/kernel/timer.c
Compiling ../../../src/drivers/pin.c
Compiling ../../../src/drivers/sd.c
Compiling ../../../src/drivers/spi.c
Compiling ../../../src/drivers/uart.c
Compiling ../../../src/slib/crc.c
Compiling ../../../src/slib/fat16.c
Compiling ../../../src/slib/harness.c
Compiling ../../../src/slib/hash_map.c
Compiling ../../../src/slib/midi.c
Compiling gen/simba_gen.c
Linking event_suite.out
Running event_suite.out
./event_suite.out
app:   event_suite-0.0.0 built 2015-12-30 13:37 CET by erik.
board: Linux
mcu:   Linux
enter: test_read_write
exit: test_read_write: PASSED
enter: test_poll
exit: test_poll: PASSED
enter: test_poll_timeout
exit: test_poll_timeout: PASSED
            NAME           PARENT        STATE  PRIO   CPU  LOGMASK
            main                     suspended     0    0%     0x3f
            idle             main        ready   127    0%     0x3f
         monitor             main    suspended   -80    0%     0x3f
harness report: total(3), passed(3), failed(0)
event_suite:
exit: test_read_write: PASSED
exit: test_poll: PASSED
exit: test_poll_timeout: PASSED
erik@erik-VirtualBox ~/workspace/simba/tst/kernel/event 
>

ivankravets · 2015-12-30T12:40:32Z

@eerimoq could you provide test-output from AVR?

eerimoq · 2015-12-30T12:50:30Z

Tonight, when I get home. I'll add output for Arduino Nano and Arduino Due. It's pretty much the same as the above though. Ideally, the output should be the same independently of what hardware it is executed on. Except cases where a device has too little memory and the whole test suite will not fit.

ivankravets · 2015-12-30T12:56:45Z

@eerimoq I agree with you. Now need to decide how to implement it: write own UNIT Tests framework or use existing solutions, like @ThrowTheSwitch https://github.com/ThrowTheSwitch/Unity

eerimoq · 2015-12-30T13:30:44Z

It's probably useful for many programmers to get the test framework generated when the create a project, but, also, I think platformio should provide a high level API that makes it possible to communicate with platformio with any test framework.

ivankravets · 2015-12-30T13:36:02Z

Good. I'll wait output from AVR platform

eerimoq · 2015-12-30T16:56:42Z

@ivankravets

Here we go.

Arduino Nano

erik@erik-PC ~/simba/simba/tst/kernel/event
$ make -s BOARD=arduino_nano test
Generating settings.bin from ../../../make/settings.ini
Generating settings.h from ../../../make/settings.ini
Compiling main.c
Compiling ../../../src/boards/arduino_nano/board.c
Compiling ../../../src/mcus/atmega328p/mcu.c
Compiling ../../../src/kernel/chan.c
Compiling ../../../src/kernel/event.c
Compiling ../../../src/kernel/fs.c
Compiling ../../../src/kernel/log.c
Compiling ../../../src/kernel/queue.c
Compiling ../../../src/kernel/sem.c
Compiling ../../../src/kernel/setting.c
Compiling ../../../src/kernel/shell.c
Compiling ../../../src/kernel/std.c
Compiling ../../../src/kernel/sys.c
Compiling ../../../src/kernel/thrd.c
Compiling ../../../src/kernel/time.c
Compiling ../../../src/kernel/timer.c
Compiling ../../../src/drivers/adc.c
Compiling ../../../src/drivers/ds18b20.c
Compiling ../../../src/drivers/ds3231.c
Compiling ../../../src/drivers/exti.c
Compiling ../../../src/drivers/mcp2515.c
Compiling ../../../src/drivers/nrf24l01.c
Compiling ../../../src/drivers/spi.c
Compiling ../../../src/drivers/owi.c
Compiling ../../../src/drivers/pin.c
Compiling ../../../src/drivers/pwm.c
Compiling ../../../src/drivers/sd.c
Compiling ../../../src/drivers/uart.c
Compiling ../../../src/drivers/uart_soft.c
Compiling ../../../src/slib/crc.c
Compiling ../../../src/slib/fat16.c
Compiling ../../../src/slib/harness.c
Compiling ../../../src/slib/hash_map.c
Compiling ../../../src/slib/midi.c
Compiling gen/simba_gen.c
Linking event_suite.out
Running event_suite.out
avrdude -p atmega328p -D -P COM3 -c arduino -V -b 57600 -U eeprom:w:settings.bin:r

avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude.exe: Device signature = 0x1e950f
avrdude.exe: reading input file "settings.bin"
avrdude.exe: writing eeprom (1024 bytes):

Writing | ################################################## | 100% 8.19s

avrdude.exe: 1024 bytes of eeprom written

avrdude.exe done.  Thank you.

avrdude -p atmega328p -D -P COM3 -c arduino -V -b 57600 -U flash:w:event_suite.hex

avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude.exe: Device signature = 0x1e950f
avrdude.exe: reading input file "event_suite.hex"
avrdude.exe: input file event_suite.hex auto detected as Intel Hex
avrdude.exe: writing flash (22476 bytes):

Writing | ################################################## | 100% 11.26s

avrdude.exe: 22476 bytes of flash written

avrdude.exe done.  Thank you.

../../../make/run.py harness report: total\(\d+\), passed\(\d+\), failed\(\d+\) harness report: total\(\d+\), passed\(\d+\), failed\(0\)
app:   event_suite-0.0.0 built 2015-12-30 17:52 CET by erik.
board: Arduino Nano
mcu:   Atmel ATMega328p AVR @ 16MHz, 2k sram, 32k flash
enter: test_read_write
exit: test_read_write: PASSED
enter: test_poll
exit: test_poll: PASSED
enter: test_poll_timeout
exit: test_poll_timeout: PASSED
            NAME           PARENT        STATE  PRIO   CPU  MAX-STACK-USAGE  LOGMASK
            main                       current     0    0%       315/   931     0x3f
            idle             main        ready   127    0%        55/   156     0x3f
         monitor             main    suspended   -80    0%       104/   256     0x3f
harness report: total(3), passed(3), failed(0)
event_suite:
exit: test_read_write: PASSED
exit: test_poll: PASSED
exit: test_poll_timeout: PASSED

erik@erik-PC ~/simba/simba/tst/kernel/event
$

Arduino Due

erik@erik-PC ~/simba/simba/tst/kernel/event
$ make -s BOARD=arduino_due test
Generating settings.bin from ../../../make/settings.ini
Generating settings.h from ../../../make/settings.ini
Compiling main.c
Compiling ../../../src/boards/arduino_due/board.c
Compiling ../../../src/mcus/sam/sam3.c
Compiling ../../../src/mcus/sam/mcu.c
Compiling settings.c
Compiling ../../../src/kernel/chan.c
Compiling ../../../src/kernel/event.c
Compiling ../../../src/kernel/fs.c
Compiling ../../../src/kernel/log.c
Compiling ../../../src/kernel/queue.c
Compiling ../../../src/kernel/sem.c
Compiling ../../../src/kernel/setting.c
Compiling ../../../src/kernel/shell.c
Compiling ../../../src/kernel/std.c
Compiling ../../../src/kernel/sys.c
Compiling ../../../src/kernel/thrd.c
Compiling ../../../src/kernel/time.c
Compiling ../../../src/kernel/timer.c
Compiling ../../../src/drivers/adc.c
Compiling ../../../src/drivers/can.c
Compiling ../../../src/drivers/chipid.c
Compiling ../../../src/drivers/dac.c
Compiling ../../../src/drivers/exti.c
Compiling ../../../src/drivers/flash.c
Compiling ../../../src/drivers/mcp2515.c
Compiling ../../../src/drivers/pin.c
Compiling ../../../src/drivers/sd.c
Compiling ../../../src/drivers/spi.c
Compiling ../../../src/drivers/uart.c
Compiling ../../../src/drivers/usb.c
Compiling ../../../src/drivers/usb_host.c
Compiling ../../../src/drivers/usb/host/class/usb_host_class_hid.c
Compiling ../../../src/drivers/usb/host/class/usb_host_class_mass_storage.c
Compiling ../../../src/slib/crc.c
Compiling ../../../src/slib/fat16.c
Compiling ../../../src/slib/harness.c
Compiling ../../../src/slib/hash_map.c
Compiling ../../../src/slib/midi.c
Compiling gen/simba_gen.c
Linking event_suite.out
Running event_suite.out
Setting /dev/arduino to 1200 baud and setting DTR to reset the board.
Erase flash
Write 40888 bytes to flash
[==============================] 100% (160/160 pages)
Set boot flash true
app:   event_suite-0.0.0 built 2015-12-30 17:55 CET by erik.
board: Arduino Due
mcu:   Atmel SAM3X8E Cortex-M3 @ 84MHz, 96k sram, 512k flash
enter: test_read_write
exit: test_read_write: PASSED
enter: test_poll
exit: test_poll: PASSED
enter: test_poll_timeout
exit: test_poll_timeout: PASSED
            NAME           PARENT        STATE  PRIO   CPU  MAX-STACK-USAGE  LOGMASK
            main                       current     0    0%       612/ 93070     0x3f
            idle             main        ready   127    0%       160/   258     0x3f
         monitor             main    suspended   -80    0%       140/   514     0x3f
harness report: total(3), passed(3), failed(0)
event_suite:
exit: test_read_write: PASSED
exit: test_poll: PASSED
exit: test_poll_timeout: PASSED

erik@erik-PC ~/simba/simba/tst/kernel/event
$

ivankravets · 2016-05-06T18:57:13Z

Another unit test framework for Arduino https://github.com/mmurdoch/arduinounit

ivankravets · 2016-06-15T11:12:52Z

Docs: http://docs.platformio.org/en/feature-platformio-30/platforms/unit_testing.html

#742, #459, #542, #763, #759, #753, #757, #749, #748, #745, #519, #709, #743, #413, #498, #410, #740, #361, #414, #554, #732, #588, #475, #461, #101, #719, #721, #537, #415, #522, #289, #556, #570, #456, #617, #432, #408, #479, #667, #510)

* develop: Fix incorrect line order when converting from INO to CPP and pointer is used Fix unit test Notify about `version` field when creating library Add support for SparkFun Blynk Board Return valid exit code from ``plaformio test`` command Disable SSL Server-Name-Indication for Python < 2.7.9 Version bump to 3.0.1 (issue #772) Disable temporary SSL for PlatformIO services // Resolve #772 Version bump to 3.0.0 (issues #770, #766, #747, #730, #765, #640, #659, #742, #459, #542, #763, #759, #753, #757, #749, #748, #745, #519, #709, #743, #413, #498, #410, #740, #361, #414, #554, #732, #588, #475, #461, #101, #719, #721, #537, #415, #522, #289, #556, #570, #456, #617, #432, #408, #479, #667, #510) Fix menu height for docs Fix issue with multiple archives when linking firmware Add migration guide for PIO2 to PIO3 Search libraries by headers/includes with ``platformio lib search --header`` option Update pio run command examples Add Unit Testing Demo Update PIO Plus badge title and link Add PlatformIO Plus badge Add links to PlatformIO Plus

ivankravets added the feature label Dec 28, 2015

ivankravets self-assigned this Dec 28, 2015

ivankravets added this to the 3.0.0 milestone Dec 28, 2015

ivankravets mentioned this issue Jan 1, 2016

PlatformIO 3.0 Roadmap #423

Closed

37 tasks

ivankravets mentioned this issue Jan 12, 2016

Travis CI check randomly fails PaulStoffregen/Audio#149

Open

ivankravets assigned valeros and unassigned ivankravets Feb 15, 2016

neuhalje mentioned this issue Feb 15, 2016

Add Support for local ("PC") unit tests #519

Closed

valeros added a commit that referenced this issue May 30, 2016

Initial support for unit testing // Issue #408

bfd66de

ivankravets added a commit that referenced this issue Jun 15, 2016

Unit Testing for Embedded // Resolve #408

bb22a12

ivankravets closed this as completed Jun 15, 2016

ivankravets added a commit that referenced this issue Jun 15, 2016

Allow to ignore tests using glob patterns // Issue #408

8a67ea9

ivankravets changed the title ~~Unit testing for embedded~~ Unit Testing for embedded Jun 16, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unit Testing for embedded #408

Unit Testing for embedded #408

ivankravets commented Dec 28, 2015 •

edited

Loading

ivankravets commented Dec 28, 2015

eerimoq commented Dec 30, 2015

ivankravets commented Dec 30, 2015

eerimoq commented Dec 30, 2015

ivankravets commented Dec 30, 2015

eerimoq commented Dec 30, 2015

ivankravets commented Dec 30, 2015

eerimoq commented Dec 30, 2015

ivankravets commented May 6, 2016

ivankravets commented Jun 15, 2016

Unit Testing for embedded #408

Unit Testing for embedded #408

Comments

ivankravets commented Dec 28, 2015 • edited Loading

ivankravets commented Dec 28, 2015

eerimoq commented Dec 30, 2015

ivankravets commented Dec 30, 2015

eerimoq commented Dec 30, 2015

ivankravets commented Dec 30, 2015

eerimoq commented Dec 30, 2015

ivankravets commented Dec 30, 2015

eerimoq commented Dec 30, 2015

ivankravets commented May 6, 2016

ivankravets commented Jun 15, 2016

ivankravets commented Dec 28, 2015 •

edited

Loading