Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add example conformance report tool #36913

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Add example conformance report tool #36913

wants to merge 4 commits into from
+382 −0

Conversation

soares-sergio
Copy link
Contributor

@soares-sergio soares-sergio commented Dec 19, 2024 •
edited
Loading

This PR adds a new tool that builds Matter SDK examples, runs conformance tests, parses logs and generates an HTML report.

This tool will run daily, together with the coverage report and publish it here: https://matter-build-automation.ue.r.appspot.com/conformance_report.html

This initial list of examples includes most of the apps built in CI. I will add more examples later.

The goal with this tool is to bring more visibility to the conformance issues in the SDK examples (which might end up in real products) and hopefully drive improvements and collaboration across the team.

image

@semanticdiff-com SemanticDiff.com
Copy link

semanticdiff-com bot commented Dec 19, 2024 •
edited
Loading

Review changes with  SemanticDiff

Changed Files
File Status
  integrations/appengine/webapp_config.yaml Unsupported file format
  integrations/compute_engine/startup-script.sh Unsupported file format
  scripts/examples/conformance_report.py  0% smaller

@soares-sergio soares-sergio requested a review from a team as a code owner December 19, 2024 21:53
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 19, 2024 •
edited
Loading

PR #36913: Size comparison from 388be5b to b4f5d8a

Full report (69 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
platform target config section 388be5b b4f5d8a change % change
bl602 lighting-app bl602+mfd+littlefs+rpc FLASH 1354380 1354380 0 0.0
RAM 104160 104160 0 0.0
bl702 lighting-app bl702+eth FLASH 651976 651976 0 0.0
RAM 25353 25353 0 0.0
bl702+wifi FLASH 829548 829548 0 0.0
RAM 14093 14093 0 0.0
bl706+mfd+rpc+littlefs FLASH 1057764 1057764 0 0.0
RAM 23933 23933 0 0.0
bl702l lighting-app bl702l+mfd+littlefs FLASH 979138 979138 0 0.0
RAM 16596 16596 0 0.0
cc13x4_26x4 lighting-app LP_EM_CC1354P10_6 FLASH 839944 839944 0 0.0
RAM 123696 123696 0 0.0
lock-ftd LP_EM_CC1354P10_6 FLASH 825484 825484 0 0.0
RAM 125584 125584 0 0.0
pump-app LP_EM_CC1354P10_6 FLASH 772208 772208 0 0.0
RAM 114060 114060 0 0.0
pump-controller-app LP_EM_CC1354P10_6 FLASH 756388 756388 0 0.0
RAM 114260 114260 0 0.0
cc32xx air-purifier CC3235SF_LAUNCHXL FLASH 539705 539705 0 0.0
RAM 205800 205800 0 0.0
lock CC3235SF_LAUNCHXL FLASH 574025 574025 0 0.0
RAM 205944 205944 0 0.0
cyw30739 light CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 681561 681561 0 0.0
RAM 78756 78756 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 701413 701413 0 0.0
RAM 81396 81396 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 701413 701413 0 0.0
RAM 81396 81396 0 0.0
CYW930739M2EVB-02 unknown 2040 2040 0 0.0
FLASH 658341 658341 0 0.0
RAM 73824 73824 0 0.0
light-switch CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 618193 618193 0 0.0
RAM 71748 71748 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 637821 637821 0 0.0
RAM 74292 74292 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 637821 637821 0 0.0
RAM 74292 74292 0 0.0
lock CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 637585 637585 0 0.0
RAM 74756 74756 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 657293 657293 0 0.0
RAM 77300 77300 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 657293 657293 0 0.0
RAM 77300 77300 0 0.0
thermostat CYW30739B2-P5-EVK-01 unknown 2040 2040 0 0.0
FLASH 614045 614045 0 0.0
RAM 68844 68844 0 0.0
CYW30739B2-P5-EVK-02 unknown 2040 2040 0 0.0
FLASH 633897 633897 0 0.0
RAM 71476 71476 0 0.0
CYW30739B2-P5-EVK-03 unknown 2040 2040 0 0.0
FLASH 633897 633897 0 0.0
RAM 71476 71476 0 0.0
efr32 lock-app BRD4187C FLASH 932452 932452 0 0.0
RAM 160228 160228 0 0.0
BRD4338a FLASH 746704 746696 -8 -0.0
RAM 233356 233356 0 0.0
window-app BRD4187C FLASH 1025368 1025368 0 0.0
RAM 128332 128332 0 0.0
esp32 all-clusters-app c3devkit DRAM 95352 95352 0 0.0
FLASH 1541618 1541618 0 0.0
IRAM 82552 82552 0 0.0
m5stack DRAM 116332 116332 0 0.0
FLASH 1548058 1548058 0 0.0
IRAM 117039 117039 0 0.0
linux air-purifier-app debug unknown 4720 4720 0 0.0
FLASH 2716981 2716981 0 0.0
RAM 130120 130120 0 0.0
all-clusters-app debug unknown 5560 5560 0 0.0
FLASH 6010628 6010628 0 0.0
RAM 523816 523816 0 0.0
all-clusters-minimal-app debug unknown 5456 5456 0 0.0
FLASH 5347104 5347104 0 0.0
RAM 242904 242904 0 0.0
bridge-app debug unknown 5440 5440 0 0.0
FLASH 4686672 4686672 0 0.0
RAM 218752 218752 0 0.0
chip-tool debug unknown 5992 5992 0 0.0
FLASH 12846546 12846546 0 0.0
RAM 582594 582594 0 0.0
chip-tool-ipv6only arm64 unknown 21360 21360 0 0.0
FLASH 10980944 10980944 0 0.0
RAM 633536 633536 0 0.0
fabric-admin debug unknown 5816 5816 0 0.0
FLASH 11253095 11253095 0 0.0
RAM 582946 582946 0 0.0
fabric-bridge-app debug unknown 4696 4696 0 0.0
FLASH 4511908 4511908 0 0.0
RAM 205920 205920 0 0.0
fabric-sync debug unknown 4936 4936 0 0.0
FLASH 5608613 5608613 0 0.0
RAM 472888 472888 0 0.0
lighting-app debug+rpc+ui unknown 6104 6104 0 0.0
FLASH 5623361 5623361 0 0.0
RAM 229080 229080 0 0.0
lock-app debug unknown 5376 5376 0 0.0
FLASH 4735912 4735912 0 0.0
RAM 205064 205064 0 0.0
ota-provider-app debug unknown 4752 4752 0 0.0
FLASH 4361618 4361618 0 0.0
RAM 198752 198752 0 0.0
ota-requestor-app debug unknown 4688 4688 0 0.0
FLASH 4500644 4500644 0 0.0
RAM 203336 203336 0 0.0
shell debug unknown 4248 4248 0 0.0
FLASH 3034493 3034493 0 0.0
RAM 160736 160736 0 0.0
thermostat-no-ble arm64 unknown 9560 9560 0 0.0
FLASH 4105728 4105728 0 0.0
RAM 243344 243344 0 0.0
tv-app debug unknown 5704 5704 0 0.0
FLASH 5957893 5957893 0 0.0
RAM 596304 596304 0 0.0
tv-casting-app debug unknown 5288 5288 0 0.0
FLASH 11056637 11056637 0 0.0
RAM 692600 692600 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 FLASH 917928 917928 0 0.0
RAM 143332 143332 0 0.0
nrf7002dk_nrf5340_cpuapp FLASH 890536 890536 0 0.0
RAM 141519 141519 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 FLASH 851808 851808 0 0.0
RAM 142244 142244 0 0.0
nxp contact k32w0+release FLASH 585632 585632 0 0.0
RAM 71112 71112 0 0.0
mcxw71+release FLASH 600200 600200 0 0.0
RAM 63208 63208 0 0.0
light k32w0+release FLASH 612492 612492 0 0.0
RAM 70504 70504 0 0.0
k32w1+release FLASH 686696 686696 0 0.0
RAM 48840 48840 0 0.0
lock mcxw71+release FLASH 763040 763040 0 0.0
RAM 70876 70876 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 FLASH 1647260 1647260 0 0.0
RAM 212128 212128 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 FLASH 1554716 1554716 0 0.0
RAM 208944 208944 0 0.0
light cy8ckit_062s2_43012 FLASH 1470012 1470012 0 0.0
RAM 200912 200912 0 0.0
lock cy8ckit_062s2_43012 FLASH 1467748 1467748 0 0.0
RAM 225272 225272 0 0.0
qpg lighting-app qpg6105+debug FLASH 664136 664136 0 0.0
RAM 105456 105456 0 0.0
lock-app qpg6105+debug FLASH 621956 621956 0 0.0
RAM 99908 99908 0 0.0
stm32 light STM32WB5MM-DK FLASH 484856 484856 0 0.0
RAM 144912 144912 0 0.0
telink bridge-app tlsr9258a FLASH 683176 683176 0 0.0
RAM 91248 91248 0 0.0
contact-sensor-app tlsr9528a_retention FLASH 623594 623594 0 0.0
RAM 31488 31488 0 0.0
light-app-ota-compress-lzma-shell-factory-data tl3218x FLASH 772478 772478 0 0.0
RAM 49348 49348 0 0.0
light-switch-app-ota-compress-lzma-shell-factory-data tlsr9528a FLASH 711036 711036 0 0.0
RAM 73544 73544 0 0.0
lighting-app-ota-factory-data tlsr9118bdk40d FLASH 628040 628040 0 0.0
RAM 142180 142180 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d FLASH 814058 814058 0 0.0
RAM 99724 99724 0 0.0
tizen all-clusters-app arm unknown 5000 5000 0 0.0
FLASH 1735660 1735660 0 0.0
RAM 90888 90888 0 0.0
chip-tool-ubsan arm unknown 10808 10808 0 0.0
FLASH 17969006 17969006 0 0.0
RAM 7841376 7841376 0 0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
integrations review - pending scripts
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@soares-sergio @restyled-commits