Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix run_testsuite.py so it works if not all of the apps are specified. #25600

Merged

Conversation

bzbarsky-apple
Copy link
Contributor

When running only a subset of tests (e.g. only DL_* tests), not all apps are needed for the tests to run.

This used to work because we used a fake "NOT_FOUND_IN_OUTPUT_" path for the missing apps, which was a string so the string manipulation on it worked, and the output was never used.

But after #25368 we end up with None where we expect a string, and the script dies.

The fix is just to skip string manipulation on the path of apps that have None as a path.

When running only a subset of tests (e.g. only DL_* tests), not all apps are
needed for the tests to run.

This used to work because we used a fake "NOT_FOUND_IN_OUTPUT_" path for the
missing apps, which was a string so the string manipulation on it worked, and
the output was never used.

But after project-chip#25368 we end up
with None where we expect a string, and the script dies.

The fix is just to skip string manipulation on the path of apps that have None
as a path.
@bzbarsky-apple bzbarsky-apple force-pushed the fix-test-definition-script branch from 63c0612 to 9d928d9 Compare March 10, 2023 01:13
@github-actions
Copy link

github-actions bot commented Mar 10, 2023

PR #25600: Size comparison from 7541b49 to 9d928d9

Increases (6 builds for psoc6, telink)
platform target config section 7541b49 9d928d9 change % change
psoc6 all-clusters cy8ckit_062s2_43012 .debug_info 27986252 27986254 2 0.0
light cy8ckit_062s2_43012 .debug_info 22872186 22872187 1 0.0
lock cy8ckit_062s2_43012 .debug_info 23123936 23123937 1 0.0
telink all-clusters-minimal-app tlsr9518adk80d text 651546 651548 2 0.0
pump-controller-app tlsr9518adk80d (read/write) 853072 853080 8 0.0
text 578120 578122 2 0.0
temperature-measurement-app tlsr9518adk80d text 573494 573496 2 0.0
Decreases (11 builds for bl702, cyw30739, efr32, esp32, qpg, telink)
platform target config section 7541b49 9d928d9 change % change
bl702 lighting-app bl702 .debug_info 40861787 40861786 -1 -0.0
.text 958528 958526 -2 -0.0
bl702+rpc .debug_info 45488646 45488645 -1 -0.0
.text 1033870 1033868 -2 -0.0
cyw30739 lock cyw930739m2evb_01 (read/write) 591898 591890 -8 -0.0
.app_xip_area 463372 463364 -8 -0.0
efr32 lighting-app BRD4187C (read/write) 1013900 1013892 -8 -0.0
.text 838432 838424 -8 -0.0
lock-app BRD4161A+wf200 (read/write) 1079988 1079980 -8 -0.0
.text 904288 904280 -8 -0.0
esp32 all-clusters-app m5stack (read/write) 502187 502183 -4 -0.0
.flash.rodata 251512 251508 -4 -0.0
qpg lock-app qpg6105+debug (read/write) 1119548 1119540 -8 -0.0
.text 566648 566640 -8 -0.0
telink light-switch-app tlsr9518adk80d (read/write) 876220 876212 -8 -0.0
text 596166 596162 -4 -0.0
lighting-app-rpc tlsr9518adk80d (read/write) 1022996 1022988 -8 -0.0
text 705622 705620 -2 -0.0
lock-app tlsr9518adk80d text 609722 609718 -4 -0.0
ota-requestor-app tlsr9518adk80d text 606672 606670 -2 -0.0
Full report (49 builds for bl602, bl702, cc13x2_26x2, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
platform target config section 7541b49 9d928d9 change % change
bl602 lighting-app bl602 (read/write) 1352022 1352022 0 0.0
.bss 94690 94690 0 0.0
.data 9744 9744 0 0.0
.text 1027976 1027976 0 0.0
bl602+rpc (read/write) 1397206 1397206 0 0.0
.bss 102738 102738 0 0.0
.data 10136 10136 0 0.0
.text 1058654 1058654 0 0.0
bl702 lighting-app bl702 (read only) 3358 3358 0 0.0
(read/write) 1191803 1191803 0 0.0
.bleromro 6342 6342 0 0.0
.bleromrw 124 124 0 0.0
.boot2 292 292 0 0.0
.bss 69793 69793 0 0.0
.bss_psram 30160 30160 0 0.0
.comment 48 48 0 0.0
.data 4072 4072 0 0.0
.debug_abbrev 1556856 1556856 0 0.0
.debug_aranges 134592 134592 0 0.0
.debug_frame 493440 493440 0 0.0
.debug_info 40861787 40861786 -1 -0.0
.debug_line 5294041 5294041 0 0.0
.debug_loc 3424603 3424603 0 0.0
.debug_ranges 373160 373160 0 0.0
.debug_str 3585238 3585238 0 0.0
.hbn 536 536 0 0.0
.hbn_noinit 260 260 0 0.0
.init 342 342 0 0.0
.init_array 144 144 0 0.0
.psram 0 0 0 0.0
.riscv.attributes 47 47 0 0.0
.rodata 108160 108160 0 0.0
.rsvd 2960 2960 0 0.0
.sha_ocram 72 72 0 0.0
.shstrtab 304 304 0 0.0
.stack 2048 2048 0 0.0
.strtab 576349 576349 0 0.0
.symtab 174160 174160 0 0.0
.tcm_data 36 36 0 0.0
.tcmcode 3358 3358 0 0.0
.text 0 0 0 0.0
958528 958526 -2 -0.0
bl702+rpc (read only) 3358 3358 0 0.0
(read/write) 1282471 1282471 0 0.0
.bleromro 6342 6342 0 0.0
.bleromrw 124 124 0 0.0
.boot2 292 292 0 0.0
.bss 77841 77841 0 0.0
.bss_psram 30432 30432 0 0.0
.comment 48 48 0 0.0
.data 4616 4616 0 0.0
.debug_abbrev 1708971 1708971 0 0.0
.debug_aranges 142920 142920 0 0.0
.debug_frame 521384 521384 0 0.0
.debug_info 45488646 45488645 -1 -0.0
.debug_line 5699384 5699384 0 0.0
.debug_loc 3622618 3622618 0 0.0
.debug_ranges 397160 397160 0 0.0
.debug_str 3989434 3989434 0 0.0
.hbn 536 536 0 0.0
.hbn_noinit 260 260 0 0.0
.init 342 342 0 0.0
.init_array 160 160 0 0.0
.psram 0 0 0 0.0
.riscv.attributes 47 47 0 0.0
.rodata 122368 122368 0 0.0
.rsvd 2960 2960 0 0.0
.sha_ocram 72 72 0 0.0
.shstrtab 304 304 0 0.0
.stack 2048 2048 0 0.0
.strtab 637674 637674 0 0.0
.symtab 192656 192656 0 0.0
.tcm_data 36 36 0 0.0
.tcmcode 3358 3358 0 0.0
.text 0 0 0 0.0
1033870 1033868 -2 -0.0
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read only) 681335 681335 0 0.0
(read/write) 169760 169760 0 0.0
.bss 80788 80788 0 0.0
.data 3352 3352 0 0.0
.rodata 88527 88527 0 0.0
.text 592492 592492 0 0.0
all-clusters-minimal-app LP_CC2652R7 (read only) 644607 644607 0 0.0
(read/write) 157456 157456 0 0.0
.bss 79988 79988 0 0.0
.data 3352 3352 0 0.0
.rodata 78399 78399 0 0.0
.text 565888 565888 0 0.0
lock-ftd LP_CC2652R7 (read only) 678407 678407 0 0.0
(read/write) 170152 170152 0 0.0
.bss 78340 78340 0 0.0
.data 3316 3316 0 0.0
.rodata 76711 76711 0 0.0
.text 601216 601216 0 0.0
lock-mtd LP_CC2652R7 (read only) 664395 664395 0 0.0
(read/write) 179324 179324 0 0.0
.bss 73500 73500 0 0.0
.data 3316 3316 0 0.0
.rodata 103523 103523 0 0.0
.text 560392 560392 0 0.0
pump-app LP_CC2652R7 (read only) 691635 691635 0 0.0
(read/write) 157668 157668 0 0.0
.bss 78316 78316 0 0.0
.data 3280 3280 0 0.0
.rodata 91083 91083 0 0.0
.text 600072 600072 0 0.0
pump-controller-app LP_CC2652R7 (read only) 676595 676595 0 0.0
(read/write) 172844 172844 0 0.0
.bss 78452 78452 0 0.0
.data 3304 3304 0 0.0
.rodata 86971 86971 0 0.0
.text 589144 589144 0 0.0
shell LP_CC2652R7 (read only) 672766 672766 0 0.0
(read/write) 180400 180400 0 0.0
.bss 82860 82860 0 0.0
.data 3348 3348 0 0.0
.rodata 85430 85430 0 0.0
.text 587024 587024 0 0.0
cc32xx lock CC3235SF_LAUNCHXL (read only) 644465 644465 0 0.0
(read/write) 203688 203688 0 0.0
.ARM.attributes 44 44 0 0.0
.ARM.exidx 8 8 0 0.0
.bss 197088 197088 0 0.0
.comment 194 194 0 0.0
.data 1480 1480 0 0.0
.debug_abbrev 930280 930280 0 0.0
.debug_aranges 87344 87344 0 0.0
.debug_frame 300048 300048 0 0.0
.debug_info 20268460 20268460 0 0.0
.debug_line 2659917 2659917 0 0.0
.debug_loc 2802993 2802993 0 0.0
.debug_ranges 282936 282936 0 0.0
.debug_str 3023875 3023875 0 0.0
.ramVecs 780 780 0 0.0
.resetVecs 64 64 0 0.0
.rodata 105929 105929 0 0.0
.shstrtab 232 232 0 0.0
.stab 204 204 0 0.0
.stabstr 441 441 0 0.0
.stack 2048 2048 0 0.0
.strtab 378571 378571 0 0.0
.symtab 256624 256624 0 0.0
.text 0 0 0 0.0
536412 536412 0 0.0
cyw30739 light cyw930739m2evb_01 (read/write) 587762 587762 0 0.0
.app_xip_area 464532 464532 0 0.0
.bss 65672 65672 0 0.0
.data 740 740 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
lock cyw930739m2evb_01 (read/write) 591898 591890 -8 -0.0
.app_xip_area 463372 463364 -8 -0.0
.bss 70960 70960 0 0.0
.data 748 748 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 552286 552286 0 0.0
.app_xip_area 434648 434648 0 0.0
.bss 60120 60120 0 0.0
.data 696 696 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A+rs9116 (read/write) 1030996 1030996 0 0.0
.bss 179276 179276 0 0.0
.data 2060 2060 0 0.0
.text 849640 849640 0 0.0
BRD4187C (read/write) 1013900 1013892 -8 -0.0
.bss 148176 148176 0 0.0
.data 2696 2696 0 0.0
.text 838432 838424 -8 -0.0
lock-app BRD4161A+wf200 (read/write) 1079988 1079980 -8 -0.0
.bss 173612 173612 0 0.0
.data 2068 2068 0 0.0
.text 904288 904280 -8 -0.0
window-app BRD4187C (read/write) 1130572 1130572 0 0.0
.bss 132600 132600 0 0.0
.data 2588 2588 0 0.0
.text 970788 970788 0 0.0
esp32 all-clusters-app c3devkit (read only) 1051632 1051632 0 0.0
(read/write) 1586122 1586122 0 0.0
.dram0.bss 77824 77824 0 0.0
.dram0.data 13752 13752 0 0.0
.flash.rodata 222784 222784 0 0.0
.flash.text 1051632 1051632 0 0.0
.iram0.text 72896 72896 0 0.0
m5stack (read only) 1103231 1103231 0 0.0
(read/write) 502187 502183 -4 -0.0
.dram0.bss 82864 82864 0 0.0
.dram0.data 34040 34040 0 0.0
.flash.rodata 251512 251508 -4 -0.0
.flash.text 1097847 1097847 0 0.0
.iram0.text 124855 124855 0 0.0
k32w contact k32w0+release (read/write) 673720 673720 0 0.0
.bss 65372 65372 0 0.0
.data 2224 2224 0 0.0
.text 574724 574724 0 0.0
light k32w0+release (read/write) 672764 672764 0 0.0
.bss 65036 65036 0 0.0
.data 2212 2212 0 0.0
.text 574116 574116 0 0.0
lock k32w0+release (read/write) 627180 627180 0 0.0
.bss 63108 63108 0 0.0
.data 2136 2136 0 0.0
.text 546920 546920 0 0.0
linux chip-tool-ipv6only arm64 (read only) 12164596 12164596 0 0.0
(read/write) 733448 733448 0 0.0
.bss 34296 34296 0 0.0
.data 3008 3008 0 0.0
.data.rel.ro 675552 675552 0 0.0
.dynamic 560 560 0 0.0
.got 15384 15384 0 0.0
.init 24 24 0 0.0
.init_array 216 216 0 0.0
.rodata 588580 588580 0 0.0
.text 9835380 9835380 0 0.0
thermostat-no-ble arm64 (read only) 2522276 2522276 0 0.0
(read/write) 145256 145256 0 0.0
.bss 56344 56344 0 0.0
.data 1784 1784 0 0.0
.data.rel.ro 77712 77712 0 0.0
.dynamic 560 560 0 0.0
.got 5368 5368 0 0.0
.init 24 24 0 0.0
.init_array 432 432 0 0.0
.rodata 151024 151024 0 0.0
.text 2108816 2108816 0 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2467664 2467664 0 0.0
.bss 215804 215804 0 0.0
.data 5880 5880 0 0.0
.text 1430308 1430308 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read only) 4 4 0 0.0
(read/write) 1172936 1172936 0 0.0
bss 155397 155397 0 0.0
rodata 132600 132600 0 0.0
text 804140 804140 0 0.0
nrf7002dk_nrf5340_cpuapp (read only) 4 4 0 0.0
(read/write) 1432732 1432732 0 0.0
bss 135137 135137 0 0.0
rodata 228652 228652 0 0.0
text 774900 774900 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 (read only) 4 4 0 0.0
(read/write) 1118136 1118136 0 0.0
bss 154553 154553 0 0.0
rodata 109416 109416 0 0.0
text 773492 773492 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 (read only) 841000 841000 0 0.0
(read/write) 1762956 1762956 0 0.0
.ARM.attributes 46 46 0 0.0
.ARM.exidx 8 8 0 0.0
.bss 189704 189704 0 0.0
.comment 200 200 0 0.0
.copy.table 24 24 0 0.0
.cy_m0p_image 6216 6216 0 0.0
.cy_sharedmem 8 8 0 0.0
.data 2640 2640 0 0.0
.debug_abbrev 1253687 1253687 0 0.0
.debug_aranges 111480 111480 0 0.0
.debug_frame 374764 374764 0 0.0
.debug_info 27986252 27986254 2 0.0
.debug_line 3795014 3795014 0 0.0
.debug_loc 3699982 3699982 0 0.0
.debug_ranges 364784 364784 0 0.0
.debug_str 3533520 3533520 0 0.0
.heap 841000 841000 0 0.0
.noinit 148 148 0 0.0
.ramVectors 736 736 0 0.0
.shstrtab 288 288 0 0.0
.stab 156 156 0 0.0
.stabstr 335 335 0 0.0
.stack_dummy 4096 4096 0 0.0
.strtab 581819 581819 0 0.0
.symtab 426112 426112 0 0.0
.text 1562224 1562224 0 0.0
.zero.table 8 8 0 0.0
text 0 0 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 (read only) 841808 841808 0 0.0
(read/write) 1703252 1703252 0 0.0
.ARM.attributes 46 46 0 0.0
.ARM.exidx 8 8 0 0.0
.bss 188904 188904 0 0.0
.comment 200 200 0 0.0
.copy.table 24 24 0 0.0
.cy_m0p_image 6216 6216 0 0.0
.cy_sharedmem 8 8 0 0.0
.data 2632 2632 0 0.0
.debug_abbrev 1239020 1239020 0 0.0
.debug_aranges 110712 110712 0 0.0
.debug_frame 377100 377100 0 0.0
.debug_info 27530718 27530718 0 0.0
.debug_line 3802513 3802513 0 0.0
.debug_loc 3683183 3683183 0 0.0
.debug_ranges 362720 362720 0 0.0
.debug_str 3520642 3520642 0 0.0
.heap 841808 841808 0 0.0
.noinit 148 148 0 0.0
.ramVectors 736 736 0 0.0
.shstrtab 288 288 0 0.0
.stab 156 156 0 0.0
.stabstr 335 335 0 0.0
.stack_dummy 4096 4096 0 0.0
.strtab 541411 541411 0 0.0
.symtab 410960 410960 0 0.0
.text 1503328 1503328 0 0.0
.zero.table 0 0 0 0.0
8 8 0 0.0
light cy8ckit_062s2_43012 (read only) 850072 850072 0 0.0
(read/write) 1616892 1616892 0 0.0
.ARM.attributes 46 46 0 0.0
.ARM.exidx 8 8 0 0.0
.bss 180808 180808 0 0.0
.comment 200 200 0 0.0
.copy.table 24 24 0 0.0
.cy_m0p_image 6216 6216 0 0.0
.cy_sharedmem 8 8 0 0.0
.data 2464 2464 0 0.0
.debug_abbrev 1074357 1074357 0 0.0
.debug_aranges 102920 102920 0 0.0
.debug_frame 347384 347384 0 0.0
.debug_info 22872186 22872187 1 0.0
.debug_line 3355241 3355241 0 0.0
.debug_loc 3358372 3358372 0 0.0
.debug_ranges 321280 321280 0 0.0
.debug_str 3319413 3319413 0 0.0
.heap 850072 850072 0 0.0
.noinit 148 148 0 0.0
.ramVectors 736 736 0 0.0
.shstrtab 288 288 0 0.0
.stab 156 156 0 0.0
.stabstr 335 335 0 0.0
.stack_dummy 4096 4096 0 0.0
.strtab 477245 477245 0 0.0
.symtab 379008 379008 0 0.0
.text 1425232 1425232 0 0.0
.zero.table 0 0 0 0.0
8 8 0 0.0
lock cy8ckit_062s2_43012 (read only) 822296 822296 0 0.0
(read/write) 1673636 1673636 0 0.0
.ARM.attributes 46 46 0 0.0
.ARM.exidx 8 8 0 0.0
.bss 208568 208568 0 0.0
.comment 200 200 0 0.0
.copy.table 24 24 0 0.0
.cy_m0p_image 6216 6216 0 0.0
.cy_sharedmem 8 8 0 0.0
.data 2480 2480 0 0.0
.debug_abbrev 1076321 1076321 0 0.0
.debug_aranges 103312 103312 0 0.0
.debug_frame 349092 349092 0 0.0
.debug_info 23123936 23123937 1 0.0
.debug_line 3357285 3357285 0 0.0
.debug_loc 3381786 3381786 0 0.0
.debug_ranges 323848 323848 0 0.0
.debug_str 3337966 3337966 0 0.0
.heap 822296 822296 0 0.0
.noinit 148 148 0 0.0
.ramVectors 736 736 0 0.0
.shstrtab 288 288 0 0.0
.stab 156 156 0 0.0
.stabstr 335 335 0 0.0
.stack_dummy 4096 4096 0 0.0
.strtab 480942 480942 0 0.0
.symtab 381152 381152 0 0.0
.text 1454200 1454200 0 0.0
.zero.table 0 0 0 0.0
8 8 0 0.0
qpg lighting-app qpg6105+debug (read/write) 1153052 1153052 0 0.0
.bss 99916 99916 0 0.0
.data 852 852 0 0.0
.text 600148 600148 0 0.0
lock-app qpg6105+debug (read/write) 1119548 1119540 -8 -0.0
.bss 96308 96308 0 0.0
.data 864 864 0 0.0
.text 566648 566640 -8 -0.0
telink all-clusters-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 1021008 1021008 0 0.0
bss 97908 97908 0 0.0
text 690394 690394 0 0.0
all-clusters-minimal-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 956052 956052 0 0.0
bss 96960 96960 0 0.0
text 651546 651548 2 0.0
contact-sensor-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 860284 860284 0 0.0
bss 89124 89124 0 0.0
text 581950 581950 0 0.0
light-switch-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 876220 876212 -8 -0.0
bss 89212 89212 0 0.0
text 596166 596162 -4 -0.0
lighting-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 954824 954824 0 0.0
bss 97340 97340 0 0.0
text 662522 662522 0 0.0
lighting-app-rpc tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 1022996 1022988 -8 -0.0
bss 98904 98904 0 0.0
text 705622 705620 -2 -0.0
lock-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 914312 914312 0 0.0
bss 89776 89776 0 0.0
text 609722 609718 -4 -0.0
ota-requestor-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 889076 889076 0 0.0
bss 90136 90136 0 0.0
text 606672 606670 -2 -0.0
pump-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 865820 865820 0 0.0
bss 89240 89240 0 0.0
text 587796 587796 0 0.0
pump-controller-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 853072 853080 8 0.0
bss 89040 89040 0 0.0
text 578120 578122 2 0.0
temperature-measurement-app tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 848788 848788 0 0.0
bss 88900 88900 0 0.0
text 573494 573496 2 0.0
thermostat tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 880364 880364 0 0.0
bss 90604 90604 0 0.0
text 597694 597694 0 0.0
window-covering tlsr9518adk80d (read only) 4 4 0 0.0
(read/write) 942644 942644 0 0.0
bss 99012 99012 0 0.0
text 648750 648750 0 0.0

@bzbarsky-apple bzbarsky-apple merged commit f1f69c0 into project-chip:master Mar 10, 2023
@bzbarsky-apple bzbarsky-apple deleted the fix-test-definition-script branch March 10, 2023 15:09
lecndav pushed a commit to lecndav/connectedhomeip that referenced this pull request Mar 22, 2023
project-chip#25600)

When running only a subset of tests (e.g. only DL_* tests), not all apps are
needed for the tests to run.

This used to work because we used a fake "NOT_FOUND_IN_OUTPUT_" path for the
missing apps, which was a string so the string manipulation on it worked, and
the output was never used.

But after project-chip#25368 we end up
with None where we expect a string, and the script dies.

The fix is just to skip string manipulation on the path of apps that have None
as a path.
mwswartwout pushed a commit to mwswartwout/connectedhomeip that referenced this pull request Mar 27, 2023
project-chip#25600)

When running only a subset of tests (e.g. only DL_* tests), not all apps are
needed for the tests to run.

This used to work because we used a fake "NOT_FOUND_IN_OUTPUT_" path for the
missing apps, which was a string so the string manipulation on it worked, and
the output was never used.

But after project-chip#25368 we end up
with None where we expect a string, and the script dies.

The fix is just to skip string manipulation on the path of apps that have None
as a path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants