Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change persistent storage SyncGetKeyValue API to match constraints of KvsStorageManager #20164

Conversation

tehampson
Copy link
Contributor

@tehampson tehampson commented Jun 30, 2022

Problem

Change overview

  • API no longer claims that size input/output argument is set to the total size of the value when CHIP_ERROR_BUFFER_TOO_SMALL is returned.
    • Rationale 1: PersistentStorageDelegate is 90%+ implemented by KeyValueStoreManager, which is itself implemented per-platform. KeyValueStoreManager doesn’t have the ability to determine value size for a given key
    • Rationale 2: PersistentStorageDelegate usage in SDK common code never makes use of that feature
    • Overall: cost of change to make all platform meet an API expectation on which no code currently relies in SDK is too high.
  • When CHIP_ERROR_BUFFER_TOO_SMALL is returned by SyncGetKeyValue, we expect that the buffer will be populated up to the size provided.
    • Rationale: A lot of persistent data storage in SDK employs larger-than-needed buffers to account for future growth, since data is often read by TLVReader and can be backwards compatible in case of only added fields. Without this behavior of SyncGetKeyValue (which matches 1:1 to KeyValueStoreManager), we cannot actually “read however much we support” and try to deal with it. It would always error-out, making it harder to actually implement and test “upgrade/downgrade-safe” code.

Testing

  • Unit test passes locally
  • CI passes
  • Run following command confirmed audit logs indicate audit was a success
    • scripts/examples/gn_build_example.sh examples/all-clusters-app/linux out/debug/standalone chip_config_network_layer_ble=false chip_support_enable_storage_api_audit=true
    • ./out/debug/standalone/chip-all-clusters-app

…e size

Some implmentation of PersistentStorageDelegate use KeyValueStoreManager
which is not capable of giving the size of for the time being, which
makes this obligation impossible to fulfill right now.

After performing a code audit no one is currently using the
functionality to get the size, so this change is safe, but we need
platforms to adhear to the this updated description.

We also are now enforcing the PersistentStorageDelegate::SyncGetKeyValue
API such that we fill up the buffer up to the size provided even if the
buffer is not large enough. This is done so that in the case a version
downgrade needs to happen that (due to OTA revert, security reason, or
any other reason) we can make sure as long as TLV data was stored you
could still ensure that you are able to work in critical situations.
@github-actions
Copy link

PR #20164: Size comparison from 31ddf98 to c182809

Increases (10 builds for cyw30739, k32w, linux, mbed, telink)
platform target config section 31ddf98 c182809 change % change
cyw30739 light cyw930739m2evb_01 (read/write) 578662 578702 40 0.0
.app_xip_area 457440 457480 40 0.0
lock cyw930739m2evb_01 (read/write) 580238 580270 32 0.0
.app_xip_area 458824 458856 32 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 581734 581774 40 0.0
.app_xip_area 461360 461400 40 0.0
k32w light k32w061+release (read/write) 658040 658080 40 0.0
.text 580732 580772 40 0.0
lock k32w061+release (read/write) 684524 684556 32 0.0
.text 606740 606772 32 0.0
linux chip-tool-no-interactive-ipv6only arm64 (read only) 9893332 9893844 512 0.0
(read/write) 671793 671809 16 0.0
.data.rel.ro 610816 610832 16 0.0
.rodata 472548 472644 96 0.0
.text 7890212 7890580 368 0.0
thermostat-no-ble arm64 (read only) 2591604 2591924 320 0.0
(read/write) 158273 158289 16 0.0
.data.rel.ro 83232 83240 8 0.0
.rodata 165236 165348 112 0.1
.text 2186624 2186800 176 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read/write) 2447080 2447152 72 0.0
.text 1409724 1409796 72 0.0
telink light-switch-app tlsr9518adk80d (read/write) 796292 796336 44 0.0
text 564816 564856 40 0.0
lighting-app tlsr9518adk80d (read/write) 816160 816204 44 0.0
text 581172 581212 40 0.0
Full report (10 builds for cyw30739, k32w, linux, mbed, telink)
platform target config section 31ddf98 c182809 change % change
cyw30739 light cyw930739m2evb_01 (read/write) 578662 578702 40 0.0
.app_xip_area 457440 457480 40 0.0
.bss 64184 64184 0 0.0
.data 716 716 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
lock cyw930739m2evb_01 (read/write) 580238 580270 32 0.0
.app_xip_area 458824 458856 32 0.0
.bss 64376 64376 0 0.0
.data 720 720 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 581734 581774 40 0.0
.app_xip_area 461360 461400 40 0.0
.bss 63392 63392 0 0.0
.data 660 660 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
k32w light k32w061+release (read/write) 658040 658080 40 0.0
.bss 69516 69516 0 0.0
.data 1992 1992 0 0.0
.text 580732 580772 40 0.0
lock k32w061+release (read/write) 684524 684556 32 0.0
.bss 69980 69980 0 0.0
.data 2004 2004 0 0.0
.text 606740 606772 32 0.0
linux chip-tool-no-interactive-ipv6only arm64 (read only) 9893332 9893844 512 0.0
(read/write) 671793 671809 16 0.0
.bss 42609 42609 0 0.0
.data 1152 1152 0 0.0
.data.rel.ro 610816 610832 16 0.0
.dynamic 528 528 0 0.0
.got 13408 13408 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 472548 472644 96 0.0
.text 7890212 7890580 368 0.0
thermostat-no-ble arm64 (read only) 2591604 2591924 320 0.0
(read/write) 158273 158289 16 0.0
.bss 65249 65249 0 0.0
.data 1704 1704 0 0.0
.data.rel.ro 83232 83240 8 0.0
.dynamic 528 528 0 0.0
.got 5072 5072 0 0.0
.init 24 24 0 0.0
.init_array 400 400 0 0.0
.rodata 165236 165348 112 0.1
.text 2186624 2186800 176 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2447080 2447152 72 0.0
.bss 213940 213940 0 0.0
.data 5872 5872 0 0.0
.text 1409724 1409796 72 0.0
telink light-switch-app tlsr9518adk80d (read/write) 796292 796336 44 0.0
bss 70560 70560 0 0.0
noinit 40416 40416 0 0.0
text 564816 564856 40 0.0
lighting-app tlsr9518adk80d (read/write) 816160 816204 44 0.0
bss 71404 71404 0 0.0
noinit 40416 40416 0 0.0
text 581172 581212 40 0.0

@github-actions
Copy link

github-actions bot commented Jun 30, 2022

PR #20164: Size comparison from 31ddf98 to 9475174

Increases above 0.2%:

platform target config section 31ddf98 9475174 change % change
efr32 lock-app BRD4161A+wf200 (read/write) 1119568 1124032 4464 0.4
.bss 139144 142128 2984 2.1
Increases (30 builds for cc13x2_26x2, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, p6, telink)
platform target config section 31ddf98 9475174 change % change
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read only) 665171 665203 32 0.0
.text 576764 576796 32 0.0
all-clusters-minimal-app LP_CC2652R7 (read only) 653947 653971 24 0.0
.text 562060 562084 24 0.0
lock-ftd LP_CC2652R7 (read only) 667759 667791 32 0.0
.text 591088 591120 32 0.0
lock-mtd LP_CC2652R7 (read only) 617175 617207 32 0.0
.text 540616 540648 32 0.0
pump-app LP_CC2652R7 (read only) 677207 677239 32 0.0
.rodata 88423 88431 8 0.0
.text 588300 588324 24 0.0
pump-controller-app LP_CC2652R7 (read only) 663039 663087 48 0.0
.rodata 84271 84279 8 0.0
.text 578288 578328 40 0.0
shell LP_CC2652R7 (read only) 657622 657670 48 0.0
.rodata 84838 84846 8 0.0
.text 572468 572508 40 0.0
cyw30739 light cyw930739m2evb_01 (read/write) 578662 578710 48 0.0
.app_xip_area 457440 457488 48 0.0
lock cyw930739m2evb_01 (read/write) 580238 580270 32 0.0
.app_xip_area 458824 458856 32 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 581734 581774 40 0.0
.app_xip_area 461360 461400 40 0.0
efr32 lighting-app BRD4161A (read/write) 1080092 1080132 40 0.0
.text 945028 945068 40 0.0
BRD4161A+rpc (read/write) 1134388 1134444 56 0.0
.text 982432 982488 56 0.0
BRD4161A+rs911x (read/write) 942556 942596 40 0.0
.text 801776 801816 40 0.0
lock-app BRD4161A+wf200 (read/write) 1119568 1124032 4464 0.4
.bss 139144 142128 2984 2.1
.text 978344 979824 1480 0.2
window-app BRD4161A (read/write) 1065324 1065380 56 0.0
.text 930148 930204 56 0.0
esp32 all-clusters-app c3devkit (read only) 1018488 1018528 40 0.0
(read/write) 1484882 1484890 8 0.0
.flash.rodata 214784 214792 8 0.0
.flash.text 1018488 1018528 40 0.0
m5stack (read only) 1072583 1072627 44 0.0
.flash.text 1067199 1067243 44 0.0
k32w light k32w061+release (read/write) 658040 658080 40 0.0
.text 580732 580772 40 0.0
lock k32w061+release (read/write) 684524 684556 32 0.0
.text 606740 606772 32 0.0
linux chip-tool-no-interactive-ipv6only arm64 (read only) 9893332 9893844 512 0.0
(read/write) 671793 671809 16 0.0
.data.rel.ro 610816 610832 16 0.0
.rodata 472548 472644 96 0.0
.text 7890212 7890580 368 0.0
thermostat-no-ble arm64 (read only) 2591604 2591924 320 0.0
(read/write) 158273 158289 16 0.0
.data.rel.ro 83232 83240 8 0.0
.rodata 165236 165348 112 0.1
.text 2186624 2186800 176 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read/write) 2447080 2447152 72 0.0
.text 1409724 1409796 72 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1172915 1172967 52 0.0
rodata 141348 141352 4 0.0
text 809776 809816 40 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 (read/write) 1153783 1153819 36 0.0
rodata 133280 133284 4 0.0
text 799488 799520 32 0.0
p6 all-clusters-app default (read/write) 2562688 2562728 40 0.0
.text 1520952 1520992 40 0.0
all-clusters-minimal-app default (read/write) 2508552 2508592 40 0.0
.text 1466816 1466856 40 0.0
light-app default (read/write) 2439048 2439104 56 0.0
.text 1397312 1397368 56 0.0
lock-app default (read/write) 2465328 2465384 56 0.0
.text 1423592 1423648 56 0.0
telink light-switch-app tlsr9518adk80d (read/write) 796292 796336 44 0.0
text 564816 564856 40 0.0
lighting-app tlsr9518adk80d (read/write) 816160 816204 44 0.0
text 581172 581210 38 0.0
Decreases (6 builds for cc13x2_26x2)
platform target config section 31ddf98 9475174 change % change
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read/write) 186052 186020 -32 -0.0
all-clusters-minimal-app LP_CC2652R7 (read/write) 196572 196548 -24 -0.0
lock-ftd LP_CC2652R7 (read/write) 173608 173576 -32 -0.0
pump-app LP_CC2652R7 (read/write) 165008 164976 -32 -0.0
pump-controller-app LP_CC2652R7 (read/write) 179296 179248 -48 -0.0
shell LP_CC2652R7 (read/write) 189104 189056 -48 -0.0
Full report (30 builds for cc13x2_26x2, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, p6, telink)
platform target config section 31ddf98 9475174 change % change
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read only) 665171 665203 32 0.0
(read/write) 186052 186020 -32 -0.0
.bss 74116 74116 0 0.0
.data 3356 3356 0 0.0
.rodata 88091 88091 0 0.0
.text 576764 576796 32 0.0
all-clusters-minimal-app LP_CC2652R7 (read only) 653947 653971 24 0.0
(read/write) 196572 196548 -24 -0.0
.bss 73412 73412 0 0.0
.data 3356 3356 0 0.0
.rodata 91571 91571 0 0.0
.text 562060 562084 24 0.0
lock-ftd LP_CC2652R7 (read only) 667759 667791 32 0.0
(read/write) 173608 173576 -32 -0.0
.bss 71148 71148 0 0.0
.data 3280 3280 0 0.0
.rodata 76191 76191 0 0.0
.text 591088 591120 32 0.0
lock-mtd LP_CC2652R7 (read only) 617175 617207 32 0.0
(read/write) 144264 144264 0 0.0
.bss 66868 66868 0 0.0
.data 3280 3280 0 0.0
.rodata 76071 76071 0 0.0
.text 540616 540648 32 0.0
pump-app LP_CC2652R7 (read only) 677207 677239 32 0.0
(read/write) 165008 164976 -32 -0.0
.bss 71228 71228 0 0.0
.data 3280 3280 0 0.0
.rodata 88423 88431 8 0.0
.text 588300 588324 24 0.0
pump-controller-app LP_CC2652R7 (read only) 663039 663087 48 0.0
(read/write) 179296 179248 -48 -0.0
.bss 71348 71348 0 0.0
.data 3276 3276 0 0.0
.rodata 84271 84279 8 0.0
.text 578288 578328 40 0.0
shell LP_CC2652R7 (read only) 657622 657670 48 0.0
(read/write) 189104 189056 -48 -0.0
.bss 76420 76420 0 0.0
.data 3360 3360 0 0.0
.rodata 84838 84846 8 0.0
.text 572468 572508 40 0.0
cyw30739 light cyw930739m2evb_01 (read/write) 578662 578710 48 0.0
.app_xip_area 457440 457488 48 0.0
.bss 64184 64184 0 0.0
.data 716 716 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
lock cyw930739m2evb_01 (read/write) 580238 580270 32 0.0
.app_xip_area 458824 458856 32 0.0
.bss 64376 64376 0 0.0
.data 720 720 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 581734 581774 40 0.0
.app_xip_area 461360 461400 40 0.0
.bss 63392 63392 0 0.0
.data 660 660 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A (read/write) 1080092 1080132 40 0.0
.bss 132996 132996 0 0.0
.data 2048 2048 0 0.0
.text 945028 945068 40 0.0
BRD4161A+rpc (read/write) 1134388 1134444 56 0.0
.bss 149676 149676 0 0.0
.data 2260 2260 0 0.0
.text 982432 982488 56 0.0
BRD4161A+rs911x (read/write) 942556 942596 40 0.0
.bss 138712 138712 0 0.0
.data 2048 2048 0 0.0
.text 801776 801816 40 0.0
lock-app BRD4161A+wf200 (read/write) 1119568 1124032 4464 0.4
.bss 139144 142128 2984 2.1
.data 2060 2060 0 0.0
.text 978344 979824 1480 0.2
window-app BRD4161A (read/write) 1065324 1065380 56 0.0
.bss 133076 133076 0 0.0
.data 2076 2076 0 0.0
.text 930148 930204 56 0.0
esp32 all-clusters-app c3devkit (read only) 1018488 1018528 40 0.0
(read/write) 1484882 1484890 8 0.0
.dram0.bss 70080 70080 0 0.0
.dram0.data 14592 14592 0 0.0
.flash.rodata 214784 214792 8 0.0
.flash.text 1018488 1018528 40 0.0
.iram0.text 62902 62902 0 0.0
m5stack (read only) 1072583 1072627 44 0.0
(read/write) 486984 486984 0 0.0
.dram0.bss 75600 75600 0 0.0
.dram0.data 34144 34144 0 0.0
.flash.rodata 245244 245244 0 0.0
.flash.text 1067199 1067243 44 0.0
.iram0.text 123267 123267 0 0.0
k32w light k32w061+release (read/write) 658040 658080 40 0.0
.bss 69516 69516 0 0.0
.data 1992 1992 0 0.0
.text 580732 580772 40 0.0
lock k32w061+release (read/write) 684524 684556 32 0.0
.bss 69980 69980 0 0.0
.data 2004 2004 0 0.0
.text 606740 606772 32 0.0
linux chip-tool-no-interactive-ipv6only arm64 (read only) 9893332 9893844 512 0.0
(read/write) 671793 671809 16 0.0
.bss 42609 42609 0 0.0
.data 1152 1152 0 0.0
.data.rel.ro 610816 610832 16 0.0
.dynamic 528 528 0 0.0
.got 13408 13408 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 472548 472644 96 0.0
.text 7890212 7890580 368 0.0
thermostat-no-ble arm64 (read only) 2591604 2591924 320 0.0
(read/write) 158273 158289 16 0.0
.bss 65249 65249 0 0.0
.data 1704 1704 0 0.0
.data.rel.ro 83232 83240 8 0.0
.dynamic 528 528 0 0.0
.got 5072 5072 0 0.0
.init 24 24 0 0.0
.init_array 400 400 0 0.0
.rodata 165236 165348 112 0.1
.text 2186624 2186800 176 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2447080 2447152 72 0.0
.bss 213940 213940 0 0.0
.data 5872 5872 0 0.0
.text 1409724 1409796 72 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1172915 1172967 52 0.0
bss 142884 142884 0 0.0
rodata 141348 141352 4 0.0
text 809776 809816 40 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 (read/write) 1153783 1153819 36 0.0
bss 142120 142120 0 0.0
rodata 133280 133284 4 0.0
text 799488 799520 32 0.0
p6 all-clusters-app default (read/write) 2562688 2562728 40 0.0
.bss 149120 149120 0 0.0
.data 2776 2776 0 0.0
.text 1520952 1520992 40 0.0
all-clusters-minimal-app default (read/write) 2508552 2508592 40 0.0
.bss 148400 148400 0 0.0
.data 2776 2776 0 0.0
.text 1466816 1466856 40 0.0
light-app default (read/write) 2439048 2439104 56 0.0
.bss 140456 140456 0 0.0
.data 2592 2592 0 0.0
.text 1397312 1397368 56 0.0
lock-app default (read/write) 2465328 2465384 56 0.0
.bss 140304 140304 0 0.0
.data 2600 2600 0 0.0
.text 1423592 1423648 56 0.0
telink light-switch-app tlsr9518adk80d (read/write) 796292 796336 44 0.0
bss 70560 70560 0 0.0
noinit 40416 40416 0 0.0
text 564816 564856 40 0.0
lighting-app tlsr9518adk80d (read/write) 816160 816204 44 0.0
bss 71404 71404 0 0.0
noinit 40416 40416 0 0.0
text 581172 581210 38 0.0

examples/chip-tool/config/PersistentStorage.cpp Outdated Show resolved Hide resolved
src/controller/python/chip/storage/__init__.py Outdated Show resolved Hide resolved
src/controller/python/chip/storage/__init__.py Outdated Show resolved Hide resolved
src/lib/core/CHIPPersistentStorageDelegate.h Outdated Show resolved Hide resolved
src/lib/core/CHIPPersistentStorageDelegate.h Outdated Show resolved Hide resolved
src/lib/core/CHIPPersistentStorageDelegate.h Outdated Show resolved Hide resolved
src/lib/support/PersistentStorageAudit.cpp Show resolved Hide resolved
src/lib/support/TestPersistentStorageDelegate.h Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Inconsistency between PersistentStorageDelegate::SyncGetKeyValue and KeyValueStoreManager
6 participants