Replace `KVStorePersister` with `KVStore` #2472

tnull · 2023-08-04T08:38:53Z

So far, the main persistence interface implemented by the user was the KVStorePersister which provided a number of blanket implementations for convenience. However, as we'll need additional functionalities soon (namely, the capability of removing previously persisted data items), this interface was rather limited.

Here, we upstream the KVStore interface with slight modifications from LDK Node and add the FilesystemStore implementation as part of ~~a new lightning-storage crate that supersedes~~ lightning-persister.

lightning-storage/Cargo.toml

codecov-commenter · 2023-08-04T11:09:38Z

Codecov Report

Patch coverage: 76.19% and project coverage change: +1.12% 🎉

Comparison is base (073f078) 90.59% compared to head (af2bd74) 91.72%.
Report is 24 commits behind head on main.

❗ Current head af2bd74 differs from pull request most recent head 7a65671. Consider uploading reports for the commit 7a65671 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2472      +/-   ##
==========================================
+ Coverage   90.59%   91.72%   +1.12%     
==========================================
  Files         110      112       +2     
  Lines       57509    66839    +9330     
  Branches    57509    66839    +9330     
==========================================
+ Hits        52101    61307    +9206     
- Misses       5408     5532     +124

Files Changed	Coverage Δ
lightning/src/util/test_utils.rs	`68.70% <0.00%> (-4.91%)`	⬇️
lightning-persister/src/utils.rs	`71.42% <71.42%> (ø)`
lightning/src/util/persist.rs	`84.78% <80.55%> (-15.22%)`	⬇️
lightning-background-processor/src/lib.rs	`81.41% <82.05%> (-0.91%)`	⬇️
lightning-persister/src/fs_store.rs	`87.71% <87.71%> (ø)`
lightning-persister/src/lib.rs	`100.00% <100.00%> (+10.83%)`	⬆️
lightning-persister/src/test_utils.rs	`100.00% <100.00%> (ø)`

... and 13 files with indirect coverage changes

☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.

tnull · 2023-08-04T11:59:49Z

Ran a few bench iterations with n = 1000 sample size each to check there is no regression in write performance. Unsurprisingly (as FilesystemStore is in good part the same code as FilesystemPersister), I saw no significant change or at times even a slight improvement.

For example some results (Linux on Intel(R) Xeon(R) CPU E3-1226 v3 @ 3.30GHz):

Benchmarking bench_filesystem_persisted_sends: Collecting 1000 samples in estimated 235.35 s (1000 iterations)
Benchmarking bench_filesystem_persisted_sends: Analyzing
bench_filesystem_persisted_sends
                        time:   [210.07 ms 217.72 ms 225.39 ms]
                        change: [-16.944% -12.610% -8.2165%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking bench_filesystem_persisted_sends: Collecting 1000 samples in estimated 114.87 s (1000 iterations)
Benchmarking bench_filesystem_persisted_sends: Analyzing
bench_filesystem_persisted_sends
                        time:   [234.67 ms 243.50 ms 252.42 ms]
                        change: [-2.5607% +2.5055% +7.8026%] (p = 0.34 > 0.05)
                        No change in performance detected.

tnull · 2023-08-09T08:07:44Z

(Not exactly sure why the BP test is stuck on windows but no other platform, will need to setup a windows box to debug this further)

Cargo.toml

lightning-storage/src/fs_store.rs

tnull · 2023-08-14T08:03:53Z

Rebased on main.

tnull · 2023-08-14T14:57:54Z

(Not exactly sure why the BP test is stuck on windows but no other platform, will need to setup a windows box to debug this further)

Windows issues are fixed, CI failure is #2438.

tnull · 2023-09-07T05:32:16Z

feel free to squash

Squashed fixups and included the following changes (as I decided to move the key length to a pub const to be in line with the alphabet):

> git diff-tree -U2 09b9fa8c 9e89a53b
diff --git a/lightning-persister/src/utils.rs b/lightning-persister/src/utils.rs
index bf042ff9..ae83aa47 100644
--- a/lightning-persister/src/utils.rs
+++ b/lightning-persister/src/utils.rs
@@ -1,6 +1,5 @@
-use lightning::util::persist::KVSTORE_NAMESPACE_KEY_ALPHABET;
+use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_MAX_KEY_LEN};
 use lightning::util::string::PrintableString;

-const KVSTORE_MAX_KEY_LEN: usize = 120;

 pub(crate) fn is_valid_kvstore_str(key: &str) -> bool {
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index d1f70e24..4f00d3de 100644
--- a/lightning/src/util/persist.rs
+++ b/lightning/src/util/persist.rs
@@ -32,4 +32,7 @@ use crate::util::ser::{ReadableArgs, Writeable};
 pub const KVSTORE_NAMESPACE_KEY_ALPHABET: &str = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-";

+/// The maximum number of characters keys may have.
+pub const KVSTORE_MAX_KEY_LEN: usize = 120;
+
 /// The namespace under which the [`ChannelManager`] will be persisted.
 pub const CHANNEL_MANAGER_PERSISTENCE_NAMESPACE: &str = "";
@@ -66,6 +69,6 @@ pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
 ///
 /// Keys and namespaces are required to be valid ASCII strings in the range of
-/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than 120 characters. Empty namespaces and
-/// sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
+/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_MAX_KEY_LEN`]. Empty namespaces
+/// and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
 /// `sub_namespace` is required to be empty, too. This means that concerns should always be
 /// separated by namespace first, before sub-namespaces are used. While the number of namespaces

tnull · 2023-09-07T09:10:45Z

So, I grew paranoid about hitting the max path length limitations on windows and added a corresponding test case. Low and behold there of course was an issue, so I now included a fix (canonicalizating paths on Windows):

g> git diff-tree -U2 9e89a53b af2bd741
diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index 7e5c2568..77c03a64 100644
--- a/lightning-persister/src/fs_store.rs
+++ b/lightning-persister/src/fs_store.rs
@@ -67,4 +67,26 @@ impl FilesystemStore {
                }
        }
+
+       fn get_dest_dir_path(&self, namespace: &str, sub_namespace: &str) -> std::io::Result<PathBuf> {
+               let mut dest_dir_path = {
+                       #[cfg(target_os = "windows")]
+                       {
+                               let data_dir = self.data_dir.clone();
+                               fs::create_dir_all(data_dir.clone())?;
+                               fs::canonicalize(data_dir)?
+                       }
+                       #[cfg(not(target_os = "windows"))]
+                       {
+                               self.data_dir.clone()
+                       }
+               };
+
+               dest_dir_path.push(namespace);
+               if !sub_namespace.is_empty() {
+                       dest_dir_path.push(sub_namespace);
+               }
+
+               Ok(dest_dir_path)
+       }
 }

@@ -73,9 +95,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, Some(key), "read")?;

-               let mut dest_file_path = self.data_dir.clone();
-               dest_file_path.push(namespace);
-               if !sub_namespace.is_empty() {
-                       dest_file_path.push(sub_namespace);
-               }
+               let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
                dest_file_path.push(key);

@@ -100,9 +118,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, Some(key), "write")?;

-               let mut dest_file_path = self.data_dir.clone();
-               dest_file_path.push(namespace);
-               if !sub_namespace.is_empty() {
-                       dest_file_path.push(sub_namespace);
-               }
+               let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
                dest_file_path.push(key);

@@ -191,9 +205,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, Some(key), "remove")?;

-               let mut dest_file_path = self.data_dir.clone();
-               dest_file_path.push(namespace);
-               if !sub_namespace.is_empty() {
-                       dest_file_path.push(sub_namespace);
-               }
+               let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
                dest_file_path.push(key);

@@ -284,10 +294,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, None, "list")?;

-               let mut prefixed_dest = self.data_dir.clone();
-               prefixed_dest.push(namespace);
-               if !sub_namespace.is_empty() {
-                       prefixed_dest.push(sub_namespace);
-               }
-
+               let prefixed_dest = self.get_dest_dir_path(namespace, sub_namespace)?;
                let mut keys = Vec::new();

diff --git a/lightning-persister/src/test_utils.rs b/lightning-persister/src/test_utils.rs
index 718b8ef1..91557500 100644
--- a/lightning-persister/src/test_utils.rs
+++ b/lightning-persister/src/test_utils.rs
@@ -1,3 +1,3 @@
-use lightning::util::persist::{KVStore, read_channel_monitors};
+use lightning::util::persist::{KVStore, KVSTORE_NAMESPACE_KEY_MAX_LEN, read_channel_monitors};
 use lightning::ln::functional_test_utils::{connect_block, create_announced_chan_between_nodes,
        create_chanmon_cfgs, create_dummy_block, create_network, create_node_cfgs, create_node_chanmgrs,
@@ -39,4 +39,20 @@ pub(crate) fn do_read_write_remove_list_persist<K: KVStore + RefUnwindSafe>(kv_s
        let listed_keys = kv_store.list(namespace, sub_namespace).unwrap();
        assert_eq!(listed_keys.len(), 0);
+
+       // Ensure we have no issue operating with namespace/sub_namespace/key being KVSTORE_NAMESPACE_KEY_MAX_LEN
+       let max_chars: String = std::iter::repeat('A').take(KVSTORE_NAMESPACE_KEY_MAX_LEN).collect();
+       kv_store.write(&max_chars, &max_chars, &max_chars, &data).unwrap();
+
+       let listed_keys = kv_store.list(&max_chars, &max_chars).unwrap();
+       assert_eq!(listed_keys.len(), 1);
+       assert_eq!(listed_keys[0], max_chars);
+
+       let read_data = kv_store.read(&max_chars, &max_chars, &max_chars).unwrap();
+       assert_eq!(data, &*read_data);
+
+       kv_store.remove(&max_chars, &max_chars, &max_chars, false).unwrap();
+
+       let listed_keys = kv_store.list(&max_chars, &max_chars).unwrap();
+       assert_eq!(listed_keys.len(), 0);
 }

diff --git a/lightning-persister/src/utils.rs b/lightning-persister/src/utils.rs
index ae83aa47..54ec230d 100644
--- a/lightning-persister/src/utils.rs
+++ b/lightning-persister/src/utils.rs
@@ -1,8 +1,8 @@
-use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_MAX_KEY_LEN};
+use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_NAMESPACE_KEY_MAX_LEN};
 use lightning::util::string::PrintableString;


 pub(crate) fn is_valid_kvstore_str(key: &str) -> bool {
-       key.len() <= KVSTORE_MAX_KEY_LEN && key.chars().all(|c| KVSTORE_NAMESPACE_KEY_ALPHABET.contains(c))
+       key.len() <= KVSTORE_NAMESPACE_KEY_MAX_LEN && key.chars().all(|c| KVSTORE_NAMESPACE_KEY_ALPHABET.contains(c))
 }

diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index 4f00d3de..c476edd7 100644
--- a/lightning/src/util/persist.rs
+++ b/lightning/src/util/persist.rs
@@ -32,6 +32,6 @@ use crate::util::ser::{ReadableArgs, Writeable};
 pub const KVSTORE_NAMESPACE_KEY_ALPHABET: &str = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-";

-/// The maximum number of characters keys may have.
-pub const KVSTORE_MAX_KEY_LEN: usize = 120;
+/// The maximum number of characters namespaces and keys may have.
+pub const KVSTORE_NAMESPACE_KEY_MAX_LEN: usize = 120;

 /// The namespace under which the [`ChannelManager`] will be persisted.
@@ -69,7 +69,7 @@ pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
 ///
 /// Keys and namespaces are required to be valid ASCII strings in the range of
-/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_MAX_KEY_LEN`]. Empty namespaces
-/// and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
-/// `sub_namespace` is required to be empty, too. This means that concerns should always be
+/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_NAMESPACE_KEY_MAX_LEN`]. Empty
+/// namespaces and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is
+/// empty, `sub_namespace` is required to be empty, too. This means that concerns should always be
 /// separated by namespace first, before sub-namespaces are used. While the number of namespaces
 /// will be relatively small and is determined at compile time, there may be many sub-namespaces

lightning/src/util/persist.rs

lightning-persister/src/fs_store.rs

lightning/src/util/persist.rs

lightning-persister/src/fs_store.rs

lightning/src/util/persist.rs

We upstream the `KVStore` interface trait from LDK Node, which will replace `KVStorePersister` in the coming commits. Besides persistence, `KVStore` implementations will also offer to `list` keys present in a given `namespace` and `read` the stored values.

We add a utility function needed by upcoming `KVStore` implementation tests.

We upstream the `FilesystemStore` implementation, which is backwards compatible with `lightning-persister::FilesystemPersister`.

This replaces the `FilesystemPersister::read_channelmonitors` method, as we can now implement a single utility for all `KVStore`s.

Firstly, we switch our BP over to use `FilesystemStore`, which also gives us test coverage and ensures the compatibility. Then, we remove the superseded `KVStorePersister` trait and the `FilesystemPersister` code.

We re-add benchmarking for `FilesystemStore` now that we switched over to it.

tnull · 2023-09-07T20:50:32Z

Addressed the feedback and force-pushed with the following changes:

> git diff-tree -U2 af2bd741 7a656719
diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index 77c03a64..56d071da 100644
--- a/lightning-persister/src/fs_store.rs
+++ b/lightning-persister/src/fs_store.rs
@@ -240,5 +240,5 @@ impl KVStore for FilesystemStore {
                                        //
                                        // In order to assert we permanently removed the file in question we therefore
-                                       // call `fsync` on the parent directory on platforms that support it,
+                                       // call `fsync` on the parent directory on platforms that support it.
                                        dir_file.sync_all()?;
                                }
@@ -328,5 +328,5 @@ impl KVStore for FilesystemStore {
                        // If we otherwise don't find a file at the given path something went wrong.
                        if !metadata.is_file() {
-                               debug_assert!(false, "Failed to list keys of {}/{}: file coulnd't be accessed.",
+                               debug_assert!(false, "Failed to list keys of {}/{}: file couldn't be accessed.",
                                        PrintableString(namespace), PrintableString(sub_namespace));
                                let msg = format!("Failed to list keys of {}/{}: file couldn't be accessed.",
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index c476edd7..ca0605c9 100644
--- a/lightning/src/util/persist.rs
+++ b/lightning/src/util/persist.rs
@@ -61,5 +61,5 @@ pub const SCORER_PERSISTENCE_SUB_NAMESPACE: &str = "";
 pub const SCORER_PERSISTENCE_KEY: &str = "scorer";

-/// Provides an interface that allows to store and retrieve persisted values that are associated
+/// Provides an interface that allows storage and retrieval of persisted values that are associated
 /// with given keys.
 ///
@@ -147,5 +147,5 @@ impl<'a, A: KVStore, M: Deref, T: Deref, ES: Deref, NS: Deref, SP: Deref, F: Der
                L::Target: 'static + Logger,
 {
-       /// Persist the given ['ChannelManager'] to disk, returning an error if persistence failed.
+       /// Persist the given [`ChannelManager`] to disk, returning an error if persistence failed.
        fn persist_manager(&self, channel_manager: &ChannelManager<M, T, ES, NS, SP, F, R, L>) -> Result<(), io::Error> {
                self.write(CHANNEL_MANAGER_PERSISTENCE_NAMESPACE,

valentinewallace

This seems like it's had a lot of detailed review already so I'll ack in case we want to merge today and take another glance tomorrow. Can't see there being any blocking feedback.

G8XSU

Ship it !

TheBlueMatt

One thing that needs addressing in a followup, but given the stack of PRs waiting, gonna go ahead and land this.

lightning-persister/src/fs_store.rs

lightning/src/util/persist.rs

tnull commented Aug 4, 2023

View reviewed changes

lightning-storage/Cargo.toml Outdated Show resolved Hide resolved

tnull force-pushed the 2023-08-add-kvstore branch 3 times, most recently from 1697806 to b4b6599 Compare August 4, 2023 09:16

tnull mentioned this pull request Aug 4, 2023

Add SQLite KVStore implementation #2473

Open

tnull force-pushed the 2023-08-add-kvstore branch 3 times, most recently from fe9ad36 to 933c834 Compare August 4, 2023 09:52

tnull mentioned this pull request Aug 4, 2023

Monitor updating persister #2359

Merged

tnull added this to the 0.0.117 milestone Aug 4, 2023

tnull force-pushed the 2023-08-add-kvstore branch 9 times, most recently from 5864ce0 to 00937c5 Compare August 7, 2023 11:51

TheBlueMatt reviewed Aug 13, 2023

View reviewed changes

tnull force-pushed the 2023-08-add-kvstore branch from 00937c5 to 4b3302f Compare August 14, 2023 08:03

tnull force-pushed the 2023-08-add-kvstore branch 4 times, most recently from a16ad2e to 157a613 Compare August 14, 2023 13:06

tnull force-pushed the 2023-08-add-kvstore branch from 09b9fa8 to 9e89a53 Compare September 7, 2023 05:30

tnull force-pushed the 2023-08-add-kvstore branch 6 times, most recently from a5d704e to bcd62ed Compare September 7, 2023 09:07

tnull force-pushed the 2023-08-add-kvstore branch from bcd62ed to af2bd74 Compare September 7, 2023 09:12

valentinewallace reviewed Sep 7, 2023

View reviewed changes

tnull force-pushed the 2023-08-add-kvstore branch from af2bd74 to 4308a8b Compare September 7, 2023 20:47

tnull added 9 commits September 7, 2023 22:49

Add KVStore interface trait

d1405f3

We upstream the `KVStore` interface trait from LDK Node, which will replace `KVStorePersister` in the coming commits. Besides persistence, `KVStore` implementations will also offer to `list` keys present in a given `namespace` and `read` the stored values.

Update lightning-persister crate

931ea26

Add test_utils

c18f43f

We add a utility function needed by upcoming `KVStore` implementation tests.

Add FilesystemStore

f22d1b6

We upstream the `FilesystemStore` implementation, which is backwards compatible with `lightning-persister::FilesystemPersister`.

Add read_channel_monitors utility

4305ee4

This replaces the `FilesystemPersister::read_channelmonitors` method, as we can now implement a single utility for all `KVStore`s.

Migrate to KVStore/FilesystemStore

cc1b505

Firstly, we switch our BP over to use `FilesystemStore`, which also gives us test coverage and ensures the compatibility. Then, we remove the superseded `KVStorePersister` trait and the `FilesystemPersister` code.

Migrate FilesystemPersister tests to FilesystemStore

413f9a7

Add benchmarking for FilesystemStore

7576c89

We re-add benchmarking for `FilesystemStore` now that we switched over to it.

Add TestStore implementation of KVStore

7a65671

tnull force-pushed the 2023-08-add-kvstore branch from 4308a8b to 7a65671 Compare September 7, 2023 20:49

valentinewallace approved these changes Sep 7, 2023

View reviewed changes

G8XSU approved these changes Sep 7, 2023

View reviewed changes

TheBlueMatt approved these changes Sep 7, 2023

View reviewed changes

lightning-persister/src/fs_store.rs Show resolved Hide resolved

lightning-persister/src/fs_store.rs Show resolved Hide resolved

lightning/src/util/persist.rs Show resolved Hide resolved

TheBlueMatt merged commit 2e113b5 into lightningdevkit:main Sep 7, 2023
10 of 14 checks passed

TheBlueMatt mentioned this pull request Sep 7, 2023

read_channel_monitors can panic on invalid keys #2561

Closed

tnull mentioned this pull request Sep 8, 2023

KVStore upstreaming followups #2563

Merged

tnull mentioned this pull request Nov 14, 2023

Move object reads to KVStorePersister post-1417 #1427

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace `KVStorePersister` with `KVStore` #2472

Replace `KVStorePersister` with `KVStore` #2472

tnull commented Aug 4, 2023 •

edited

Loading

codecov-commenter commented Aug 4, 2023 •

edited

Loading

tnull commented Aug 4, 2023 •

edited

Loading

tnull commented Aug 9, 2023

tnull commented Aug 14, 2023

tnull commented Aug 14, 2023

tnull commented Sep 7, 2023

tnull commented Sep 7, 2023 •

edited

Loading

tnull commented Sep 7, 2023

valentinewallace left a comment

G8XSU left a comment

TheBlueMatt left a comment

Replace KVStorePersister with KVStore #2472

Replace KVStorePersister with KVStore #2472

Conversation

tnull commented Aug 4, 2023 • edited Loading

codecov-commenter commented Aug 4, 2023 • edited Loading

Codecov Report

tnull commented Aug 4, 2023 • edited Loading

tnull commented Aug 9, 2023

tnull commented Aug 14, 2023

tnull commented Aug 14, 2023

tnull commented Sep 7, 2023

tnull commented Sep 7, 2023 • edited Loading

tnull commented Sep 7, 2023

valentinewallace left a comment

Choose a reason for hiding this comment

G8XSU left a comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Replace `KVStorePersister` with `KVStore` #2472

Replace `KVStorePersister` with `KVStore` #2472

tnull commented Aug 4, 2023 •

edited

Loading

codecov-commenter commented Aug 4, 2023 •

edited

Loading

tnull commented Aug 4, 2023 •

edited

Loading

tnull commented Sep 7, 2023 •

edited

Loading