Implement lazy deletes for VssStore
#689
Conversation
We bump the LDK depedencies to the just-released 2.0 release candidate and account for minor last-minute API changes.
We implement `lazy` deletion for `VssStore` by tracking pending lazy deletes and supplying them as `delete_items` on the next `put` operation.
|
👋 Thanks for assigning @joostjager as a reviewer! |
We add a testcase that ensures we only delete a lazily-deleted key after the next write operation succeeds. Co-authored by Claude AI
tnull
force-pushed
the
2025-10-bump-to-ldk-2.0rc1
branch
from
a64be1d to
0a26053
Compare
| .pending_lazy_deletes | ||
| .try_lock() | ||
| .ok() | ||
| .and_then(|mut guard| guard.take()) |
There was a problem hiding this comment.
Hmm, we might lose some lazy deletes if the write below would fail. I do wonder if we should go out of our way to restore the pending items in such a case, or if we're fine just leaning into the 'may or may not succeed' API contract here.
@joostjager Any opinion?
Similiarly, I do wonder if we should spawn-and-forget some tasks on Drop to attempt cleaning up the pending deletes on shutdown?
There was a problem hiding this comment.
Good question how loose we can get away with. Those keys are then never deleted anymore, which isn't great? I am not sure.
There was a problem hiding this comment.
What do you both think about re-adding the delete_items back to pending_lazy_deletes if write fails? We get to retry deleting them on a subsequent write attempt.
joostjager
left a comment
joostjager
left a comment
There was a problem hiding this comment.
I wonder if lazy delete is needed for VSS? Maybe we can just ignore the flag and delete sync always.
Well, esp. now that we're on |
|
I just don't know if it is worth it for end-user nodes to worry about this. Even persisting full monitors always apparently was good enough performance? In this PR the delete semantics are again different, because a write is needed before deletes are executed. It doesn't make it easier to understand. |
I'm not sure the 'end user' would ever need to worry about, as there are very little As you know I was open to dropping the |
joostjager
left a comment
joostjager
left a comment
There was a problem hiding this comment.
Code looks good. Just not sure about the implications of lingering data if something goes wrong / a write never happens.
In particular for the incremental channel updates, it would be good if that is always followed by a write somehow without much delay. Idk if that's indeed the case?
| .pending_lazy_deletes | ||
| .try_lock() | ||
| .ok() | ||
| .and_then(|mut guard| guard.take()) |
There was a problem hiding this comment.
Good question how loose we can get away with. Those keys are then never deleted anymore, which isn't great? I am not sure.
|
|
||
| let delete_items = self | ||
| .pending_lazy_deletes | ||
| .try_lock() |
There was a problem hiding this comment.
I briefly thought "oh why can't we do this inside the lock that we already obtain", but that doesn't work because here we are also processing deletes on other keys.
| let obfuscated_key = | ||
| self.build_obfuscated_key(&primary_namespace, &secondary_namespace, &key); | ||
|
|
||
| let key_value = KeyValue { key: obfuscated_key, version: -1, value: vec![] }; |
There was a problem hiding this comment.
Can't we store just keys here?
tnull
changed the title
lazy deletes for VssStorelazy deletes for VssStore
4 participants
Now based on #691
We bump the LDK depedencies to the just-released 2.0 release candidate and account for minor last-minute API changes.
Furthermore, we implement
lazydeletion forVssStoreby tracking pending lazy deletes and supplying them asdelete_itemson the nextputoperation.