kernel pos index #3228
kernel pos index #3228
Conversation
antiochp
force-pushed
the
kernel_pos_index
branch
from
d36ff75
to
4311559
Compare
antiochp
force-pushed
the
kernel_pos_index
branch
from
7cb74a6
to
ecd7175
Compare
| debug!( | ||
| "init_kernel_pos_index: {} kernels, took {}s", | ||
| kernel_count, | ||
| now.elapsed().as_secs() |
There was a problem hiding this comment.
Would be interesting to see if there is any difference with the comment above (without looping on an inclusive range).
| .kernel_pos_iter()? | ||
| .filter(|(_, (_, height))| *height < cutoff_height) | ||
| .map(|(key, _)| batch.delete(&key)) | ||
| .count(); |
There was a problem hiding this comment.
Out of curiosity I'm wondering if there is a more efficient way to do that. Like some kind of filter_map or for_each.
Also we are loosing the error of the batch delete without any logging (I know it's actually the same situation for the output_pos)?
There was a problem hiding this comment.
What do you mean in terms of "efficient"?
We need to iterate over the full set of entries in the db.
|
Edit: We do not want to maintain this purely in memory. We actually want transactional (lmdb database) semantics to allow efficient "rewind" via the per block kernel "undo list". |
antiochp
force-pushed
the
kernel_pos_index
branch
2 times, most recently
from
25105bf
to
531506c
Compare
…undo this is conceptually similar to output_pos but we support duplicates rather than spending to remove
store the kernel_pos index when we call apply_kernel
antiochp
force-pushed
the
kernel_pos_index
branch
from
3342a65
to
eec3137
Compare
|
Closing, replaced with #3273 (which is a more general solution). |
Maintain index of "recent" kernels so we can quickly lookup a kernel MMR pos (and block height) by kernel excess commitment.
This is similar to the existing
output_posindex.Important difference is outputs can be removed and pruned (after being spent), kernels are never removed. But we only want to maintain an index of "recent" kernels sufficient for the proposed NSKR relative kernel locks.
Compact the kernel pos index when we compact the txhashset. We maintain 2 weeks of kernel history in the index. This covers 7 days of full block history back to the horizon (for rewind purposes) and a further 7 days of kernel history to cover the max NSKR relative kernel lock period.
Populate the index as we go when calling
apply_kernel()during processing of new full blocks.Add test coverage around chain compaction and interaction with the kernel pos index.
Track an "undo list" for each full block so we can rewind the kernel_pos index as required when rewinding a block.
We do not need to track the kernel_pos index currently as it is unused.
But we need this implemented for efficient handling of NRD kernel lookups.
Once we have the NRD kernel feature variant we can modify the kernel_pos index to only index recent NRD kernels (currently we index all recent kernels).
Related #3273.
If we track
kernel_posvia the "linked list" impl then we no longer need to maintain the "undo list" per block.We can simply pop entries off the head of the list during rewind, rewinding each kernel in the block, leaving the index in the previous state in the case of duplicates.