Skip to content

Conversation

@antonilol
Copy link
Contributor

@antonilol antonilol commented May 24, 2024

In the WriteBatch struct, I changed the Vec<Box<[u8]>> fields to Vec<[u8, N]>, which saves on memory allocations when indexing blocks and putting the results of that in the db, before this pr that adds up to 1 per unique output address per block (funding), 1 per spent utxo per block (spending) and 1 per transaction (txid) (and a bit more for tip_row and header_rows).

When retrieving from the db currently Boxes are still used, but this can be fixed by using rocksdb's raw iterators. (The optimizer might be able to even remove the Boxes now that they are directly dropped, but as with almost all optimizations, they are not guaranteed).
I also managed to fix this, 800000+ less Boxes are now made when retrieving the headers on startup, and a lot too on electrum protocol requests (i think the amount here depends on the amount of blocks the address is used in, for history and balance at least)

And when loading headers, the iterator is not .collect().into_iter()'ed anymore (found that 2 times, so another 2 intermediate allocations saved).

I was experimenting with redb a bit, redb uses fixed size types, unlike rocksdb that only allows &[u8], this change would also make it easier to use redb as db with electrs in the future, I plan on making a pr for adding redb if I get it to work.

Copy link
Contributor

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice!

Copy link
Contributor

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Copy link
Owner

@romanz romanz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for the contribution!
A single comment regarding DB scan:

@romanz romanz merged commit b101e30 into romanz:master Jul 13, 2024
@romanz
Copy link
Owner

romanz commented Jul 13, 2024

Many thanks!

@antonilol antonilol deleted the fixed-size-rows branch July 13, 2024 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants