Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(chain): add get and range methods to CheckPoint #1369

Merged
merged 3 commits into from
Apr 6, 2024

Conversation

evanlinjin
Copy link
Member

@evanlinjin evanlinjin commented Mar 6, 2024

Partially fixes #1354

Description

These methods allow us to query for checkpoints contained within the linked list by height and height range. This is useful to determine checkpoints to fetch for chain sources without having to refer back to the LocalChain.

Currently this is not implemented efficiently, but in the future, we will change CheckPoint to use a skip list structure.

Notes to the reviewers

Please refer to the conversation in #1354 for more details. In summary, both TxGraph::missing_heights and tx_graph::ChangeSet::missing_heights_from are problematic in their own ways. Additionally, it's a better API for chain sources if we only need to lock our receiving structures twice (once to obtain initial state and once for applying the update). Instead of relying on methods such as EsploraExt::update_local_chain to get relevant checkpoints, we can use these query methods instead. This allows up to get rid of ::missing_heights-esc methods and remove the need to lock receiving structures in the middle of the sync.

Changelog notice

  • Added get and range methods to CheckPoint (and in turn, LocalChain). This simulates an API where we have implemented a skip list of checkpoints (to implement in the future). This is a better API because we can query for any height or height range with just a checkpoint tip instead of relying on a separate checkpoint index (which needs to live in LocalChain).
  • Changed LocalChain to have a faster Eq implementation. We now maintain an xor value of all checkpoint block hashes. We compare this xor value to determine whether two chains are equal.
  • Added PartialEq implementation for CheckPoint and local_chain::Update.

Checklists

All Submissions:

  • I've signed all my commits
  • I followed the contribution guidelines
  • I ran cargo fmt and cargo clippy before committing

New Features:

  • I've added tests for the new feature
  • I've added docs for the new feature

@evanlinjin evanlinjin marked this pull request as ready for review March 7, 2024 10:00
@nondiremanuel nondiremanuel added this to the 1.0.0-alpha milestone Mar 12, 2024
@nondiremanuel nondiremanuel added the api A breaking API change label Mar 12, 2024
@evanlinjin evanlinjin requested a review from LLFourn March 14, 2024 03:03
Copy link
Contributor

@LLFourn LLFourn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm slightly concerned we're introducing these methods without using them anywhere. Especially query_multi which I don't think needs to exist yet. I suggest cutting query_multi.

crates/chain/src/local_chain.rs Outdated Show resolved Hide resolved
crates/chain/src/local_chain.rs Outdated Show resolved Hide resolved
@evanlinjin evanlinjin force-pushed the fix/1354 branch 5 times, most recently from 2af78ae to 013f7cc Compare March 17, 2024 12:03
@evanlinjin evanlinjin changed the title feat(chain): add query and query_multi methods to CheckPoint feat(chain): add query and query_from methods to CheckPoint Mar 17, 2024
@evanlinjin evanlinjin force-pushed the fix/1354 branch 2 times, most recently from f0aa380 to 99c1865 Compare March 17, 2024 12:27
@evanlinjin evanlinjin marked this pull request as draft March 17, 2024 13:30
@evanlinjin evanlinjin marked this pull request as ready for review March 18, 2024 12:26
@evanlinjin evanlinjin requested a review from LLFourn March 18, 2024 12:28
@evanlinjin evanlinjin mentioned this pull request Mar 21, 2024
5 tasks
Copy link
Contributor

@LLFourn LLFourn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK aa84ecf

Copy link
Member

@notmandatory notmandatory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK aa84ecf

Appreciate the focused scope for this series of chain PRs, much easier to review. I have one small test code suggestion but otherwise looks good to me.

crates/chain/tests/test_local_chain.rs Outdated Show resolved Hide resolved
crates/chain/tests/test_local_chain.rs Show resolved Hide resolved
evanlinjin added a commit that referenced this pull request Mar 31, 2024
8ab58af feat(chain)!: wrap `TxGraph` txs with `Arc` (志宇)

Pull request description:

  ### Description

  This PR makes `TxGraph` store transactions as `Arc<Transaction>`.

  `Arc<Transaction>` can be shared between the chain-source and receiving structures. This allows the chain-source to keep the already-fetched transactions (save bandwith) and have a shared pointer to the transaction (save memory).

  Our current logic to avoid re-fetching transactions is to refer back to the receiving structures. However, this means an additional round of locking our receiving structures.

  ### Notes to the reviewers

  This will make more sense once I update the esplora/electrum chain sources to make use of both #1369 and this PR. The result would be chain sources which would only require locking the receiving structures twice (once for initiating the update, and once for applying the update).

  ### Changelog notice

  * Changed `TxGraph` to store transactions as `Arc<Transaction>`. This allows chain-sources to cheaply keep a copy of already-fetched transactions.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [ ] I've added tests for the new feature
  * [x] I've added docs for the new feature

ACKs for top commit:
  LLFourn:
    ACK 8ab58af
  notmandatory:
    ACK 8ab58af

Tree-SHA512: 81d7ad35fed7f253a3b902f09a41aaef2877f6201d4f6e78318741bf00e4b898a8000d878ffcbfe75701094ce3e9021bd718b190a655331a9e11f0ad66bdb02f
@evanlinjin
Copy link
Member Author

Ahhh I probably still need to add a few tests.

crates/chain/src/local_chain.rs Outdated Show resolved Hide resolved
crates/chain/src/local_chain.rs Outdated Show resolved Hide resolved
@evanlinjin evanlinjin changed the title feat(chain): add query and range methods to CheckPoint feat(chain): add get and range methods to CheckPoint Apr 3, 2024
Copy link
Contributor

@ValuedMammal ValuedMammal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good.

get sounds like a better name than query. My other suggestion would have been to call it find, since that is how it's implemented.

Interesting that LocalChain now delegates virtually everything to the checkpoint. Curious to see if the structure of LocalChain becomes more relevant when skip list is involved.

crates/chain/src/local_chain.rs Outdated Show resolved Hide resolved
crates/chain/src/local_chain.rs Outdated Show resolved Hide resolved
.range(height..)
.next()
.last()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LLFourn CheckPoint::range is also used here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I missed that one and had forgot about the API of this function. On a side note this is interesting because we are anchoring a tx not to the block that it's in. So even if we made a rule that esplora/electrum etc must tell us about the block that the tx is in we'd still need the concept of anchoring if we want to keep this API.

@LLFourn
Copy link
Contributor

LLFourn commented Apr 4, 2024

ACK cdbc131

@evanlinjin
Copy link
Member Author

I've addressed all review comments and removed the From<LocalChain> impl for BTreeMap<u32, BlockHash> as it wasn't being used.

These methods allow us to query for checkpoints contained within the
linked list by height and height range. This is useful to determine
checkpoints to fetch for chain sources without having to refer back to
the `LocalChain`.

Currently this is not implemented efficiently, but in the future, we
will change `CheckPoint` to use a skip list structure.
We impl `PartialEq` on `CheckPoint` instead of directly on `LocalChain`.
We also made the implementation more efficient.
I don't think this was ever used. The only possible usecase I can think
of is for tests, but I don't think that is a strong enough incentive for
us to keep this.
Copy link
Contributor

@jirijakes jirijakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR helped me better understand some internals. Still learning.

Only have one question and one minor suggestion (not too important).

ACK 6d78f0e

crates/chain/src/local_chain.rs Show resolved Hide resolved
Comment on lines +565 to +575
self.tip = match remove_from.map(|cp| cp.prev()) {
// The checkpoint below the earliest checkpoint to remove will be the new tip.
Some(Some(new_tip)) => new_tip,
// If there is no checkpoint below the earliest checkpoint to remove, it means the
// "earliest checkpoint to remove" is the genesis block. We disallow removing the
// genesis block.
Some(None) => return Err(MissingGenesisError),
// If there is nothing to remove, we return an empty changeset.
None => return Ok(ChangeSet::default()),
};
Ok(changeset)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not too important, but it took me a while to understand what changeset is returned when and what is when assigned to self.tip and when it is left untouched.

Perhaps?

Suggested change
self.tip = match remove_from.map(|cp| cp.prev()) {
// The checkpoint below the earliest checkpoint to remove will be the new tip.
Some(Some(new_tip)) => new_tip,
// If there is no checkpoint below the earliest checkpoint to remove, it means the
// "earliest checkpoint to remove" is the genesis block. We disallow removing the
// genesis block.
Some(None) => return Err(MissingGenesisError),
// If there is nothing to remove, we return an empty changeset.
None => return Ok(ChangeSet::default()),
};
Ok(changeset)
match remove_from.map(|cp| cp.prev()) {
// The checkpoint below the earliest checkpoint to remove will be the new tip.
Some(Some(new_tip)) => {
self.tip = new_tip;
Ok(changeset)
}
// If there is no checkpoint below the earliest checkpoint to remove, it means the
// "earliest checkpoint to remove" is the genesis block. We disallow removing the
// genesis block.
Some(None) => Err(MissingGenesisError),
// If there is nothing to remove, we return an empty changeset.
None => Ok(ChangeSet::default()),
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your suggestion is definitely better, but I'll ignore it now because I want this merged and rebasing is annoying 😅

Copy link
Contributor

@LLFourn LLFourn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 53942cc

@evanlinjin evanlinjin merged commit 53791eb into bitcoindevkit:master Apr 6, 2024
12 checks passed
@notmandatory notmandatory mentioned this pull request Apr 12, 2024
25 tasks
evanlinjin added a commit that referenced this pull request Apr 12, 2024
446b045 test(chain): introduce proptest for `CheckPoint::range` (志宇)

Pull request description:

  ### Description

  This is based on #1369. It made sense for me to introduce the `CheckPoint::range` test as a proptest. I think we should also start introducing it for other methods too.

  ### Changelog notice

  * Added proptest for `CheckPoint::range`.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [x] I've added tests for the new feature
  * [x] I've added docs for the new feature

Top commit has no ACKs.

Tree-SHA512: 2d8aad5a1ad152413fbc716fed4a47ad621e723928c5331728da2fbade995ebd677acfa05d2e7639d74d0f66e0971c72582d9cd562ea008b012774057b480d62
evanlinjin added a commit that referenced this pull request Apr 22, 2024
96a9aa6 feat(chain): refactor `merge_chains` (志宇)
2f22987 chore(chain): fix comment (志宇)
daf588f feat(chain): optimize `merge_chains` (志宇)
77d3595 feat(chain)!: rm `local_chain::Update` (志宇)
1269b06 test(chain): fix incorrect test case (志宇)
72fe65b feat(esplora)!: simplify chain update logic (志宇)
eded1a7 feat(chain): introduce `CheckPoint::insert` (志宇)
519cd75 test(esplora): move esplora tests into src files (志宇)
a6e613e test(esplora): add `test_finalize_chain_update` (志宇)
494d253 feat(testenv): add `genesis_hash` method (志宇)
886d72e chore(chain)!: rm `missing_heights` and `missing_heights_from` methods (志宇)
bd62aa0 feat(esplora)!: remove `EsploraExt::update_local_chain` (志宇)
1e99793 feat(testenv): add `make_checkpoint_tip` (志宇)

Pull request description:

  Fixes #1354

  ### Description

  Built on top of both #1369 and #1373, we simplify the `EsploraExt` API by removing the `update_local_chain` method and having `full_scan` and `sync` update the local chain in the same call. The `full_scan` and `sync` methods now takes in an additional input (`local_tip`) which provides us with the view of the `LocalChain` before the update. These methods now return structs `FullScanUpdate` and `SyncUpdate`.

  The examples are updated to use this new API. `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` are no longer needed, therefore they are removed.

  Additionally, we used this opportunity to simplify the logic which updates `LocalChain`. We got rid of the `local_chain::Update` struct (which contained the update `CheckPoint` tip and a `bool` which signaled whether we want to introduce blocks below point of agreement). It turns out we can use something like `CheckPoint::insert` so the chain source can craft an update based on the old tip. This way, we can make better use of `merge_chains`' optimization that compares the `Arc` pointers of the local and update chain (before we were crafting the update chain NOT based on top of the previous local chain). With this, we no longer need the `Update::introduce_older_block` field since the logic will naturally break when we reach a matching `Arc` pointer.

  ### Notes to the reviewers

  * Obtaining the `LocalChain`'s update now happens within `EsploraExt::full_scan` and `EsploraExt::sync`. Creating the `LocalChain` update is now split into two methods (`fetch_latest_blocks` and `chain_update`) that are called before and after fetching transactions and anchors.
  * We need to duplicate code for `bdk_esplora`. One for blocking and one for async.

  ### Changelog notice

  * Changed `EsploraExt` API so that sync only requires one round of fetching data. The `local_chain_update` method is removed and the `local_tip` parameter is added to the `full_scan` and `sync` methods.
  * Removed `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` methods.
  * Introduced `CheckPoint::insert` which allows convenient checkpoint-insertion. This is intended for use by chain-sources when crafting an update.
  * Refactored `merge_chains` to also return the resultant `CheckPoint` tip.
  * Optimized the update `LocalChain` logic - use the update `CheckPoint` as the new `CheckPoint` tip when possible.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [x] I've added tests for the new feature
  * [x] I've added docs for the new feature

ACKs for top commit:
  LLFourn:
    ACK 96a9aa6

Tree-SHA512: 3d4f2eab08a1fe94eb578c594126e99679f72e231680b2edd4bfb018ba1d998ca123b07acb2d19c644d5887fc36b8e42badba91cd09853df421ded04de45bf69
notmandatory added a commit that referenced this pull request May 10, 2024
b45897e feat(electrum): update docs and simplify logic of `ElectrumExt` (志宇)
92fb6cb chore(electrum): do not use `anyhow::Result` directly (志宇)
b2f3cac feat(electrum): include option for previous `TxOut`s for fee calculation (Wei Chen)
c0d7d60 feat(chain)!: use custom return types for `ElectrumExt` methods (志宇)
2945c6b fix(electrum): fixed `sync` functionality (Wei Chen)
9ed33c2 docs(electrum): fixed `full_scan`, `sync`, and crate documentation (Wei Chen)
b1f861b feat: update logging of electrum examples (志宇)
a6fdfb2 feat(electrum)!: use new sync/full-scan structs for `ElectrumExt` (志宇)
653e4fe feat(wallet): cache txs when constructing full-scan/sync requests (志宇)
58f27b3 feat(chain): introduce `TxCache` to `SyncRequest` and `FullScanRequest` (志宇)
721bb7f fix(chain): Make `Anchor` type in `FullScanResult` generic (志宇)
e3cfb84 feat(chain): `TxGraph::insert_tx` reuses `Arc` (志宇)
2ffb656 refactor(electrum): remove `RelevantTxids` and track txs in `TxGraph` (Wei Chen)

Pull request description:

  Fixes #1265
  Possibly fixes #1419

  ### Context

  Previous changes such as

  * Universal structures for full-scan/sync (PR #1413)
  * Making `CheckPoint` linked list query-able (PR #1369)
  * Making `Transaction`s cheaply-clonable (PR #1373)

  has allowed us to simplify the interaction between chain-source and receiving-structures (`bdk_chain`).

  The motivation is to accomplish something like this ([as mentioned here](#1153 (comment))):
  ```rust
  let things_I_am_interested_in = wallet.lock().unwrap().start_sync();
  let update = electrum_or_esplora.sync(things_i_am_interested_in)?;
  wallet.lock().unwrap().apply_update(update)?:
  ```

  ### Description

  This PR greatly simplifies the API of our Electrum chain-source (`bdk_electrum`) by making use of the aforementioned changes. Instead of referring back to the receiving `TxGraph` mid-sync/scan to determine which full transaction to fetch, we provide the Electrum chain-source already-fetched full transactions to start sync/scan (this is cheap, as transactions are wrapped in `Arc`s since #1373).

  In addition, an option has been added to include the previous `TxOut` for transactions received from an external wallet for fee calculation.

  ### Changelog notice

  * Change `TxGraph::insert_tx` to take in anything that satisfies `Into<Arc<Transaction>>`. This allows us to reuse the `Arc` pointer of what is being inserted.
  * Add `tx_cache` field to `SyncRequest` and `FullScanRequest`.
  * Make `Anchor` type in `FullScanResult` generic for more flexibility.
  * Change `ElectrumExt` methods to take in `SyncRequest`/`FullScanRequest` and return `SyncResult`/`FullScanResult`. Also update electrum examples accordingly.
  * Add `ElectrumResultExt` trait which allows us to convert the update `TxGraph` of `SyncResult`/`FullScanResult` for `bdk_electrum`.
  * Added an option for `full_scan` and `sync` to also fetch previous `TxOut`s to allow for fee calculation.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [x] I've added tests for the new feature
  * [x] I've added docs for the new feature

ACKs for top commit:
  ValuedMammal:
    ACK b45897e
  notmandatory:
    ACK b45897e

Tree-SHA512: 1e274546015e7c7257965b36079ffe0cb3c2c0b7c2e0c322bcf32a06925a0c3e1119da1c8fd5318f1dbd82c2e952f6a07f227a9b023c48f506a62c93045d96d3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api A breaking API change module-blockchain
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

The TxGraph::missing_heights implementation is problematic.
6 participants