Add queues to track the need for transaction enhancement and/or verification of mined status. by nuttycom · Pull Request #1473 · zcash/librustzcash

nuttycom · 2024-07-30T18:35:09Z

No description provided.

…key values.

codecov · 2024-08-01T23:12:50Z

Codecov Report

Attention: Patch coverage is 79.27461% with 40 lines in your changes missing coverage. Please review.

Project coverage is 61.16%. Comparing base (7f7b685) to head (5a1645a).
Report is 8 commits behind head on main.

Files	Patch %	Lines
zcash_client_sqlite/src/wallet.rs	83.07%	11 Missing ⚠️
zcash_client_sqlite/src/wallet/transparent.rs	73.80%	11 Missing ⚠️
zcash_client_sqlite/src/lib.rs	86.00%	7 Missing ⚠️
zcash_client_backend/src/data_api.rs	28.57%	5 Missing ⚠️
...e/src/wallet/init/migrations/tx_retrieval_queue.rs	66.66%	4 Missing ⚠️
zcash_client_sqlite/src/wallet/orchard.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1473      +/-   ##
==========================================
+ Coverage   60.82%   61.16%   +0.34%     
==========================================
  Files         140      141       +1     
  Lines       16472    16650     +178     
==========================================
+ Hits        10019    10184     +165     
- Misses       6453     6466      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

str4d

Flushing comments up to 40d94d0 (just before looking at db.rs).

str4d

Reviewed 1cbced3.

…ion.

request in `set_transaction_status`. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

…nsfer`. fixes zcash#1485 Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

str4d

utACK 171cc3d once my concern about Enhancement results has been addressed (either by explaining why it's fine, or making a fix).

str4d · 2024-08-09T21:17:06Z

+    // In order to have made a Status request, we must already have had the
+    // raw transaction (the only callers of `queue_tx_retrieval` are in contexts
+    // where we must have already called `put_tx_data`). Therefore, it is safe
+    // to unconditionally delete the request from `tx_retrieval_queue` below
+    // (both in the expired case and the case where it has been mined), because
+    // we already have all the data we need about this transaction.


This is true for Status requests, but not true for Enhancement requests which subsume Status.

Yes. I noticed that and the first commit of nuttycom#39 (nuttycom@239f7ff) fixes it. The implementation is fine, it's just the comment that is incorrect.

It occurs to me that https://github.com/zcash/librustzcash/pull/1473/files#diff-a1724f7547ecabc613a07520eaf7917d9084d4277219d186698b0c8e352495a4R2425-R2427 is superfluous; we should never have a Status request for a transaction unless we have the raw data. So one way to fix this would be to remove the query type argument from the queue_tx_retrieval function, and to set it based upon the presence or absence of raw transaction data.

The comment change doesn't resolve my concern. GetStatus is indeed only generated when we have the raw transaction (as I noted), but Enhancement is generated by definition when we don't have the raw transaction, and its documentation states that when the caller is unable to look up the transaction, it must call this method.

The reason it is okay to delete for Enhancement requests is that:

If a transaction can't be enhanced, we assume that the caller calls this method with TransactionStatus::TxidNotRecognized (as the other two statuses mean that the chain data source knew about the transaction, and therefore must have observed it and could return its raw data), so we don't need to consider what happens in the other statuses.

We only delete for TransactionStatus::TxidNotRecognized if the transaction is expired, at which point we don't necessarily care about enhancement (though the user might still care about the memo), and it is very likely that we will be unable to ever enhance (as the tx will have been dropped from all mempools).

Simplified in 5a1645a

The fact is that an Enhancement request is a Status request. That's what it means for an Enhancement request to subsume a Status request. It's either "get me the status" or "get me the status and the data if it's available".

@daira Enhancement is getting the status; it subsumes / is a superset of GetStatus. That's why I raised my initial concern that this method was being written ostensibly assuming it would only be called via GetStatus, which was false.

As @nuttycom says, the actual implementation does correctly work with both GetStatus and Enhancement; the comment was just misleading.

I have now updated this comment again. (b47c7bf)

The fact is that an Enhancement request is a Status request.

Not so. Otherwise it wouldn't have to requeue a Status request when queue_status_retrieval is true. It doesn't directly update the mined height in the case where d_tx.mined_height().is_none().

Otherwise it wouldn't have to requeue a Status request when queue_status_retrieval is true.

The re-queueing is so that the wallet continues to monitor for when the transaction is eventually mined. A relevant case here is that store_decrypted_tx was called with a transaction that was read from the mempool; we have complete transaction data, but we don't know yet whether the transaction will be mined, so we stick the status request back on the queue. For a fully transparent transaction that we get from the mempool we have no other way of detecting that it becomes mined.

daira

utACK modulo suggested changes in nuttycom#39 . We can either merge that PR into this one, or merge a copy of it separately. (I have checked that it passes tests locally.)

…o wallet/enrichment_queue

…of raw tx data.

…action_status`

str4d

utACK b47c7bf

daira

Post-hoc ACK

…14868de..d978256a2 d978256a2 Merge pull request #1 from zcash/compact_tx_transparent 7eeb82e7c Merge pull request #4 from zcash/add_changelog a95359dc9 Apply suggestions from code review 592b637a8 Add transparent data to the `CompactBlock` format. 9d1fb2c41 Add a CHANGELOG.md that documents the evolution of the light client protocol. 180717dfa Merge pull request #3 from zcash/merge_librustzcash_history 450bd4181 Merge the history of the .proto files from `librustzcash` for complete history preservation. a4859d11d Move protobuf files into place for use in `zcash/lightwallet-protocol` 2e66cdd9e Update zcash_client_backend/proto/service.proto eda012519 fix comment f838d10ad Add gRPC LightdInfo Donation Address db12c0415 Merge pull request #1473 from nuttycom/wallet/enrichment_queue 698feba96 Apply suggestions from code review 20ce57ab3 zcash_client_backend: Add `block_height` argument to `decrypt_and_store_transaction` a6dea1da8 Merge pull request #1482 from zancas/doc_tweak 4d2d45fc9 fix incorrect doc-comment e826f4740 update CompactBlock doc-comment, to cover non-Sapling shielded notes, and addresses e9a6c00bf Various documentation improvements 988bc7214 Merge pull request #872 from nuttycom/feature/pre_dag_sync-suggest_scan_ranges 58d07d469 Implement `suggest_scan_ranges` and `update_chain_tip` a9222b338 Address comments from code review. e20310857 Rename proto::compact::{BlockMetadata => ChainMetadata} ac63418c5 Reorganize Sapling and Orchard note commitment tree sizes in CompactBlock. 0fdca14f1 zcash_client_backend: Add note commitment tree sizes to `CompactBlock` serialization. 2a0c2b8b7 zcash_client_backend: Add gRPC bindings behind feature flag 1342f0480 zcash_client_backend: Address compact_formats.proto comments 68aa4e01b zcash_client_backend: Bring in latest `compact_formats.proto` e712eb1bc Add prevHash field to CompactBlock 440384c3e Build protobufs for compact formats git-subtree-dir: zcash_client_backend/lightwallet-protocol git-subtree-split: d978256a2bf33bb5f83846d8994e7377512f5f05

…14868de..23f0768ea 23f0768ea Release lightwallet-protocol v0.4.0 41156c767 Merge pull request #11 from zcash/feature/get_mempool_tx_pools 7c130e883 Add `lightwalletProtocolVersion` field to `LightdInfo` struct. edbb726d7 Apply suggestion from code review 38fddd73b Apply suggestions from code review 0250f2720 Add pool type filtering to `GetMempoolTx` argument. 54ccaadd5 Change semantics of pool-based pruning of compact transactions from "may prune" to "must prune". b0667ec99 Merge pull request #9 from zcash/2025-11-doc-TransparentAddressBlockFilter f3fea7bd4 doc: TransparentAddressBlockFilter doesn't include mempool a67dd323a Merge pull request #8 from zcash/2025-11-lightdinfo-upgrade-info 11da4b7e3 add next upgrade info to LightdInfo structure (GetLightdInfo) 42cd8f720 Transparent data docs update (#7) c0cf957ac Merge pull request #5 from zcash/2025-11-comments 912fc3609 Minor clarification in GetBlockRange documentation. 6b03f2cce Documentation (comments) only d978256a2 Merge pull request #1 from zcash/compact_tx_transparent 7eeb82e7c Merge pull request #4 from zcash/add_changelog a95359dc9 Apply suggestions from code review 592b637a8 Add transparent data to the `CompactBlock` format. 9d1fb2c41 Add a CHANGELOG.md that documents the evolution of the light client protocol. 180717dfa Merge pull request #3 from zcash/merge_librustzcash_history 450bd4181 Merge the history of the .proto files from `librustzcash` for complete history preservation. a4859d11d Move protobuf files into place for use in `zcash/lightwallet-protocol` 2e66cdd9e Update zcash_client_backend/proto/service.proto eda012519 fix comment f838d10ad Add gRPC LightdInfo Donation Address db12c0415 Merge pull request #1473 from nuttycom/wallet/enrichment_queue 698feba96 Apply suggestions from code review 20ce57ab3 zcash_client_backend: Add `block_height` argument to `decrypt_and_store_transaction` a6dea1da8 Merge pull request #1482 from zancas/doc_tweak 4d2d45fc9 fix incorrect doc-comment e826f4740 update CompactBlock doc-comment, to cover non-Sapling shielded notes, and addresses e9a6c00bf Various documentation improvements 988bc7214 Merge pull request #872 from nuttycom/feature/pre_dag_sync-suggest_scan_ranges 58d07d469 Implement `suggest_scan_ranges` and `update_chain_tip` a9222b338 Address comments from code review. e20310857 Rename proto::compact::{BlockMetadata => ChainMetadata} ac63418c5 Reorganize Sapling and Orchard note commitment tree sizes in CompactBlock. 0fdca14f1 zcash_client_backend: Add note commitment tree sizes to `CompactBlock` serialization. 2a0c2b8b7 zcash_client_backend: Add gRPC bindings behind feature flag 1342f0480 zcash_client_backend: Address compact_formats.proto comments 68aa4e01b zcash_client_backend: Bring in latest `compact_formats.proto` e712eb1bc Add prevHash field to CompactBlock 440384c3e Build protobufs for compact formats git-subtree-dir: zcash_client_backend/lightwallet-protocol git-subtree-split: 23f0768ea4471b63285f3c0e9b6fbb361674aa2b

zcash_client_sqlite: Add an internal newtype for transaction primary …

301e8b4

…key values.

nuttycom marked this pull request as draft July 30, 2024 18:35

daira reviewed Jul 30, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/wallet/init/migrations/tx_retrieval_queue.rs Outdated

nuttycom force-pushed the wallet/enrichment_queue branch from 7709120 to 9e22493 Compare July 30, 2024 18:45

daira reviewed Jul 30, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/wallet/init/migrations/tx_retrieval_queue.rs Outdated

daira reviewed Jul 30, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/wallet/init/migrations/tx_retrieval_queue.rs

daira reviewed Jul 30, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/wallet/init/migrations/tx_retrieval_queue.rs Outdated

nuttycom force-pushed the wallet/enrichment_queue branch 3 times, most recently from ecee800 to 8749d37 Compare August 1, 2024 22:51

str4d reviewed Aug 2, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/wallet.rs Outdated

nuttycom force-pushed the wallet/enrichment_queue branch 2 times, most recently from d3837d4 to 9ffcb8e Compare August 2, 2024 00:47

str4d reviewed Aug 2, 2024

View reviewed changes

nuttycom force-pushed the wallet/enrichment_queue branch 5 times, most recently from 7994e82 to cc3c34f Compare August 5, 2024 17:06

str4d mentioned this pull request Aug 5, 2024

bug: getrawtransaction output conflicts with zcashd output ZcashFoundation/zebra#8744

Closed

4 tasks

nuttycom force-pushed the wallet/enrichment_queue branch 3 times, most recently from 93d75b0 to 1cbced3 Compare August 5, 2024 17:57

str4d requested changes Aug 5, 2024

View reviewed changes

nuttycom force-pushed the wallet/enrichment_queue branch from 1cbced3 to 4741662 Compare August 5, 2024 21:28

daira reviewed Aug 5, 2024

View reviewed changes

Comment thread zcash_client_backend/src/data_api.rs

daira reviewed Aug 5, 2024

View reviewed changes

Comment thread zcash_client_backend/src/data_api.rs Outdated

daira reviewed Aug 5, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/lib.rs Outdated

daira reviewed Aug 5, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/lib.rs Outdated

nuttycom and others added 2 commits August 9, 2024 11:36

zcash_client_sqlite: Add testing for transaction data request generat…

dada980

…ion.

Add a comment explaining why it is safe to unconditionally delete the

d96bc11

request in `set_transaction_status`. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

nuttycom requested review from daira and str4d and removed request for str4d August 9, 2024 18:34

zcash_client_backend: Add target_height to SentTransaction

8752ad7

daira reviewed Aug 9, 2024

View reviewed changes

Comment thread zcash_client_backend/src/data_api.rs

daira reviewed Aug 9, 2024

View reviewed changes

Comment thread zcash_client_backend/src/data_api.rs

nuttycom and others added 4 commits August 9, 2024 15:10

Address review comments in transaction_data_requests

171cc3d

Improve the accuracy of a comment in set_transaction_status.

239f7ff

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Minor simplifications.

56e4233

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Replace the put_tx_meta workaround in `send_multi_step_proposed_tra…

1023ce7

…nsfer`. fixes zcash#1485 Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

str4d requested changes Aug 9, 2024

View reviewed changes

daira previously approved these changes Aug 9, 2024

View reviewed changes

Merge remote-tracking branch 'daira/1485-improve-multi-step-test' int…

0bfa3d3

…o wallet/enrichment_queue

nuttycom dismissed daira’s stale review via 0bfa3d3 August 9, 2024 21:57

nuttycom added 2 commits August 9, 2024 16:55

zcash_client_sqlite: Make transaction retrieval depend upon presence …

5a1645a

…of raw tx data.

zcash_client_sqlite: Improve the internal documentation of `set_trans…

b47c7bf

…action_status`

str4d approved these changes Aug 9, 2024

View reviewed changes

str4d merged commit 52abb1f into zcash:main Aug 9, 2024

nuttycom deleted the wallet/enrichment_queue branch August 9, 2024 23:41

daira reviewed Aug 10, 2024

View reviewed changes

Comment thread zcash_client_sqlite/src/lib.rs

daira reviewed Aug 10, 2024

View reviewed changes

daira mentioned this pull request Aug 10, 2024

Improve the accuracy of a comment in send_multi_step_proposed_transfer #1487

Merged

This was referenced Sep 4, 2024

zcash_client_backend: Add support for full P2PKH history tracking #1379

Closed

Transparent input details should be retrieved with transactions on account recovery #1074

Closed

nuttycom mentioned this pull request Oct 24, 2024

Enable UTXOs spent in expired shielding transactions to be spendable #986

Closed

nuttycom mentioned this pull request Jul 7, 2025

Restore of the fees #1706

Closed

Conversation

nuttycom commented Jul 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

str4d left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

str4d left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

str4d left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

str4d Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

daira Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nuttycom Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

str4d Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

nuttycom Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

nuttycom Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

str4d Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

nuttycom Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daira Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

nuttycom Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daira left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

codecov Bot commented Aug 1, 2024 •

edited

Loading

daira Aug 9, 2024 •

edited

Loading

nuttycom Aug 9, 2024 •

edited

Loading

nuttycom Aug 9, 2024 •

edited

Loading

nuttycom Aug 9, 2024 •

edited

Loading

daira left a comment •

edited

Loading