Skip to content

2/5: External-only scanning with sync loop enhancement pipeline#15

Closed
czarcas7ic wants to merge 1 commit into
adam/enriched-decrypted-outputfrom
adam/sync-enhancement-pipeline
Closed

2/5: External-only scanning with sync loop enhancement pipeline#15
czarcas7ic wants to merge 1 commit into
adam/enriched-decrypted-outputfrom
adam/sync-enhancement-pipeline

Conversation

@czarcas7ic
Copy link
Copy Markdown

@czarcas7ic czarcas7ic commented Apr 10, 2026

Summary

  • Switch scan_cached_blocks to use only Scope::External IVKs, halving key-agreement work per output
  • Restructure the sync loop: run() / running() now service enhancement requests between scan chunks via ServiceOutcome (Drained / Stabilized)
  • store_raw_transaction enriches decrypted outputs with nullifiers and tree positions, then calls notify_wallet_note_positions
  • Nullifier pruning is deferred from put_blocks to the sync orchestrator so enhancement can still consult the tracked-nullifier map
  • Carry the minimal SQLite queue semantics this slice depends on: mined transactions with raw are queued as Enhancement (not GetStatus), and transaction_data_requests() is returned in deterministic order for the stabilized-queue detection
  • Restore an explicit safe path for non-sync::run callers by having scan_cached_blocks() prune automatically only when no follow-up transaction-data requests were enqueued, and add focused regressions for both that behavior and the transparent mempool cursor-advance edge case

Context

This is PR 2/5 — the core behavioral change. It depends on PR 1 for the enrichment API, includes the minimum SQLite queue behavior needed for the new sync logic to be correct in isolation, and now also makes the direct-scan_cached_blocks pruning contract explicit for non-sync callers.

Full stack:

  1. adam/enriched-decrypted-output — API foundations
  2. This PR — Core behavioral change
  3. adam/sqlite-enhancement-support — SQLite backend implementation
  4. adam/memory-backend-enhancement — Memory backend implementation
  5. adam/external-ivk-scan-tests — Tests and changelogs

Review focus

  • Drained / Stabilized termination logic in sync.rs
  • store_raw_transaction enrichment and note-position notification flow
  • The deterministic SQLite queue behavior that makes stabilized-request detection and late change recovery valid in this slice
  • The direct-scan pruning contract and the transparent mempool request edge case

Test plan

  • cargo check -p zcash_client_backend --features sync
  • cargo test -p zcash_client_backend --all-features scan_cached_blocks_prunes_when_request_queue_is_empty
  • cargo test -p zcash_client_backend --all-features scan_cached_blocks_defers_pruning_when_requests_are_pending
  • cargo test -p zcash_client_backend --all-features mempool_all_request_does_not_advance_address_checked
  • cargo test -p zcash_client_backend --all-features mined_all_request_advances_address_checked

@czarcas7ic czarcas7ic force-pushed the adam/enriched-decrypted-output branch from d988658 to 1196ffb Compare April 10, 2026 00:08
@czarcas7ic czarcas7ic force-pushed the adam/sync-enhancement-pipeline branch from d64c5f0 to 93600c8 Compare April 10, 2026 00:10
@czarcas7ic czarcas7ic force-pushed the adam/enriched-decrypted-output branch from e980ac7 to 44e9b4e Compare April 10, 2026 00:14
@czarcas7ic czarcas7ic force-pushed the adam/sync-enhancement-pipeline branch from 93600c8 to 2ef0016 Compare April 10, 2026 00:15
@czarcas7ic czarcas7ic force-pushed the adam/enriched-decrypted-output branch from 44e9b4e to 399e05b Compare April 10, 2026 00:25
@czarcas7ic czarcas7ic force-pushed the adam/sync-enhancement-pipeline branch 7 times, most recently from d345f0e to da376b5 Compare April 10, 2026 07:06
greg0x pushed a commit that referenced this pull request Apr 14, 2026
…f0768ea4..dd0ea2c3c5

dd0ea2c3c5 Merge pull request #23 from zcash/2026-03-doc-fixes
93b26b7db1 v0.4.1 release - minor doc fixups
d528fa82e3 Merge pull request #22 from zcash/2025-12-doc-git-subtree
b421ef0b34 Merge pull request #22 from zcash/2025-12-doc-git-subtree
4a26e54c36 Fix git subtree command syntax in README
34bb38b606 Merge pull request #20 from zcash/doc/get_block
a507182f92 Document GetBlock transparent data behavior and clarify nullifier RPCs
fc5cc9a4b1 Merge pull request #19 from zcash/deprecate_get_nullifiers
f0ebc72cad Mark `GetBlockNullifiers` and `GetBlockRangeNullifiers` as deprecated.
9a5f7a0eec Merge pull request #16 from zcash/2026-01-remove-fullHeader
f1095897d9 revert the previous PR that added BlockID.fullHeader
99d9bf9fff Merge pull request #15 from zcash/2025-12-compactblock-doc
bbdd689d73 either CompactBlock.{hash, prevHash}, or CompactBlock.header
aa42926ed3 Merge pull request #13 from pacu/add-README
da4d3d57d8 Add rationale suggested by @LarryRuane
4edd22465b fix type
e36df314b8 clean up whitespace and line width
215cb5a3f3 Add PR suggestions. Organize Clients section
3e78d5d872 Apply suggestions from code review
96b86544db Apply suggestion from @LarryRuane
6259b97df0 Apply suggestion from @nuttycom
f2fcef3e42 Apply suggestion from @nuttycom
8cc1e66b11 Apply suggestion from @LarryRuane
99132222dd Apply suggestion from @nuttycom
207cc5e6f9 create a README file with context and instructions
7392b9706f Merge pull request #12 from zcash/release/v0.4.0

git-subtree-dir: zcash_client_backend/lightwallet-protocol
git-subtree-split: dd0ea2c3c5827a433e62c2f936b89efa2dec5a9a
Restructure the sync loop to service enhancement requests between
scan chunks. This prepares for External-only batch scanning by
ensuring change notes discovered during enhancement are properly
handled.

- Add ServiceOutcome (Drained/Stabilized) for termination logic
- store_raw_transaction enriches outputs with nullifiers and tree
  positions, then calls notify_wallet_note_positions
- Defer nullifier pruning from put_blocks to sync orchestrator
- Add collect_wallet_note_positions to decrypt module
- Add zip-233 feature for TransactionData::from_parts under NU7

Co-Authored-By: Claude <noreply@anthropic.com>
@czarcas7ic czarcas7ic force-pushed the adam/enriched-decrypted-output branch from 0695f91 to a1f3f63 Compare April 15, 2026 17:17
@czarcas7ic czarcas7ic force-pushed the adam/sync-enhancement-pipeline branch from da376b5 to 6b150a3 Compare April 15, 2026 17:17
@czarcas7ic czarcas7ic closed this Apr 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant