zcash_client_sqlite: Add PIR spent-note tracking#9
Closed
p0mvn wants to merge 17 commits into
Closed
Conversation
This was referenced Apr 3, 2026
Nullifier PIR (Private Information Retrieval) lets the wallet discover Orchard note spendability by querying an external PIR server for nullifier inclusion, rather than waiting for sequential shard-tree scanning to complete. This significantly reduces the time before notes become spendable. The implementation is gated behind the `sync-nullifier-pir` feature flag and comprises: - A `pir_spent_notes` table (migration created unconditionally to keep the DAG identical across all builds; empty when the feature is off). - A `wallet::pir` module with queries for unspent notes eligible for PIR checking, pending-spend tracking, and idempotent insert logic. - Feature-gated changes to `get_wallet_summary` and note selection that skip the unscanned-range spendability gate for Orchard notes when PIR is enabled. - Unconditional `pir_spent_notes` cleanup in `truncate_to_height` to prevent stale exclusions from persisting after reorgs.
The PIR tests were using a hand-crafted SQL schema (PIR_TEST_SCHEMA_SQL) that could drift from the actual migration-created tables. Notably, the test schema still used the old `tx` column name while the real schema uses `transaction_id` (renamed in the account_delete_cascade migration). Replace the standalone schema with WalletMigrator::init_or_migrate() so the test database always matches production. The tests insert synthetic prerequisite rows (one account, one transaction) into the fully migrated schema, then exercise the PIR queries against it. Also fixes create_pir_test_db_on_disk (used by the SDK crate for concurrent-access tests) to use the same migration-based setup.
p0mvn
force-pushed
the
sync-pir-0.19.x
branch
3 times, most recently
from
6abc353 to
9910b9d
Compare
Enable Orchard notes to be spent before the wallet finishes scanning by fetching Merkle authentication paths from an external PIR server. When ShardTree witnesses are unavailable (shard incomplete or checkpoints pruned), the transaction builder falls back to PIR-stored witnesses. This adds: - `pir_witness_data` migration and `pir_witness` data layer module - `WalletCommitmentTrees::get_pir_orchard_merkle_path` trait method - PIR witness fallback in `create_proposed_transactions` - Coin selection and wallet summary awareness of PIR-witnessed notes - Comprehensive unit and integration tests Renames the feature flag from `sync-nullifier-pir` to `spendability-pir` since it now covers both nullifier spent-note detection and witness construction.
These imports are only used inside the #[cfg(test)] function migrate_and_setup, but were declared at the outer testing module scope (gated by #[cfg(any(test, feature = "test-dependencies"))]). When clippy runs without the test cfg, the function is excluded but the imports remain visible, triggering -Dunused-imports. Made-with: Cursor
The pir_witness_data migration was added in "Add PIR witness support for Orchard note spendability" but the corresponding TABLE_PIR_WITNESS_DATA constant in db.rs and its entry in the verify_schema expected tables list were omitted, causing the test to fail (29 actual tables vs 28 expected). Made-with: Cursor
Letting replayed witness batches overwrite a newer stored anchor can silently downgrade the wallet's witness state. Refresh the stored row only when the incoming snapshot is at least as recent, and cover both the upgrade and older-snapshot cases in the insert tests. Made-with: Cursor
Keep transaction construction anchored to a consistent Orchard tree state by validating both ShardTree and PIR witnesses against the selected notes before using them. Surface anchor drift as a recoverable mismatch so callers can refresh witnesses instead of building spends from stale paths. Made-with: Cursor
Document that Orchard readiness is decided note-by-note from scan state and PIR witnesses, and add a regression covering mixed witness availability in wallet summary balances. Made-with: Cursor
When nullifier PIR detects that a note has been spent, the SDK trial-decrypts the spending transaction to find change notes. These "provisional" notes are ahead-of-scan hints that give the wallet immediate visibility into the user's actual balance. The pir_provisional_notes table stores the decrypted note data (position, value, diversifier, rseed, rho, nullifier, cmx) along with the spend_height and a has_pir_witness flag that tracks whether a witness has been obtained. Balance integration: provisional notes with a PIR witness add to spendable_value; those without add to value_pending_spendability. Scanner reconciliation: when put_received_note inserts a canonical note at a position that matches a provisional row, the provisional row is deleted. On reorg (truncate_to_height), all provisional notes are cleared unconditionally. Made-with: Cursor
The FFI layer needs to resolve a spent note's account_id and UFVK when discovering change notes. Rather than exposing the raw connection, provide get_account_for_orchard_note on WalletDb which returns the internal row ID and AccountUuid for the note's account. Also simplifies insert_pir_provisional_note to take the internal account_id directly, matching the DB column type. Made-with: Cursor
Public but never read anywhere — dead code that violates CONTRIBUTING.md's requirement that public API members be part of the actual API. The struct can be reintroduced if a read query for provisional notes is needed later. Made-with: Cursor
The depth-1 change discovery only tracks immediate change outputs from a spent note. In practice, change notes themselves may be spent in subsequent transactions, creating multi-hop chains (A -> B -> C -> ...) that must be fully resolved to determine the wallet's spendable balance. Add a migration with depth, parent_provisional_id, pir_checked, is_spent, and discovered_by_scanner columns to pir_provisional_notes. Replace the delete-on-reconcile strategy with a discovered_by_scanner flag so the recursive chain persists when the canonical scanner catches up, avoiding wasteful re-PIRing of already-known chains. The balance query now excludes mid-chain spent notes and scanner-reconciled notes, counting only active leaf nodes. When reconciliation finds a provisional note that was PIR-marked as spent, it propagates that status to pir_spent_notes for the canonical note ID. Made-with: Cursor
The three separate PIR tables (pir_spent_notes, pir_witness_data, pir_provisional_notes) fragmented note lifecycle data, requiring cross-table joins and duplicated logic for spend-checking, witness storage, and balance calculation. A single pir_notes table now tracks the full PIR lifecycle: canonical-note linkage, spend state, witness data, recursive change-discovery chain, and scanner reconciliation. This replaces four migrations with one, since the feature is not yet live and no production databases exist with the old schema. The storage functions (pir.rs, pir_provisional.rs, pir_witness.rs) are rewritten against the unified table, and the balance/coin-selection queries in common.rs and wallet.rs use EXISTS subqueries instead of LEFT JOINs to avoid column-name ambiguity. Made-with: Cursor
Provisional notes discovered by PIR trial decryption need Merkle witnesses fetched separately since they aren't in the canonical scan's note table. This adds the database query and WalletDb method to identify those notes, along with unit tests. Made-with: Cursor
The three modules (pir.rs, pir_witness.rs, pir_provisional.rs) all operated on the same pir_notes table but were split by an artificial entity distinction. "Provisional" is a row state (canonical_note_id IS NULL), not a separate data store, and witness writes were duplicated across two modules with slightly different SQL. Test infrastructure was also split arbitrarily, with cross-module imports between them. Merge everything into wallet/pir.rs organized by logical sections (spend tracking, witness storage, Merkle path construction, provisional lifecycle), with unified test helpers and a single test suite. Made-with: Cursor
The old get_pir_pending_spends query returned gross note values, so the activity view showed the full amount of each spent note even when most of it came back as change. Replace it with get_pir_activity_entries, a recursive CTE that walks the provisional note tree to compute net spend (gross minus unspent change descendants) per spending transaction. Schema: add spending_tx_hash, spending_block_time, and spending_fee columns to pir_notes so change discovery can record which transaction spent each note and when. This lets the UI build real "Sent" entries (with correct amounts, timestamps, and fees) that seamlessly transition to scanner-confirmed rows once the scanner catches up, since both use the same txid as their identity. Remove PirPendingSpend/PirPendingSpendsResult and their query — the activity entries subsume them entirely. The balance-gating role of pending spends is already handled by spent_notes_clause, which reads pir_notes.is_spent directly. Made-with: Cursor
Point test-only dependencies (commitment-tree-db, pir-types, witness-server) at the latest spendability-pir main which includes extended nullifier entries, witness window bootstrap fix, and decryption-types crate. Switch from path to git deps so the PR is buildable without sibling repos. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PIR spent-note tracking onto the
maint/zcash_client_sqlite-0.19.xrelease branch.sync-mainis made from the following release branch:https://github.com/valargroup/librustzcash/compare/maint/zcash_client_sqlite-0.19.x...valargroup:librustzcash:sync-main?expand=1
Summary
Adds support for nullifier PIR (Private Information Retrieval) spent-note tracking to
zcash_client_sqlite, gated behind async-nullifier-pirfeature flag.Nullifier PIR lets the wallet discover Orchard note spendability by querying an external PIR server for nullifier inclusion, rather than waiting for sequential shard-tree scanning to complete.
What's included
pir_spent_notesmigration: Created unconditionally (not feature-gated) to keep the migration DAG identical across all builds.wallet::pirmodule (sync-nullifier-pirfeature): Queries for unspent notes eligible for PIR checking, pending-spend tracking, and idempotent spent-note insertion.sync-nullifier-pirfeature):get_wallet_summaryand note selection skip the unscanned-range spendability gate for Orchard notes.truncate_to_heightunconditionally clearspir_spent_notes.All Relevant PRs
librustzcash(upstream main): zcash_client_sqlite: Add PIR spent-note tracking zcash/librustzcash#2267zcash-swift-wallet-sdk: zcash-swift-wallet-sdk: Nullifier PIR integration zcash-swift-wallet-sdk#12Made with Cursor