diff --git a/docs/book/src/count-indexed-tree.md b/docs/book/src/count-indexed-tree.md index 56564f548..0d94aa09e 100644 --- a/docs/book/src/count-indexed-tree.md +++ b/docs/book/src/count-indexed-tree.md @@ -152,19 +152,51 @@ The secondary Merk holds one entry per element in the primary, keyed by: ```text secondary_key = count_be_bytes(8) ‖ original_key -secondary_val = () // empty; the original_key is encoded in the key +secondary_val = ReferenceWithSumItem( + SiblingReference(original_key), + max_reference_hop = Some(1), + sum = count_value, + ) ``` - **`count_be_bytes`** is the element's `count_value` encoded big-endian, 8 bytes. Big-endian gives natural numeric order under lexicographic comparison, so right-to-left iteration yields highest-count-first. - **`original_key`** is appended to break ties among elements with equal - counts and to make each secondary key unique and reversible. + counts and to make each secondary key unique and reversible. It stays in the + key because ordering, uniqueness, and boundary decoding must not depend on + resolving the row value. -The secondary Merk uses node feature type `ProvableCountedMerkNode(1)` — -every entry contributes a count of `1`, so the aggregated count at the -secondary's root equals the total number of indexed entries (which also -equals the number of entries in the primary). +#### Canonical reference rows + +Every secondary row is a canonical one-hop combined reference back to its +primary entry. Its committed value hash is: + +```text +combine_hash(H(reference bytes), primary_node_committed_value_hash) +``` + +The binding is deliberately to the immediate primary node, not directly to a +terminal value reached through another reference. This keeps refresh local: +every operation that rewrites the primary entry also refreshes its rows. Reads +then apply ordinary GroveDB reference semantics and return the terminal value. +This immediate binding is dedicated indexed-tree behavior; ordinary user +references retain their existing terminal-reference rules. + +Because the row binds the primary commitment, value-only updates and deep +subtree-root changes rewrite the row even when its ordering aggregates stay +unchanged. This write amplification is intentional and included in cost +tracking. + +`SiblingReference` keeps row size independent of grove depth. The secondary's +physical prefix (`blake3(primary_prefix ‖ axis_tag)`) is not a GroveDB path, so +the row is interpreted with the indexed primary as its purpose-built logical +origin rather than by manufacturing a fake `SubtreePath`. + +The secondary Merk uses node feature type +`ProvableCountedAndProvableSummedMerkNode(1, count_value)`. Every row therefore +contributes count `1`, while its carried sum preserves the count band's total +as an authenticated scalar. The reason the secondary is a *provable* count tree (rather than the simpler `BasicMerkNode`) is that this lets the existing @@ -426,8 +458,8 @@ Merk is not touched. The verifier receives the primary's root hash plus a ### Top-k by count ```rust -// Shipped API on `GroveDb`: -let entries: Vec<(u64, Vec)> = db +// Public API on `GroveDb`: +let entries: Vec> = db .indexed_count_top_k(path, k, /* descending: */ true, transaction, grove_version)? .expect("top-k"); @@ -436,14 +468,11 @@ let proof_bytes = db .prove_indexed_count_top_k(path, k, /* descending: */ true, transaction, grove_version)? .expect("prove"); let result = GroveDb::verify_indexed_count_top_k(&proof_bytes, &path, k)?; -// result.entries: Vec<(u64, Vec)>, result.root_hash: [u8; 32] +// result.entries: Vec>, result.root_hash: [u8; 32] ``` -The query returns `(count, key)` pairs. To resolve a primary value the -caller follows up with `db.get(path, key, ...)`; the dedicated proof -shape carries only the secondary range proof + a 32-byte attestation -of the primary's root hash. Workloads that don't need values -(leaderboards, ranking views) pay nothing for data they wouldn't read. +Each entry contains `ordering_value`, `primary_key`, and the resolved terminal +`Element`; callers do not perform a second primary lookup per result. Internally: @@ -453,19 +482,24 @@ Internally: 3. Run a **descending range query** with `limit = k` over the full secondary keyspace. This yields the k highest-count entries, with a standard Merk range proof. -4. *(only if `resolve_values: true`)* For each `(c_be ‖ k)` in the - result, open the **primary** Merk and query for `k`. Each resolution - is one extra Merk read with one extra Merk inclusion proof. - -The default keeps the proof minimal: secondary range proof + a 32-byte -attestation of the primary's root hash. Workloads that don't need the -values (leaderboards, ranking views, "top N usernames") pay nothing for -data they wouldn't read. +4. Resolve each row and attach a compact target-shape witness. The verifier + reconstructs the immediate primary commitment directly from the target + bytes and shape data, then checks it against the hash already committed by + the secondary row. Reference-shaped primaries carry a bounded chain to the + terminal value; nodes outside the indexed primary retain ordinary root + authentication. + +For ordinary direct primary values, the target witness does not repeat a +GroveDB inclusion proof per row. The canonical row's combined-reference hash is +the authentication anchor, so proof growth is the resolved value plus its shape +commitment rather than another root-to-primary path. A primary that is itself a +reference pays for authentication only after its chain leaves that immediate +row binding. ### Range by count ```rust -let entries: Vec<(u64, Vec)> = db +let entries: Vec> = db .indexed_count_range( path, min, // u64, inclusive @@ -601,14 +635,18 @@ existing GroveDB layer proofs, with these additions: graph TD L0["Layer proof: root → … → CountIndexedTree element
standard, unchanged"] EL["Element bytes: (primary_root_key, secondary_root_key, count_value, flags)
actual_value_hash = Blake3(varint(len) || element_bytes)"] - L1A["Primary Merk proof
only if primary values were touched"] + PR["Primary root hash attestation"] L1B["Secondary Merk range proof
over (count_be ‖ key) keys"] + TW["Per-row compact target witness
value bytes + commitment shape; root proofs only after reference hops"] + ROW["Canonical secondary row
binds H(reference bytes) + immediate primary commitment"] COMB["combined_value_hash = Blake3(actual_value_hash || primary_root_hash || secondary_root_hash)
order is primary, then secondary"] L0 --> EL - EL --> L1A + EL --> PR EL --> L1B - L1A --> COMB + L1B --> ROW + TW --> ROW + PR --> COMB L1B --> COMB ``` @@ -616,16 +654,19 @@ Verifier obligations: - Parent layer verifies the element bytes (carrying both root keys) up to the GroveDB root. -- Each Merk proof produces its own root hash (`primary_root_hash` and/or - `secondary_root_hash`). +- The secondary Merk proof produces `secondary_root_hash`; the envelope carries + the untouched `primary_root_hash` attestation already committed by the outer + indexed-tree element. +- Every returned row is checked for its canonical reference bytes and binds the + immediate primary commitment reconstructed from its target witness. - The verifier reconstructs `combined_value_hash` from `actual_value_hash`, `primary_root_hash`, `secondary_root_hash` (in that order) and checks it matches the value hash committed in the parent layer. -Both root hashes must be made available to the verifier — when a query -touches only one of the two trees, the proof carries the *other* tree's -root hash as a 32-byte attestation (it is hashed but not traversed). +Both root hashes must be available to the verifier. The primary root is hashed +but not traversed for direct rows; the secondary row itself authenticates their +immediate primary commitments. ## When to use which element type diff --git a/grovedb-element/src/element/mod.rs b/grovedb-element/src/element/mod.rs index 3fc5e9022..7872bc463 100644 --- a/grovedb-element/src/element/mod.rs +++ b/grovedb-element/src/element/mod.rs @@ -293,11 +293,13 @@ pub enum Element { /// `ProvableCountedAndProvableSummedMerkNode` (both count AND sum /// baked into node hash) and carries a TLV list of 1..=3 secondary /// Merks — one per selected axis (count, sum, avg). Each secondary - /// lives at its own derived storage prefix; the count axis is a - /// `ProvableCountTree` while the sum and avg axes are - /// `ProvableCountProvableSumTree`s, so every axis carries a - /// hash-bound count (enabling count-bound offset pagination) and - /// the sum/avg axes can additionally produce sum-on-range proofs. + /// lives at its own derived storage prefix and is a + /// `ProvableCountProvableSumTree`, so every axis carries a + /// hash-bound count (enabling count-bound offset pagination). Its + /// canonical rows are `ReferenceWithSumItem` values that point to + /// the corresponding primary key and bind the immediate primary + /// node's committed value hash; sum/avg axes can additionally + /// produce sum-on-range proofs. /// /// Fields: `(primary_root_key, count_value, sum_value, axes, flags)` /// - `primary_root_key`: root key of the primary diff --git a/grovedb-element/src/indexed/mod.rs b/grovedb-element/src/indexed/mod.rs index 28f31461a..8ff5ea428 100644 --- a/grovedb-element/src/indexed/mod.rs +++ b/grovedb-element/src/indexed/mod.rs @@ -17,7 +17,7 @@ pub use sort_keys::{ encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key, AVG_FIXED_POINT_SCALE, }; -use crate::error::ElementError; +use crate::{error::ElementError, reference_path::ReferencePathType, Element}; /// Axis tag for a `ProvableCountProvableSumIndexedTree` secondary entry. /// @@ -50,6 +50,32 @@ pub type IndexedTreeAxisEntry = (u8, Option>); /// encoding. pub type IndexedTreeAxes = Vec; +/// Build the canonical secondary row for an indexed-tree axis. +/// +/// The row is always a one-hop sibling reference to the primary key. Its +/// explicit sum preserves the secondary's dual count/sum aggregates. +pub fn canonical_axis_reference( + axis: IndexAxis, + primary_key: &[u8], + count: u64, + sum: i64, +) -> Result { + let axis_sum = match axis { + IndexAxis::Count => i64::try_from(count).map_err(|_| { + ElementError::CorruptedData(format!( + "count value {count} exceeds i64::MAX and cannot be mirrored into an indexed \ + count-axis secondary" + )) + })?, + IndexAxis::Sum | IndexAxis::Avg => sum, + }; + Ok(Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(primary_key.to_vec()), + Some(1), + axis_sum, + )) +} + #[cfg(test)] mod tests { use super::*; diff --git a/grovedb-element/src/lib.rs b/grovedb-element/src/lib.rs index e009b60ef..20b924741 100644 --- a/grovedb-element/src/lib.rs +++ b/grovedb-element/src/lib.rs @@ -10,7 +10,7 @@ pub mod reference_path; pub(crate) mod visualize_helpers; pub use indexed::{ - compute_avg_fixed_point, decode_avg_sort_key, decode_count_sort_key, decode_sum_sort_key, - encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key, IndexAxis, IndexedTreeAxes, - IndexedTreeAxisEntry, AVG_FIXED_POINT_SCALE, + canonical_axis_reference, compute_avg_fixed_point, decode_avg_sort_key, decode_count_sort_key, + decode_sum_sort_key, encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key, + IndexAxis, IndexedTreeAxes, IndexedTreeAxisEntry, AVG_FIXED_POINT_SCALE, }; diff --git a/grovedb/src/batch/estimated_costs/average_case_costs.rs b/grovedb/src/batch/estimated_costs/average_case_costs.rs index d5f1b7c88..63c0dd3e2 100644 --- a/grovedb/src/batch/estimated_costs/average_case_costs.rs +++ b/grovedb/src/batch/estimated_costs/average_case_costs.rs @@ -2343,7 +2343,7 @@ mod tests { } /// The ≥-actual contract must hold for the MULTI-axis variant, not just - /// PCIT. The sum and avg axes' secondary rows (`SumItem`, + /// PCIT. The sum and avg axes' secondary rows (`ReferenceWithSumItem`, /// `ItemWithSumItem`) are larger than the count axis's empty `Item`; /// sizing all three as an empty item put the PCPSIT estimate ~25 bytes /// per key UNDER actual `added_bytes` — and the PCIT-only test above diff --git a/grovedb/src/batch/indexed_tree/mirror.rs b/grovedb/src/batch/indexed_tree/mirror.rs index 50cfab09a..e8afd1786 100644 --- a/grovedb/src/batch/indexed_tree/mirror.rs +++ b/grovedb/src/batch/indexed_tree/mirror.rs @@ -22,9 +22,9 @@ use grovedb_merk::{ use grovedb_storage::StorageContext; use grovedb_version::version::GroveVersion; -use super::{read_entry_aggregates, AggregatePair, AggregateTransition}; +use super::{read_entry_state, AggregateTransition, MaybeEntryState}; use crate::{ - operations::indexed_tree::{axis_row_payload, make_axis_secondary_key}, + operations::indexed_tree::{axis_row_reference, make_axis_secondary_key}, Element, Error, }; @@ -63,17 +63,17 @@ fn enforce_axis_item_key_bound<'a>( /// what makes each axis's assembled batch deterministic. pub(crate) fn read_post_apply_transitions<'db, S: StorageContext<'db>>( primary_merk: &Merk, - pre: &BTreeMap, AggregatePair>, + pre: &BTreeMap, MaybeEntryState>, grove_version: &GroveVersion, ) -> CostResult, Error> { let mut cost = OperationCost::default(); let mut transitions: Vec = Vec::with_capacity(pre.len()); - for (key, old_aggregates) in pre { - let new_aggregates = cost_return_on_error!( + for (key, old_state) in pre { + let new_state = cost_return_on_error!( &mut cost, - read_entry_aggregates(primary_merk, key, "post", grove_version) + read_entry_state(primary_merk, key, "post", grove_version) ); - transitions.push((key.clone(), *old_aggregates, new_aggregates)); + transitions.push((key.clone(), *old_state, new_state)); } Ok(transitions).wrap_with_cost(cost) } @@ -92,22 +92,22 @@ fn build_axis_mirror_batch( let mut cost = OperationCost::default(); let secondary_tree_type = crate::operations::indexed_tree::axis_secondary_tree_type(axis); let mut secondary_batch: Vec>> = Vec::with_capacity(transitions.len() * 2); - for (key, old_aggregates, new_aggregates) in transitions { - let entry_for = |aggregates: &AggregatePair| -> Result<_, Error> { - aggregates - .map(|(c, s)| { - Ok(( - make_axis_secondary_key(axis, c, s, key), - axis_row_payload(axis, c, s)?, - )) - }) - .transpose() - }; - let old_entry = cost_return_on_error_no_add!(cost, entry_for(old_aggregates)); - let new_entry = cost_return_on_error_no_add!(cost, entry_for(new_aggregates)); - if old_entry == new_entry { + for (key, old_state, new_state) in transitions { + if old_state == new_state { continue; } + let old_key = + old_state.map(|state| make_axis_secondary_key(axis, state.count, state.sum, key)); + let new_entry = new_state + .map(|state| { + Ok(( + make_axis_secondary_key(axis, state.count, state.sum, key), + axis_row_reference(axis, key, state.count, state.sum)?, + state.value_hash, + )) + }) + .transpose(); + let new_entry = cost_return_on_error_no_add!(cost, new_entry); // Delete the old row ONLY if the new one lands on a different key. // On the avg axis a change can alter the payload while leaving the // sort key fixed — (count, sum) going (1, 5) -> (2, 10) keeps @@ -115,8 +115,8 @@ fn build_axis_mirror_batch( // Merk batch is rejected outright ("Keys in batch must be unique"), // failing the whole GroveDB batch. Where the key is unchanged the put // alone overwrites the payload, which is what the row needs. - let new_secondary_key_ref = new_entry.as_ref().map(|(key, _)| key); - if let Some((old_secondary_key, _)) = &old_entry + let new_secondary_key_ref = new_entry.as_ref().map(|(key, ..)| key); + if let Some(old_secondary_key) = &old_key && Some(old_secondary_key) != new_secondary_key_ref { cost_return_on_error!( @@ -131,7 +131,7 @@ fn build_axis_mirror_batch( .map_err(Error::MerkError) ); } - if let Some((new_secondary_key, entry)) = new_entry { + if let Some((new_secondary_key, entry, target_hash)) = new_entry { let feature_type = cost_return_on_error_no_add!( cost, entry @@ -141,8 +141,9 @@ fn build_axis_mirror_batch( cost_return_on_error!( &mut cost, entry - .insert_into_batch_operations( + .insert_reference_into_batch_operations( new_secondary_key, + target_hash, &mut secondary_batch, feature_type, grove_version, diff --git a/grovedb/src/batch/indexed_tree/mod.rs b/grovedb/src/batch/indexed_tree/mod.rs index fdccd0929..f9c29a00a 100644 --- a/grovedb/src/batch/indexed_tree/mod.rs +++ b/grovedb/src/batch/indexed_tree/mod.rs @@ -31,7 +31,8 @@ //! - [`pre_state`] runs against an indexed primary's level just before the //! merk apply: [`capture_indexed_pre_state`] validates the level's ops //! against the indexed-primary rules and reads each mutated key's *old* -//! `(count, sum)` pair so the mirror can compute old → new transitions. +//! `(count, sum, value_hash)` state so the mirror can compute exact old → new +//! transitions. //! - [`mirror`] runs once per configured axis after the primary merk's //! `apply_with_specialized_costs` returns: //! [`apply_indexed_secondary_mirror_post_apply`] re-reads each captured @@ -41,9 +42,9 @@ //! parent's H1-A composition — directly for the single-axis variants, //! through `axes_digest` for PCPSIT. //! -//! What lives in this file is what more than one phase needs: the aggregate -//! type aliases and [`read_entry_aggregates`], the primary-entry read that -//! the capture ("pre") and the mirror ("post") both perform. +//! What lives in this file is what more than one phase needs: the entry-state +//! type aliases and [`read_entry_state`], the primary-entry read that the +//! capture ("pre") and the mirror ("post") both perform. mod delete_tree; mod mirror; @@ -55,7 +56,7 @@ pub(crate) use delete_tree::validate_delete_tree_type; use grovedb_costs::{ cost_return_on_error, cost_return_on_error_no_add, CostResult, CostsExt, OperationCost, }; -use grovedb_merk::{element::costs::ElementCostExtensions, Merk}; +use grovedb_merk::{element::costs::ElementCostExtensions, CryptoHash, Merk}; use grovedb_storage::StorageContext; use grovedb_version::version::GroveVersion; pub(crate) use mirror::{apply_indexed_secondary_mirror_post_apply, read_post_apply_transitions}; @@ -65,27 +66,34 @@ pub(crate) use preflight::reject_indexed_overwrite_with_descendants; use crate::{Element, Error}; -/// A primary entry's `(count, sum)` aggregate pair, `None` when the entry -/// does not exist on that side of the transition. -type AggregatePair = Option<(u64, i64)>; +/// Complete state that decides one indexed secondary row. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) struct IndexedEntryState { + count: u64, + sum: i64, + value_hash: CryptoHash, +} + +/// A primary entry's state, `None` when the entry does not exist on that side +/// of the transition. +pub(super) type MaybeEntryState = Option; -/// One captured key's aggregate transition: `(item_key, old, new)`. -type AggregateTransition = (Vec, AggregatePair, AggregatePair); +/// One captured key's transition: `(item_key, old state, new state)`. +type AggregateTransition = (Vec, MaybeEntryState, MaybeEntryState); -/// Read one primary entry's current `(count, sum)` pair, `None` if the key -/// does not exist. `phase` labels error messages ("pre" for the capture -/// before the primary apply, "post" for the delta read after it). -fn read_entry_aggregates<'db, S: StorageContext<'db>>( +/// Read one primary entry's current aggregate and commitment state. `phase` +/// labels error messages ("pre" before the primary apply, "post" after it). +fn read_entry_state<'db, S: StorageContext<'db>>( primary_merk: &Merk, key: &[u8], phase: &str, grove_version: &GroveVersion, -) -> CostResult { +) -> CostResult { let mut cost = OperationCost::default(); - let maybe_bytes = cost_return_on_error!( + let maybe = cost_return_on_error!( &mut cost, primary_merk - .get( + .get_value_and_value_hash( key, true, Some(&Element::value_defined_cost_for_serialized_value), @@ -96,16 +104,21 @@ fn read_entry_aggregates<'db, S: StorageContext<'db>>( hex::encode(key) ))) ); - let aggregates = if let Some(bytes) = maybe_bytes { + let state = if let Some((bytes, value_hash)) = maybe { let elem = cost_return_on_error_no_add!( cost, Element::deserialize(bytes.as_slice(), grove_version).map_err(|e| { Error::CorruptedData(format!("indexed {phase}-state deserialize: {e}")) }) ); - Some(elem.count_sum_value_or_default()) + let (count, sum) = elem.count_sum_value_or_default(); + Some(IndexedEntryState { + count, + sum, + value_hash, + }) } else { None }; - Ok(aggregates).wrap_with_cost(cost) + Ok(state).wrap_with_cost(cost) } diff --git a/grovedb/src/batch/indexed_tree/pre_state.rs b/grovedb/src/batch/indexed_tree/pre_state.rs index 2f342d8cd..4f62dfb2e 100644 --- a/grovedb/src/batch/indexed_tree/pre_state.rs +++ b/grovedb/src/batch/indexed_tree/pre_state.rs @@ -14,7 +14,7 @@ use grovedb_merk::{element::insert::ElementInsertToStorageExtensions, Merk}; use grovedb_storage::StorageContext; use grovedb_version::version::GroveVersion; -use super::{read_entry_aggregates, AggregatePair}; +use super::{read_entry_state, MaybeEntryState}; use crate::{ batch::{GroveOp, KeyInfo}, operations::indexed_tree::MAX_CIDX_ITEM_KEY_LEN, @@ -150,13 +150,15 @@ fn validate_indexed_child_ops( /// the avg axis derives its sort key from the pair, and a PCPSIT can index /// count, sum and avg simultaneously. /// -/// Only ops whose `can_mutate_child_count()` is true are captured — -/// non-count-mutating ops (e.g., `CommitmentTreeInsert`) are skipped. +/// Every op that can rewrite the entry is captured — see +/// `GroveOp::can_mutate_indexed_secondary_row`. This is intentionally wider +/// than count mutation because a canonical row also binds the primary node's +/// committed value hash. pub(crate) fn capture_indexed_pre_state<'db, S: StorageContext<'db>>( primary_merk: &Merk, ops_at_path_by_key: &BTreeMap, grove_version: &GroveVersion, -) -> CostResult, AggregatePair>, Error> { +) -> CostResult, MaybeEntryState>, Error> { let mut cost = OperationCost::default(); cost_return_on_error_no_add!(cost, enforce_indexed_item_key_ceiling(ops_at_path_by_key)); @@ -165,20 +167,20 @@ pub(crate) fn capture_indexed_pre_state<'db, S: StorageContext<'db>>( validate_indexed_child_ops(ops_at_path_by_key, primary_merk.tree_type) ); - let mut pre: BTreeMap, AggregatePair> = BTreeMap::new(); + let mut pre: BTreeMap, MaybeEntryState> = BTreeMap::new(); for (key_info, op) in ops_at_path_by_key.iter() { let key_bytes = key_info.get_key_clone(); - // Single source of truth: `GroveOp::can_mutate_child_count` + // Single source of truth: `GroveOp::can_mutate_indexed_secondary_row` // uses an exhaustive match so adding a new variant forces // explicit classification at the type-system level. This is // the structural guard against the nested-cidx bug class // (commit a8bb34fb). - if op.can_mutate_child_count() && !pre.contains_key(&key_bytes) { - let old_aggregates = cost_return_on_error!( + if op.can_mutate_indexed_secondary_row() && !pre.contains_key(&key_bytes) { + let old_state = cost_return_on_error!( &mut cost, - read_entry_aggregates(primary_merk, &key_bytes, "pre", grove_version) + read_entry_state(primary_merk, &key_bytes, "pre", grove_version) ); - pre.insert(key_bytes, old_aggregates); + pre.insert(key_bytes, old_state); } } Ok(pre).wrap_with_cost(cost) diff --git a/grovedb/src/batch/mod.rs b/grovedb/src/batch/mod.rs index e01228bd3..cf9f3b8fd 100644 --- a/grovedb/src/batch/mod.rs +++ b/grovedb/src/batch/mod.rs @@ -577,6 +577,39 @@ impl GroveOp { | GroveOp::DenseTreeInsert { .. } => false, } } + + /// Whether this operation can change an indexed primary entry's + /// canonical secondary row. + /// + /// This is wider than [`Self::can_mutate_child_count`]: the row binds the + /// primary node's committed value hash as well as its aggregates, so any + /// operation that rewrites the entry must participate in pre/post state + /// comparison. The four direct non-Merk append variants are converted to + /// `ReplaceNonMerkTreeRoot` before indexed pre-state capture; listing them + /// here keeps the classification exhaustive and makes future pipeline + /// changes safe rather than silently excluding them. + pub(crate) fn can_mutate_indexed_secondary_row(&self) -> bool { + match self { + GroveOp::InsertWithKnownToNotAlreadyExist { .. } + | GroveOp::InsertIfNotExists { .. } + | GroveOp::InsertOrReplace { .. } + | GroveOp::Replace { .. } + | GroveOp::Patch { .. } + | GroveOp::Delete + | GroveOp::DeleteTree(..) + | GroveOp::RefreshReference { .. } + | GroveOp::ReplaceTreeRootKey { .. } + | GroveOp::InsertTreeWithRootHash { .. } + | GroveOp::ReplaceNonMerkTreeRoot { .. } + | GroveOp::InsertNonMerkTree { .. } + | GroveOp::ReplaceAggregateIndexedTreeRootKeys { .. } + | GroveOp::InsertAggregateIndexedTreeRootKeys { .. } + | GroveOp::CommitmentTreeInsert { .. } + | GroveOp::MmrTreeAppend { .. } + | GroveOp::BulkAppend { .. } + | GroveOp::DenseTreeInsert { .. } => true, + } + } } impl PartialOrd for GroveOp { @@ -2386,17 +2419,20 @@ where // primary level represent a child subtree's bubble-up — the // child's element bytes have a new aggregate count, so its // secondary entry needs to move; we capture it here too. - let indexed_pre_state: Option, Option<(u64, i64)>>> = if in_tree_type - .is_indexed_primary() - { - let merk = self.merks.get(path).expect("the Merk is cached"); - Some(cost_return_on_error!( - &mut cost, - indexed_tree::capture_indexed_pre_state(merk, &ops_at_path_by_key, grove_version,) - )) - } else { - None - }; + let indexed_pre_state: Option, indexed_tree::MaybeEntryState>> = + if in_tree_type.is_indexed_primary() { + let merk = self.merks.get(path).expect("the Merk is cached"); + Some(cost_return_on_error!( + &mut cost, + indexed_tree::capture_indexed_pre_state( + merk, + &ops_at_path_by_key, + grove_version, + ) + )) + } else { + None + }; // V4 gates: keys whose ops need the OLD element they displace. The // merk apply surfaces those bytes for free through the old-value diff --git a/grovedb/src/estimated_costs/average_case_costs.rs b/grovedb/src/estimated_costs/average_case_costs.rs index 4182a717d..3afa6301b 100644 --- a/grovedb/src/estimated_costs/average_case_costs.rs +++ b/grovedb/src/estimated_costs/average_case_costs.rs @@ -31,12 +31,6 @@ use crate::{ Element, ElementFlags, Error, GroveDb, }; -/// Upper bound on an indexed secondary row's value. The payload is fixed -/// per axis — a `SumItem` (count and sum axes) or an empty -/// `ItemWithSumItem` (avg) — and the largest of those serializes well under -/// this bound, which also leaves room for the feature type and flags byte. -pub const INDEXED_SECONDARY_MAX_VALUE_SIZE: u32 = 16; - impl GroveDb { /// Add average case for getting a merk tree pub fn add_average_case_get_merk_at_path<'db, S: Storage<'db>>( @@ -481,10 +475,8 @@ impl GroveDb { /// sparse or conditional indexing). /// - **Key size** is the primary's key size plus the axis sort-key width /// (8 bytes for count/sum, 16 for avg). - /// - **Value size** is bounded by the fixed per-axis payload shape: - /// a `SumItem` (count and sum axes), or an empty - /// `ItemWithSumItem` (avg) — all under - /// [`INDEXED_SECONDARY_MAX_VALUE_SIZE`]. + /// - **Value size** is the canonical one-hop sibling + /// `ReferenceWithSumItem`, including the primary key bytes. /// - **Tree type** is fixed per axis. /// /// Each axis is charged one Merk open, one delete of the old row and one @@ -508,13 +500,30 @@ impl GroveDb { for axis in axes { let secondary_key_size = (primary_key_size + axis_sort_key_len(*axis) as u32).min(u8::MAX as u32) as u8; + let estimated_primary_key = vec![0u8; primary_key_size as usize]; + let worst_case_row = cost_return_on_error_no_add!( + cost, + crate::operations::indexed_tree::axis_row_reference( + *axis, + &estimated_primary_key, + i64::MAX as u64, + i64::MAX, + ) + ); + let secondary_value_size = cost_return_on_error_no_add!( + cost, + worst_case_row + .serialized_size(grove_version) + .map(|size| size as u32) + .map_err(Error::ElementError) + ); let secondary_layer = EstimatedLayerInformation { tree_type: axis_secondary_tree_type(*axis), // 1:1 with the primary. estimated_layer_count: primary_layer_information.estimated_layer_count, - estimated_layer_sizes: EstimatedLayerSizes::AllItems( + estimated_layer_sizes: EstimatedLayerSizes::AllReferencesWithSumItem( secondary_key_size, - INDEXED_SECONDARY_MAX_VALUE_SIZE, + secondary_value_size, None, ), }; @@ -540,21 +549,8 @@ impl GroveDb { // The mirror is delete-old-row then insert-new-row, each of which // rebalances and re-roots the secondary. // - // The inserted row must be sized with the AXIS's real payload - // shape: `average_case_merk_insert_element` charges non-tree - // elements by their own serialized size, and under-sizing put - // the PCPSIT estimate ~25 bytes per key UNDER actual - // `added_bytes` — the one dimension a storage-fee reservation - // cannot come in under. The shape comes from THE payload - // function the mirror writes with (`axis_row_payload`), fed - // worst-case aggregates: sum values are charged at their fixed - // worst-case varint width, so `i64::MAX` is the upper bound, - // not an average (and is in `count_value_as_sum`'s domain, so - // the conversion cannot fail here). - let worst_case_row = cost_return_on_error_no_add!( - cost, - crate::operations::indexed_tree::axis_row_payload(*axis, i64::MAX as u64, i64::MAX,) - ); + // The inserted row is the same canonical reference shape the + // mirror writes, including the estimated primary-key bytes. cost_return_on_error!( &mut cost, Self::average_case_merk_delete_element( diff --git a/grovedb/src/lib.rs b/grovedb/src/lib.rs index a035fd440..e61470a22 100644 --- a/grovedb/src/lib.rs +++ b/grovedb/src/lib.rs @@ -259,6 +259,8 @@ pub use query::{ GroveTrunkQueryResult, LeafInfo, PathBranchChunkQuery, PathQuery, PathQueryShape, PathTrunkChunkQuery, SizedQuery, }; +#[cfg(any(feature = "minimal", feature = "verify"))] +pub use query_result_type::IndexedAxisEntry; #[cfg(feature = "minimal")] use reference_path::path_from_reference_path_type; #[cfg(feature = "grovedbg")] @@ -730,6 +732,27 @@ impl GroveDb { merk_cache, path, None, + None, + transaction, + batch, + grove_version, + ) + } + + fn propagate_changes_with_transaction_refreshing_indexed_row<'b, B: AsRef<[u8]>>( + &self, + merk_cache: HashMap, Merk>, + path: SubtreePath<'b, B>, + changed_key: &[u8], + transaction: &Transaction, + batch: &StorageBatch, + grove_version: &GroveVersion, + ) -> CostResult<(), Error> { + self.propagate_changes_with_transaction_with_initial_deferred( + merk_cache, + path, + None, + Some(changed_key), transaction, batch, grove_version, @@ -741,6 +764,7 @@ impl GroveDb { mut merk_cache: HashMap, Merk>, path: SubtreePath<'b, B>, initial_deferred_secondary: Option<(Hash, Option>)>, + changed_indexed_primary_key: Option<&[u8]>, transaction: &Transaction, batch: &StorageBatch, grove_version: &GroveVersion, @@ -797,6 +821,119 @@ impl GroveDb { // over the old state. let mut deferred_axes: Option>)>> = None; + // Typed in-place mutations of a child under an indexed primary can + // leave its ordering aggregates unchanged while changing its committed + // value hash (MMR/dense/bulk/commitment roots are the common case). + // Refresh that one canonical secondary reference before propagating the + // indexed element's roots upward. + if let Some(changed_key) = changed_indexed_primary_key + && child_tree.tree_type.is_indexed_primary() + { + let (container_path, indexed_key) = cost_return_on_error_no_add!( + cost, + current_path + .derive_parent() + .ok_or(Error::CorruptedCodeExecution( + "an indexed primary requires a parent holding its indexed element", + )) + ); + let container = cost_return_on_error!( + &mut cost, + self.open_transactional_merk_at_path( + container_path, + transaction, + Some(batch), + grove_version, + ) + ); + let indexed_element = cost_return_on_error!( + &mut cost, + Element::get(&container, indexed_key, true, grove_version) + .map_err(Error::MerkError) + ); + let (changed_element, changed_value_hash) = cost_return_on_error!( + &mut cost, + Element::get_with_value_hash(&child_tree, changed_key, true, grove_version) + .map_err(Error::MerkError) + ); + let (count, sum) = changed_element.count_sum_value_or_default(); + let axes: Vec<(u8, Option>)> = match indexed_element.underlying() { + Element::ProvableCountIndexedTree(_, secondary, ..) => vec![( + grovedb_element::indexed::IndexAxis::Count.tag(), + secondary.clone(), + )], + Element::ProvableSumIndexedTree(_, secondary, ..) => vec![( + grovedb_element::indexed::IndexAxis::Sum.tag(), + secondary.clone(), + )], + Element::ProvableCountProvableSumIndexedTree(_, _, _, axes, _) => axes.clone(), + _ => { + return Err(Error::CorruptedData( + "indexed primary is not held by an indexed-tree element".to_string(), + )) + .wrap_with_cost(cost); + } + }; + let is_single_axis = !matches!( + indexed_element.underlying(), + Element::ProvableCountProvableSumIndexedTree(..) + ); + let mut refreshed = Vec::with_capacity(axes.len()); + for (tag, secondary_root_key) in axes { + let axis = cost_return_on_error_no_add!( + cost, + grovedb_element::indexed::IndexAxis::try_from_tag(tag).map_err(|e| { + Error::CorruptedData(format!( + "invalid axis tag on indexed element during primary hash refresh: {e}" + )) + }) + ); + let mut secondary = cost_return_on_error!( + &mut cost, + self.open_indexed_secondary_at_path( + current_path.clone(), + axis, + secondary_root_key, + transaction, + Some(batch), + grove_version, + ) + ); + cost_return_on_error!( + &mut cost, + mirror_indexed_axis_to_secondary( + &mut secondary, + axis, + changed_key, + Some(count), + Some(sum), + Some(count), + Some(sum), + Some(changed_value_hash), + grove_version, + ) + ); + let (secondary_hash, secondary_key, _) = cost_return_on_error!( + &mut cost, + secondary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + refreshed.push((tag, secondary_hash, secondary_key)); + } + if is_single_axis { + let (_, hash, key) = cost_return_on_error_no_add!( + cost, + refreshed.pop().ok_or(Error::CorruptedCodeExecution( + "single-axis indexed tree has no secondary", + )) + ); + deferred_secondary = Some((hash, key)); + } else { + deferred_axes = Some(refreshed); + } + } + while let Some((parent_path, parent_key)) = current_path.derive_parent() { let mut parent_tree: Merk = cost_return_on_error!( &mut cost, @@ -1025,9 +1162,9 @@ impl GroveDb { // the aggregate here made `db.insert` and `apply_batch` place // the same child in different secondary buckets, committing // different root hashes for byte-identical writes. - let new_element_in_parent = cost_return_on_error!( + let (new_element_in_parent, new_primary_value_hash) = cost_return_on_error!( &mut cost, - Element::get(&parent_tree, parent_key, true, grove_version) + Element::get_with_value_hash(&parent_tree, parent_key, true, grove_version) .map_err(Error::MerkError) ); let (new_count, new_sum) = new_element_in_parent.count_sum_value_or_default(); @@ -1124,6 +1261,7 @@ impl GroveDb { Some(old_sum), Some(new_count), Some(new_sum), + Some(new_primary_value_hash), grove_version, ) ); @@ -1633,7 +1771,11 @@ impl GroveDb { issues: &mut VerificationIssues, grove_version: &GroveVersion, ) -> Result<(), Error> { - use crate::operations::indexed_tree::{axis_sort_key_len, make_axis_secondary_key}; + use grovedb_element::reference_path::ReferencePathType; + + use crate::operations::indexed_tree::{ + axis_row_reference, axis_sort_key_len, make_axis_secondary_key, + }; let mut all_query = Query::new(); all_query.insert_all(); @@ -1674,7 +1816,7 @@ impl GroveDb { // Expected secondary key for every primary entry, derived from the // entry's own aggregates exactly as the mirror derives it. - let mut expected: HashMap, (Vec, Element)> = HashMap::new(); + let mut expected: HashMap, (Vec, Element, CryptoHash)> = HashMap::new(); let mut content_iter = KVIterator::new(primary_merk.storage.raw_iter(), &all_query).unwrap(); while let Some((p_key, p_value)) = content_iter.next_kv().unwrap() { @@ -1695,13 +1837,37 @@ impl GroveDb { } let p_elem = Element::raw_decode(&p_value, grove_version)?; let (count, sum) = p_elem.count_sum_value_or_default(); - // The payload the mirror writes for this axis, so a row filed - // under the right key but carrying the wrong value is caught too - // — the key alone does not pin the stored aggregate. - let payload = crate::operations::indexed_tree::axis_row_payload(axis, count, sum)?; + let row = axis_row_reference(axis, &p_key, count, sum)?; + let primary_value_hash = primary_merk + .get_value_hash( + &p_key, + true, + Some(&Element::value_defined_cost_for_serialized_value), + grove_version, + ) + .unwrap() + .map_err(MerkError)?; + let primary_value_hash = match primary_value_hash { + Some(hash) => hash, + None => { + // The storage iterator can still surface a physically + // present node that is no longer reachable from the + // Merk root. That is database drift to report, not an + // infrastructure failure that should abort the walk. + let mut p = new_path.to_vec(); + p.push(sentinel("primary_unreachable_node")); + p.push(p_key.clone()); + issues.insert(p, ([0u8; 32], [0u8; 32], [0u8; 32])); + [0u8; 32] + } + }; expected.insert( p_key.clone(), - (make_axis_secondary_key(axis, count, sum, &p_key), payload), + ( + make_axis_secondary_key(axis, count, sum, &p_key), + row, + primary_value_hash, + ), ); } drop(content_iter); @@ -1711,7 +1877,7 @@ impl GroveDb { // real drift class: the same item in two sort buckets) are visible // instead of silently collapsing. let sort_len = axis_sort_key_len(axis); - let mut actual: HashMap, Vec<(Vec, Element)>> = HashMap::new(); + let mut actual: HashMap, Vec<(Vec, Element, CryptoHash)>> = HashMap::new(); let mut sec_iter = KVIterator::new(secondary_merk.storage.raw_iter(), &all_query).unwrap(); while let Some((sec_key, sec_value)) = sec_iter.next_kv().unwrap() { if sec_key.len() < sort_len { @@ -1722,11 +1888,33 @@ impl GroveDb { continue; } let sec_elem = Element::raw_decode(&sec_value, grove_version)?; + let sec_value_hash = secondary_merk + .get_value_hash( + &sec_key, + true, + Some(&Element::value_defined_cost_for_serialized_value), + grove_version, + ) + .unwrap() + .map_err(MerkError)?; + let sec_value_hash = match sec_value_hash { + Some(hash) => hash, + None => { + // As above, an unreachable physical node is itself a + // consistency finding. Keep walking so callers receive + // the full relational drift report. + let mut p = new_path.to_vec(); + p.push(sentinel("secondary_unreachable_node")); + p.push(sec_key.clone()); + issues.insert(p, ([0u8; 32], [0u8; 32], [0u8; 32])); + [0u8; 32] + } + }; let item_key = sec_key[sort_len..].to_vec(); actual .entry(item_key) .or_default() - .push((sec_key.clone(), sec_elem)); + .push((sec_key.clone(), sec_elem, sec_value_hash)); } drop(sec_iter); @@ -1742,7 +1930,7 @@ impl GroveDb { } } - for (item_key, (want_key, want_payload)) in &expected { + for (item_key, (want_key, want_row, primary_value_hash)) in &expected { match actual.get(item_key).and_then(|v| v.first()) { None => { let mut p = new_path.to_vec(); @@ -1750,7 +1938,7 @@ impl GroveDb { p.push(item_key.clone()); issues.insert(p, ([0u8; 32], [0u8; 32], [0u8; 32])); } - Some((got_key, _)) if got_key != want_key => { + Some((got_key, ..)) if got_key != want_key => { // The item is indexed, but under the wrong sort key — // i.e. the secondary's ordering value is stale. Report // the DECODED ordering value (right-aligned in the hash @@ -1768,16 +1956,37 @@ impl GroveDb { ), ); } - Some((_, got_payload)) if got_payload != want_payload => { - // Filed under the right key, but the stored payload - // disagrees with what the primary implies — the key - // encodes the ordering value, not the stored aggregate, - // so this is invisible to a key-only comparison. + Some((_, got_row, _)) if got_row != want_row => { let mut p = new_path.to_vec(); - p.push(sentinel("secondary_value_mismatch")); + let expected_path = ReferencePathType::SiblingReference(item_key.clone()); + let mismatch = match got_row { + Element::ReferenceWithSumItem(got_path, ..) + if got_path != &expected_path => + { + "secondary_reference_path_mismatch" + } + Element::ReferenceWithSumItem(_, got_hops, _, _) + if *got_hops != Some(1) => + { + "secondary_reference_hop_mismatch" + } + Element::ReferenceWithSumItem(_, _, got_sum, _) => { + let expected_sum = match want_row { + Element::ReferenceWithSumItem(_, _, sum, _) => *sum, + _ => unreachable!("axis_row_reference always returns reference"), + }; + if *got_sum != expected_sum { + "secondary_reference_sum_mismatch" + } else { + "secondary_reference_non_canonical" + } + } + _ => "secondary_legacy_or_non_reference_row", + }; + p.push(sentinel(mismatch)); p.push(item_key.clone()); - let want_bytes = want_payload.serialize(grove_version)?; - let got_bytes = got_payload.serialize(grove_version)?; + let want_bytes = want_row.serialize(grove_version)?; + let got_bytes = got_row.serialize(grove_version)?; issues.insert( p, ( @@ -1787,7 +1996,20 @@ impl GroveDb { ), ); } - Some(_) => { /* indexed under the expected sort key and value */ } + Some((_, _, got_value_hash)) => { + let row_bytes = want_row.serialize(grove_version)?; + let row_hash = value_hash(&row_bytes).unwrap(); + let expected_value_hash = combine_hash(&row_hash, primary_value_hash).unwrap(); + if *got_value_hash != expected_value_hash { + let mut p = new_path.to_vec(); + p.push(sentinel("secondary_stale_target_hash")); + p.push(item_key.clone()); + issues.insert( + p, + (*primary_value_hash, expected_value_hash, *got_value_hash), + ); + } + } } } @@ -2384,7 +2606,7 @@ impl GroveDb { /// Compute the child hash for a non-Merk tree element by reconstructing /// its tree from storage and computing the state root. /// Falls back to `merk_root_hash` on any error or for standard Merk trees. - fn compute_non_merk_child_hash<'b, B: AsRef<[u8]>>( + pub(crate) fn compute_non_merk_child_hash<'b, B: AsRef<[u8]>>( &self, element: &Element, subtree_path: SubtreePath<'b, B>, diff --git a/grovedb/src/operations/bulk_append_tree.rs b/grovedb/src/operations/bulk_append_tree.rs index 50d13cccb..8686a7948 100644 --- a/grovedb/src/operations/bulk_append_tree.rs +++ b/grovedb/src/operations/bulk_append_tree.rs @@ -143,9 +143,10 @@ impl GroveDb { cost_return_on_error!( &mut cost, - self.propagate_changes_with_transaction( + self.propagate_changes_with_transaction_refreshing_indexed_row( merk_cache, path, + key, tx.as_ref(), &batch, grove_version, diff --git a/grovedb/src/operations/commitment_tree.rs b/grovedb/src/operations/commitment_tree.rs index b55a2f4e9..9da119909 100644 --- a/grovedb/src/operations/commitment_tree.rs +++ b/grovedb/src/operations/commitment_tree.rs @@ -228,9 +228,10 @@ impl GroveDb { cost_return_on_error!( &mut cost, - self.propagate_changes_with_transaction( + self.propagate_changes_with_transaction_refreshing_indexed_row( merk_cache, path, + key, tx.as_ref(), &batch, grove_version, diff --git a/grovedb/src/operations/dense_tree.rs b/grovedb/src/operations/dense_tree.rs index d8116edd2..6fc3a4877 100644 --- a/grovedb/src/operations/dense_tree.rs +++ b/grovedb/src/operations/dense_tree.rs @@ -141,9 +141,10 @@ impl GroveDb { cost_return_on_error!( &mut cost, - self.propagate_changes_with_transaction( + self.propagate_changes_with_transaction_refreshing_indexed_row( merk_cache, path, + key, tx.as_ref(), &batch, grove_version, diff --git a/grovedb/src/operations/get/mod.rs b/grovedb/src/operations/get/mod.rs index 40f4f0997..2ce2a6780 100644 --- a/grovedb/src/operations/get/mod.rs +++ b/grovedb/src/operations/get/mod.rs @@ -29,8 +29,8 @@ use crate::{ Element, Error, GroveDb, Transaction, TransactionArg, }; -/// Limit of possible indirections -pub const MAX_REFERENCE_HOPS: usize = 10; +/// Limit of possible indirections. +pub use super::MAX_REFERENCE_HOPS; impl GroveDb { /// Get an element from the backing store diff --git a/grovedb/src/operations/indexed_tree.rs b/grovedb/src/operations/indexed_tree.rs index 8edfe00a0..55bc06835 100644 --- a/grovedb/src/operations/indexed_tree.rs +++ b/grovedb/src/operations/indexed_tree.rs @@ -41,8 +41,8 @@ use grovedb_merk::{ }, merk::KVIterator, proofs::Query, - tree::{kv::ValueDefinedCostType, AggregateData, TreeNode}, - Merk, TreeType, + tree::{combine_hash, kv::ValueDefinedCostType, value_hash, AggregateData, TreeNode}, + CryptoHash, Merk, TreeType, }; use grovedb_path::SubtreePath; use grovedb_storage::{ @@ -51,7 +51,10 @@ use grovedb_storage::{ }; use grovedb_version::version::GroveVersion; -use crate::{util::TxRef, Element, Error, GroveDb, Transaction, TransactionArg}; +use crate::{ + query_result_type::IndexedAxisEntry, util::TxRef, Element, Error, GroveDb, Transaction, + TransactionArg, +}; /// Per-axis Merk tree type to open the secondary with. /// @@ -88,31 +91,14 @@ pub(crate) fn axis_secondary_tree_type(axis: IndexAxis) -> TreeType { } } -/// The count-axis secondary stores each entry's `count_value` as its -/// sum item, and sum items are `i64`: a count above `i64::MAX` cannot -/// be mirrored faithfully, so it FAILS CLOSED rather than clamping — -/// a clamped total would silently lie through authenticated state. -/// Unreachable for any real tree (a count is bounded by the number of -/// elements), so the guard is a type-level seam, not a live limit. -#[inline] -pub(crate) fn count_value_as_sum(count: u64) -> Result { - i64::try_from(count).map_err(|_| { - Error::CorruptedData(format!( - "count value {count} exceeds i64::MAX and cannot be mirrored into the \ - count-axis secondary's sum aggregate" - )) - }) -} - -/// THE per-axis secondary row payload — the single definition every -/// writer and every checker uses. The sort KEY encodes the ordering -/// value; this payload carries what the secondary's own dual -/// aggregates must fold to: +/// Build THE canonical per-axis secondary row. The sort key encodes the +/// ordering value while the row is a one-hop sibling reference to the primary +/// entry. Its explicit sum keeps the secondary's dual aggregates identical to +/// the pre-reference representation: /// -/// - Count → `SumItem(count_value)` — contributes `(1, count)`, so a -/// band TOTAL is one committed scalar (issue #806) -/// - Sum → `SumItem(sum)` — contributes `(1, sum)` -/// - Avg → `ItemWithSumItem(empty, sum)` — contributes `(1, sum)` +/// - Count contributes `(1, i64::try_from(count))` and fails closed on +/// overflow. +/// - Sum and Avg contribute `(1, primary_sum)`. /// /// Callers: the batch mirror row builder, the direct-path mirror, the /// reconcile repair loop, `verify_grovedb`'s expected-payload check, @@ -124,13 +110,18 @@ pub(crate) fn count_value_as_sum(count: u64) -> Result { /// writes. (This function exists because exactly that drift risk was /// flagged by the #809 security audit.) /// -/// Fallible only through [`count_value_as_sum`]'s fail-closed guard. -pub(crate) fn axis_row_payload(axis: IndexAxis, count: u64, sum: i64) -> Result { - Ok(match axis { - IndexAxis::Count => Element::new_sum_item(count_value_as_sum(count)?), - IndexAxis::Sum => Element::new_sum_item(sum), - IndexAxis::Avg => Element::new_item_with_sum_item(Vec::new(), sum), - }) +/// `max_reference_hop = Some(1)` is an internal indexed-row marker and a +/// commitment rule: the row combines with the immediate primary node's +/// Merk-stored value hash. It does not relax ordinary user-reference +/// semantics. +pub(crate) fn axis_row_reference( + axis: IndexAxis, + primary_key: &[u8], + count: u64, + sum: i64, +) -> Result { + grovedb_element::canonical_axis_reference(axis, primary_key, count, sum) + .map_err(Error::ElementError) } /// Build the secondary key bytes for an entry at `item_key` under the @@ -1024,7 +1015,7 @@ impl GroveDb { all_query.insert_all(); let mut iter = KVIterator::new(primary_merk.storage.raw_iter(), &all_query).unwrap_add_cost(&mut cost); - let mut entries: Vec<(Vec, (u64, i64))> = Vec::new(); + let mut entries: Vec<(Vec, (u64, i64), CryptoHash)> = Vec::new(); while let Some((key, value_bytes)) = iter.next_kv().unwrap_add_cost(&mut cost) { // Reject oversized primary keys before they can drive // make_axis_secondary_key to synthesize a secondary key that @@ -1052,7 +1043,30 @@ impl GroveDb { )) }) ); - entries.push((key, element.count_sum_value_or_default())); + let primary_value_hash = cost_return_on_error!( + &mut cost, + primary_merk + .get_value_hash( + &key, + true, + Some(&Element::value_defined_cost_for_serialized_value), + grove_version, + ) + .map_err(|e| Error::CorruptedData(format!( + "failed to read primary value hash while reconciling: {e}" + ))) + ) + .ok_or_else(|| { + Error::CorruptedData( + "primary entry disappeared while reconciling indexed secondaries".to_string(), + ) + }); + let primary_value_hash = cost_return_on_error_no_add!(cost, primary_value_hash); + entries.push(( + key, + element.count_sum_value_or_default(), + primary_value_hash, + )); } // 4. Rebuild each axis's secondary and capture its post-repair @@ -1080,21 +1094,13 @@ impl GroveDb { // shape on every run — two operators repairing identical // databases must derive identical secondary root hashes, or the // H1-A parent bindings would disagree. - let mut desired: std::collections::BTreeMap, Vec> = + let mut desired: std::collections::BTreeMap, (Element, CryptoHash)> = std::collections::BTreeMap::new(); - for (key, (count, sum)) in &entries { + for (key, (count, sum), primary_value_hash) in &entries { let secondary_key = make_axis_secondary_key(axis, *count, *sum, key); - let payload = - cost_return_on_error_no_add!(cost, axis_row_payload(axis, *count, *sum)); - let payload_bytes = cost_return_on_error_no_add!( - cost, - payload.serialize(grove_version).map_err(|e| { - Error::CorruptedData(format!( - "failed to serialize desired secondary payload: {e}" - )) - }) - ); - desired.insert(secondary_key, payload_bytes); + let row = + cost_return_on_error_no_add!(cost, axis_row_reference(axis, key, *count, *sum)); + desired.insert(secondary_key, (row, *primary_value_hash)); } // Existing row KEYS, raw-iterated so unlinked-but-present rows @@ -1132,12 +1138,23 @@ impl GroveDb { // Insert missing rows and rewrite payload-damaged ones. Sum and // avg rows carry real payloads, so key-presence alone does not // imply row-correctness; compare the stored element bytes. - for (desired_key, desired_payload) in &desired { + for (desired_key, (desired_row, primary_value_hash)) in &desired { + let desired_bytes = cost_return_on_error_no_add!( + cost, + desired_row.serialize(grove_version).map_err(|e| { + Error::CorruptedData(format!( + "failed to serialize desired secondary reference: {e}" + )) + }) + ); + let row_hash = value_hash(&desired_bytes).unwrap_add_cost(&mut cost); + let expected_combined_hash = + combine_hash(&row_hash, primary_value_hash).unwrap_add_cost(&mut cost); let needs_write = if existing_keys.contains(desired_key) { let stored = cost_return_on_error!( &mut cost, secondary_merk - .get( + .get_value_and_value_hash( desired_key.as_slice(), true, Some(&Element::value_defined_cost_for_serialized_value), @@ -1147,27 +1164,23 @@ impl GroveDb { "reading secondary row for payload compare: {e}" ))) ); - stored.as_deref() != Some(desired_payload.as_slice()) + !matches!( + stored, + Some((stored_bytes, stored_value_hash)) + if stored_bytes == desired_bytes + && stored_value_hash == expected_combined_hash + ) } else { true }; if needs_write { - let entry = cost_return_on_error_no_add!( - cost, - Element::deserialize(desired_payload.as_slice(), grove_version).map_err( - |e| { - Error::CorruptedData(format!( - "failed to round-trip desired secondary payload: {e}" - )) - } - ) - ); cost_return_on_error!( &mut cost, - entry - .insert( + desired_row + .insert_reference( &mut secondary_merk, desired_key.as_slice(), + *primary_value_hash, None, grove_version, ) @@ -1317,7 +1330,7 @@ impl GroveDb { transaction: TransactionArg, grove_version: &GroveVersion, decode: impl Fn(&[u8]) -> Option<(T, Vec)>, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, { @@ -1331,10 +1344,16 @@ impl GroveDb { let secondary_merk = cost_return_on_error!( &mut cost, - self.open_validated_axis_secondary(path, axis, tx_ref, grove_version) + self.open_validated_axis_secondary(path.clone(), axis, tx_ref, grove_version) ); - collect_top_k_via_iterator(&secondary_merk, axis, k, descending, &decode).add_cost(cost) + let decoded = cost_return_on_error!( + &mut cost, + collect_top_k_via_iterator(&secondary_merk, axis, k, descending, &decode) + ); + drop(secondary_merk); + self.resolve_indexed_axis_entries(path, decoded, tx_ref, grove_version) + .add_cost(cost) } /// One implementation of the `indexed__top_k_paginated` shape. @@ -1373,12 +1392,20 @@ impl GroveDb { // skips zero entries, so `skipped = min(0, population) = 0` needs // no tree read. if offset == 0 { - return collect_top_k_via_iterator(&secondary_merk, axis, k, descending, &decode) - .map_ok(|entries| IndexedTopKPage { - entries, - skipped: 0, - }) - .add_cost(cost); + let decoded = cost_return_on_error!( + &mut cost, + collect_top_k_via_iterator(&secondary_merk, axis, k, descending, &decode) + ); + drop(secondary_merk); + let entries = cost_return_on_error!( + &mut cost, + self.resolve_indexed_axis_entries(path, decoded, tx_ref, grove_version,) + ); + return Ok(IndexedTopKPage { + entries, + skipped: 0, + }) + .wrap_with_cost(cost); } // The open above serves validation (path shape, element variant, // axis compatibility) and the offset-0 fast path only. For the @@ -1407,7 +1434,7 @@ impl GroveDb { }; let parent_prefix = RocksDbStorage::build_prefix(parent_path.clone()).unwrap_add_cost(&mut cost); - let primary_prefix = RocksDbStorage::build_prefix(path).unwrap_add_cost(&mut cost); + let primary_prefix = RocksDbStorage::build_prefix(path.clone()).unwrap_add_cost(&mut cost); let secondary_prefix = RocksDbStorage::secondary_prefix_for(&primary_prefix, axis.tag()) .unwrap_add_cost(&mut cost); let parent_ctx = self @@ -1456,16 +1483,20 @@ impl GroveDb { grove_version ) ); - let mut entries = Vec::with_capacity(secondary_keys.len()); + let mut decoded_entries = Vec::with_capacity(secondary_keys.len()); for secondary_key in secondary_keys { match decode(&secondary_key) { - Some(decoded) => entries.push(decoded), + Some(decoded) => decoded_entries.push(decoded), None => { return Err(corrupted_secondary_key_error(axis, &secondary_key)) .wrap_with_cost(cost); } } } + let entries = cost_return_on_error!( + &mut cost, + self.resolve_indexed_axis_entries(path, decoded_entries, tx_ref, grove_version) + ); Ok(IndexedTopKPage { entries, skipped }).wrap_with_cost(cost) } @@ -1487,7 +1518,7 @@ impl GroveDb { transaction: TransactionArg, grove_version: &GroveVersion, decode: impl Fn(&[u8]) -> Option<(T, Vec)>, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, { @@ -1501,7 +1532,7 @@ impl GroveDb { let secondary_merk = cost_return_on_error!( &mut cost, - self.open_validated_axis_secondary(path, axis, tx_ref, grove_version) + self.open_validated_axis_secondary(path.clone(), axis, tx_ref, grove_version) ); let mut q = Query::new(); @@ -1514,21 +1545,100 @@ impl GroveDb { let mut iter = KVIterator::new(secondary_merk.storage.raw_iter(), &q).unwrap_add_cost(&mut cost); - let mut results = Vec::new(); - while results.len() < limit as usize { + let mut decoded_results = Vec::new(); + while decoded_results.len() < limit as usize { match iter.next_kv().unwrap_add_cost(&mut cost) { Some((secondary_key, _)) => { let Some(decoded) = decode(&secondary_key) else { return Err(corrupted_secondary_key_error(axis, &secondary_key)) .wrap_with_cost(cost); }; - results.push(decoded); + decoded_results.push(decoded); } None => break, } } + drop(iter); + drop(secondary_merk); - Ok(results).wrap_with_cost(cost) + self.resolve_indexed_axis_entries(path, decoded_results, tx_ref, grove_version) + .add_cost(cost) + } + + /// Resolve decoded secondary keys through the indexed primary. This is + /// intentionally separate from interpreting the secondary row's logical + /// origin: the row authenticates one hop to the immediate primary node, + /// while the public read applies ordinary GroveDB semantics and follows a + /// reference-shaped primary to its terminal value. + fn resolve_indexed_axis_entries<'b, B, T>( + &self, + path: SubtreePath<'b, B>, + decoded: Vec<(T, Vec)>, + transaction: &Transaction, + grove_version: &GroveVersion, + ) -> CostResult>, Error> + where + B: AsRef<[u8]> + 'b, + { + let mut cost = OperationCost::default(); + if decoded.is_empty() { + return Ok(Vec::new()).wrap_with_cost(cost); + } + + // Open the primary once for the whole result page. Calling `get` + // for every row would reopen this same Merk and repeat the same + // path validation `k` times. + let primary_merk = cost_return_on_error!( + &mut cost, + self.open_transactional_merk_at_path(path.clone(), transaction, None, grove_version) + ); + let mut entries = Vec::with_capacity(decoded.len()); + for (ordering_value, primary_key) in decoded { + let element = cost_return_on_error!( + &mut cost, + Element::get(&primary_merk, &primary_key, true, grove_version).map_err(|e| { + Error::CorruptedData(format!( + "indexed axis read: primary entry {} named by a secondary row is missing: {e}", + hex::encode(&primary_key) + )) + }) + ); + let value = match element.underlying() { + Element::Reference(reference_path, ..) + | Element::ReferenceWithSumItem(reference_path, ..) => { + // Match `GroveDb::get`: relative reference paths are + // resolved from the indexed tree's path, not from a + // qualified path that already includes `primary_key`. + // The latter would append sibling keys one level too + // deep (`.../primary_key/sibling`). + let absolute = cost_return_on_error_no_add!( + cost, + crate::reference_path::path_from_reference_path_type( + reference_path.clone(), + &path.to_vec(), + Some(primary_key.as_slice()), + ) + .map_err(Error::from) + ); + cost_return_on_error!( + &mut cost, + self.follow_reference( + absolute.as_slice().into(), + true, + Some(transaction), + grove_version, + ) + ) + } + _ => element, + }; + entries.push(IndexedAxisEntry { + ordering_value, + primary_key, + value, + }); + } + Ok(entries).wrap_with_cost(cost) } // ---- count axis ---- @@ -1544,9 +1654,8 @@ impl GroveDb { /// contains the count axis). Any other variant — or a PCPSIT /// without the count axis — is rejected with `Error::InvalidPath`. /// - /// Each returned entry is `(count, original_key)`. Resolving the - /// primary value is the caller's responsibility (use - /// `db.get(path, original_key, ...)`). + /// Each returned entry includes the count, primary key, and resolved + /// primary value. /// /// For a verifiable variant, see [`Self::prove_indexed_count_top_k`] /// and [`Self::verify_indexed_count_top_k`]. @@ -1557,7 +1666,7 @@ impl GroveDb { descending: bool, transaction: TransactionArg, grove_version: &GroveVersion, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, P: Into>, @@ -1631,7 +1740,7 @@ impl GroveDb { limit: u16, transaction: TransactionArg, grove_version: &GroveVersion, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, P: Into>, @@ -1765,9 +1874,9 @@ impl GroveDb { /// contains the sum axis). Any other variant — or a PCPSIT without /// the sum axis — is rejected with `Error::InvalidPath`. /// - /// Each returned entry is `(sum, original_key)`. The signed `i64` - /// sum is decoded from the secondary's sign-flipped big-endian - /// prefix (see [`grovedb_element::indexed::encode_sum_sort_key`]). + /// Each returned entry includes the signed sum, primary key, and + /// resolved primary value. The sum is decoded from the secondary's + /// sign-flipped big-endian prefix. pub fn indexed_sum_top_k<'b, B, P>( &self, path: P, @@ -1775,7 +1884,7 @@ impl GroveDb { descending: bool, transaction: TransactionArg, grove_version: &GroveVersion, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, P: Into>, @@ -1839,7 +1948,7 @@ impl GroveDb { limit: u16, transaction: TransactionArg, grove_version: &GroveVersion, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, P: Into>, @@ -2089,7 +2198,8 @@ impl GroveDb { /// PCPSIT without the avg axis — is rejected with /// `Error::InvalidPath`. /// - /// Each returned entry is `(avg_fixed_point_i128, original_key)`. + /// Each returned entry includes the fixed-point average, primary key, + /// and resolved primary value. /// Divide by `AVG_FIXED_POINT_SCALE` (`10^19`) to recover a float /// view if you need one — noting an `f64` view is approximate at /// this scale; the `i128` fixed-point value is the exact consensus @@ -2101,7 +2211,7 @@ impl GroveDb { descending: bool, transaction: TransactionArg, grove_version: &GroveVersion, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, P: Into>, @@ -2174,7 +2284,7 @@ impl GroveDb { limit: u16, transaction: TransactionArg, grove_version: &GroveVersion, - ) -> CostResult)>, Error> + ) -> CostResult>, Error> where B: AsRef<[u8]> + 'b, P: Into>, @@ -2459,8 +2569,8 @@ impl GroveDb { /// from the prior primary state and `new_count`/`new_sum` from the /// post-mutation state. /// -/// The row payload is [`axis_row_payload`] — see its doc for the -/// per-axis shapes and why every writer must share it. +/// The row is [`axis_row_reference`] and binds the immediate primary node's +/// committed value hash. #[allow(clippy::too_many_arguments)] pub(crate) fn mirror_indexed_axis_to_secondary<'db, S: StorageContext<'db>>( secondary: &mut Merk, @@ -2470,17 +2580,12 @@ pub(crate) fn mirror_indexed_axis_to_secondary<'db, S: StorageContext<'db>>( old_sum: Option, new_count: Option, new_sum: Option, + new_target_hash: Option, grove_version: &GroveVersion, ) -> CostResult<(), Error> { let mut cost = OperationCost::default(); let secondary_tree_type = axis_secondary_tree_type(axis); - // The payload is a function of BOTH aggregates (the count axis - // mirrors count_value into its sum half), so the fast-path equality - // check below must see the full pair — a sum-only view would - // silently skip real payload updates. - let axis_payload = |count: u64, sum: i64| axis_row_payload(axis, count, sum); - // Compute old and new sort keys. Either may be None (no entry). let old_key = match (old_count, old_sum) { (Some(c), Some(s)) => Some(make_axis_secondary_key(axis, c, s, item_key)), @@ -2491,44 +2596,11 @@ pub(crate) fn mirror_indexed_axis_to_secondary<'db, S: StorageContext<'db>>( _ => None, }; - // Fast path: skip the delete+insert only when BOTH the sort key AND - // the stored payload are unchanged. - // - // Equal sort keys do NOT imply equal payloads on the Avg axis: the - // avg key is floor(sum * 10^19 / count) while the payload carries the - // raw `sum`, so e.g. (count, sum) = (1, 5) and (2, 10) share a key - // (avg 5.0) but differ in payload sum (5 vs 10). Returning early on - // key-equality alone would leave a stale hash-committed sum in the - // avg secondary. On the Count / Sum axes the payload is a pure - // function of what the key already encodes, so this reduces to the - // former key-only check and preserves the fast path for the common - // case. - if old_key == new_key && new_key.is_some() { - let payload_unchanged = match ((old_count, old_sum), (new_count, new_sum)) { - ((Some(oc), Some(os)), (Some(nc), Some(ns))) => { - cost_return_on_error_no_add!(cost, axis_payload(oc, os)) - == cost_return_on_error_no_add!(cost, axis_payload(nc, ns)) - } - _ => false, - }; - if payload_unchanged { - // The sort key didn't move and the stored value is identical; - // the previous insert already wrote the correct value, so - // nothing more is needed. - return Ok(()).wrap_with_cost(cost); - } - } - - // A payload change at a FIXED key (avg axis only: a proportional - // (count, sum) change keeps the average) must replace in place. - // Deleting and reinserting the same key rebalances the AVL twice - // and can settle a DIFFERENT shape than the batch mirror's single - // replacement write — two write paths committing different - // secondary (hence grove) root hashes for identical data, which - // `direct_and_batch_agree_on_root_for_a_fixed_key_avg_payload_change` - // reproduced on an interior node before this skip existed. The - // insert below overwrites the value in place, exactly like the - // batch path's put. + // A touched surviving entry is always rewritten, even when its sort key + // and aggregates are unchanged. Its primary node hash may have changed + // because of an item update, reference refresh, or child-root + // propagation. A same-key combined-reference put updates the value in + // place without a delete/reinsert rebalance. let key_moved = old_key != new_key; if let (true, Some(ok)) = (key_moved, &old_key) { cost_return_on_error!( @@ -2545,19 +2617,20 @@ pub(crate) fn mirror_indexed_axis_to_secondary<'db, S: StorageContext<'db>>( ); } if let (Some(nk), Some(new_count_val), Some(new_sum_val)) = (&new_key, new_count, new_sum) { - // Derived by the same `axis_payload` closure the fast-path - // equality check above uses, so the two stay in lockstep. - // Every axis's secondary is a dual-aggregate - // ProvableCountProvableSumTree: - // - Count → SumItem(count_value) — contributes (1, count), so a - // band TOTAL is one committed scalar (issue #806) - // - Sum → SumItem(sum) — contributes (1, sum) - // - Avg → ItemWithSumItem(empty, sum) — contributes (1, sum) - let entry = cost_return_on_error_no_add!(cost, axis_payload(new_count_val, new_sum_val)); + let target_hash = cost_return_on_error_no_add!( + cost, + new_target_hash.ok_or(Error::CorruptedCodeExecution( + "surviving indexed primary entry is missing its committed value hash" + )) + ); + let entry = cost_return_on_error_no_add!( + cost, + axis_row_reference(axis, item_key, new_count_val, new_sum_val) + ); cost_return_on_error!( &mut cost, entry - .insert(secondary, nk.as_slice(), None, grove_version) + .insert_reference(secondary, nk.as_slice(), target_hash, None, grove_version,) .map_err(Error::MerkError) ); } @@ -2734,8 +2807,8 @@ fn provable_count_from_aggregate(aggregate: AggregateData) -> Result /// One page of an `indexed__top_k_paginated` read. #[derive(Debug, Clone, PartialEq, Eq)] pub struct IndexedTopKPage { - /// Page entries, `(axis_value, original_key)`, in directional order. - pub entries: Vec<(T, Vec)>, + /// Resolved page entries in directional order. + pub entries: Vec>, /// How many entries the offset actually skipped: /// `min(offset, population)`. When the offset runs past the end this /// reports the secondary's true population instead of echoing the @@ -3191,7 +3264,7 @@ impl GroveDb { } #[cfg(test)] -mod axis_row_payload_tests { +mod axis_row_reference_tests { //! The payload function is THE definition every writer and checker //! shares; this grid pins its output per (axis, count, sum) so any //! change to the shape is a deliberate, reviewed event — the bytes @@ -3199,9 +3272,10 @@ mod axis_row_payload_tests { //! mirrors and checkers disagree about healthy databases. use grovedb_element::indexed::IndexAxis; + use grovedb_element::reference_path::ReferencePathType; use grovedb_version::version::GroveVersion; - use super::axis_row_payload; + use super::axis_row_reference; use crate::Element; #[test] @@ -3211,24 +3285,36 @@ mod axis_row_payload_tests { for &count in &counts { for &sum in &sums { assert_eq!( - axis_row_payload(IndexAxis::Count, count, sum).unwrap(), - Element::new_sum_item(count as i64), - "count axis stores the COUNT as its sum item; the sum input is ignored" + axis_row_reference(IndexAxis::Count, b"key", count, sum).unwrap(), + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"key".to_vec()), + Some(1), + count as i64, + ), + "count axis references the primary and carries COUNT as its sum" ); assert_eq!( - axis_row_payload(IndexAxis::Sum, count, sum).unwrap(), - Element::new_sum_item(sum), - "sum axis stores the sum; the count input is ignored" + axis_row_reference(IndexAxis::Sum, b"key", count, sum).unwrap(), + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"key".to_vec()), + Some(1), + sum, + ), + "sum axis references the primary and carries its sum" ); assert_eq!( - axis_row_payload(IndexAxis::Avg, count, sum).unwrap(), - Element::new_item_with_sum_item(Vec::new(), sum), - "avg axis stores an empty item carrying the sum" + axis_row_reference(IndexAxis::Avg, b"key", count, sum).unwrap(), + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"key".to_vec()), + Some(1), + sum, + ), + "avg axis references the primary and carries its sum" ); } } // Above the sum-item domain the count axis fails closed. - axis_row_payload(IndexAxis::Count, i64::MAX as u64 + 1, 0) + axis_row_reference(IndexAxis::Count, b"key", i64::MAX as u64 + 1, 0) .expect_err("count above i64::MAX must fail closed"); } @@ -3241,52 +3327,18 @@ mod axis_row_payload_tests { // bytes in authenticated state have changed: that is a // consensus event, not a refactor. let grove_version = GroveVersion::latest(); - assert_eq!( - axis_row_payload(IndexAxis::Count, 7, 0) - .unwrap() - .serialize(grove_version) - .unwrap(), - vec![3, 14, 0], - "count axis: SumItem(7) as [variant, zigzag-varint 7, no flags]" - ); - assert_eq!( - axis_row_payload(IndexAxis::Sum, 1, -3) - .unwrap() - .serialize(grove_version) - .unwrap(), - vec![3, 5, 0], - "sum axis: SumItem(-3) as [variant, zigzag-varint -3, no flags]" - ); - assert_eq!( - axis_row_payload(IndexAxis::Avg, 1, 5) - .unwrap() - .serialize(grove_version) - .unwrap(), - vec![9, 0, 10, 0], - "avg axis: ItemWithSumItem(empty, 5) as [variant, empty item, \ - zigzag-varint 5, no flags]" - ); - } -} - -#[cfg(test)] -mod count_value_as_sum_tests { - //! The count-axis secondary stores count_value as an i64 sum item; - //! the conversion FAILS CLOSED above i64::MAX rather than clamping, - //! because a clamped value would flow into hash-bound authenticated - //! state as a silently wrong total. - - use super::count_value_as_sum; - - #[test] - fn converts_in_domain_and_fails_closed_above_i64_max() { - assert_eq!(count_value_as_sum(0).unwrap(), 0); - assert_eq!(count_value_as_sum(8).unwrap(), 8); - assert_eq!(count_value_as_sum(i64::MAX as u64).unwrap(), i64::MAX); - let err = count_value_as_sum(i64::MAX as u64 + 1) - .expect_err("one past i64::MAX must fail closed"); - assert!(err.to_string().contains("cannot be mirrored"), "{err}"); - count_value_as_sum(u64::MAX).expect_err("u64::MAX must fail closed"); + for (axis, count, sum) in [ + (IndexAxis::Count, 7, 0), + (IndexAxis::Sum, 1, -3), + (IndexAxis::Avg, 1, 5), + ] { + let row = axis_row_reference(axis, b"key", count, sum).unwrap(); + let bytes = row.serialize(grove_version).unwrap(); + assert_eq!(Element::deserialize(&bytes, grove_version).unwrap(), row); + assert_eq!(bytes[0], 18, "canonical row uses ReferenceWithSumItem"); + } + axis_row_reference(IndexAxis::Count, b"key", i64::MAX as u64 + 1, 0) + .expect_err("count-axis sum conversion must fail closed"); } } @@ -3423,7 +3475,7 @@ mod bug2_avg_axis_mirror_tests { //! the stored payload sum differs. //! //! The avg sort key is `floor(sum * 10^19 / count)`, while the stored - //! payload is `ItemWithSumItem(_, sum)`. Two `(count, sum)` pairs can + //! primary payload is `ItemWithSumItem(_, sum)`. Two `(count, sum)` pairs can //! share a key yet differ in payload sum — e.g. `(1, 5)` and `(2, 10)` //! both encode avg `5.0` but carry payload sums `5` and `10`. The old //! key-only early-return left the stale hash-committed sum `5` in the @@ -3437,7 +3489,10 @@ mod bug2_avg_axis_mirror_tests { use grovedb_costs::OperationCost; use grovedb_element::indexed::{compute_avg_fixed_point, IndexAxis}; - use grovedb_merk::{element::get::ElementFetchFromStorageExtensions, tree::AggregateData}; + use grovedb_merk::{ + element::{costs::ElementCostExtensions, get::ElementFetchFromStorageExtensions}, + tree::AggregateData, + }; use grovedb_path::SubtreePath; use grovedb_storage::StorageBatch; use grovedb_version::version::GroveVersion; @@ -3531,6 +3586,7 @@ mod bug2_avg_axis_mirror_tests { None, Some(1), Some(5), + Some([1; 32]), grove_version, ) .unwrap() @@ -3554,6 +3610,7 @@ mod bug2_avg_axis_mirror_tests { Some(5), Some(2), Some(10), + Some([2; 32]), grove_version, ) .unwrap() @@ -3570,11 +3627,10 @@ mod bug2_avg_axis_mirror_tests { ); } - /// The fast path must still short-circuit when BOTH the key and the - /// payload are unchanged (a genuine no-op transition). We verify the - /// stored payload is untouched for an identical (count, sum) rewrite. + /// Even when key and aggregates are unchanged, a touched primary entry + /// must refresh the secondary's immediate-target commitment. #[test] - fn avg_axis_mirror_noop_when_key_and_payload_unchanged() { + fn avg_axis_mirror_refreshes_when_key_and_payload_unchanged() { let grove_version = GroveVersion::latest(); let db = make_test_grovedb(grove_version); let axes: Vec<(u8, Option>)> = vec![(IndexAxis::Avg.tag(), None)]; @@ -3614,16 +3670,25 @@ mod bug2_avg_axis_mirror_tests { None, Some(2), Some(10), + Some([1; 32]), grove_version, ) .unwrap() .expect("insert (2,10)"); - // Identical (count, sum) rewrite: key AND payload unchanged. - // Assert on the COST, not just the stored value — a delete+reinsert - // would leave the same value behind, so a value-only assertion passes - // even with the fast path deleted entirely. - let mut noop_cost = OperationCost::default(); + let key = make_axis_secondary_key(IndexAxis::Avg, 2, 10, item_key); + let before_hash = secondary + .get_value_hash( + &key, + true, + Some(&Element::value_defined_cost_for_serialized_value), + grove_version, + ) + .unwrap() + .expect("value hash read") + .expect("entry present"); + + let mut refresh_cost = OperationCost::default(); mirror_indexed_axis_to_secondary( &mut secondary, IndexAxis::Avg, @@ -3632,32 +3697,31 @@ mod bug2_avg_axis_mirror_tests { Some(10), Some(2), Some(10), + Some([2; 32]), grove_version, ) - .unwrap_add_cost(&mut noop_cost) - .expect("noop rewrite"); - // Byte deltas are NOT a usable signal here: a delete-then-reinsert of - // an identical entry nets zero added/replaced/removed bytes. Merk - // work is the discriminator — the fast path touches storage not at - // all, while the delete+insert it replaces seeks and rehashes. - assert_eq!( - noop_cost.seek_count, 0, - "an unchanged rewrite must not touch storage" - ); - assert_eq!( - noop_cost.hash_node_calls, 0, - "an unchanged rewrite must not rehash the secondary" - ); - assert_eq!( - noop_cost.storage_loaded_bytes, 0, - "an unchanged rewrite must not load" + .unwrap_add_cost(&mut refresh_cost) + .expect("same-key reference refresh"); + assert!( + refresh_cost.hash_node_calls > 0, + "refresh must rehash the row" ); - let key = make_axis_secondary_key(IndexAxis::Avg, 2, 10, item_key); let entry = Element::get(&secondary, key.as_slice(), true, grove_version) .unwrap() .expect("entry present"); assert_eq!(entry.sum_value_or_default(), 10, "payload must remain 10"); + let after_hash = secondary + .get_value_hash( + &key, + true, + Some(&Element::value_defined_cost_for_serialized_value), + grove_version, + ) + .unwrap() + .expect("value hash read") + .expect("entry present"); + assert_ne!(before_hash, after_hash, "target commitment must refresh"); } /// A delete reaches the mirror as `new_count = new_sum = None`, which must @@ -3705,6 +3769,7 @@ mod bug2_avg_axis_mirror_tests { None, Some(2), Some(10), + Some([1; 32]), grove_version, ) .unwrap() @@ -3734,6 +3799,7 @@ mod bug2_avg_axis_mirror_tests { Some(10), None, None, + None, grove_version, ) .unwrap() diff --git a/grovedb/src/operations/mmr_tree.rs b/grovedb/src/operations/mmr_tree.rs index 586c55b27..64a81539d 100644 --- a/grovedb/src/operations/mmr_tree.rs +++ b/grovedb/src/operations/mmr_tree.rs @@ -156,9 +156,10 @@ impl GroveDb { cost_return_on_error!( &mut cost, - self.propagate_changes_with_transaction( + self.propagate_changes_with_transaction_refreshing_indexed_row( merk_cache, path, + key, tx.as_ref(), &batch, grove_version, diff --git a/grovedb/src/operations/mod.rs b/grovedb/src/operations/mod.rs index 8e556f3ad..445300bcc 100644 --- a/grovedb/src/operations/mod.rs +++ b/grovedb/src/operations/mod.rs @@ -1,5 +1,8 @@ //! Operations for the manipulation of GroveDB state +/// Maximum number of ordinary reference indirections GroveDB follows. +pub const MAX_REFERENCE_HOPS: usize = 10; + #[cfg(feature = "minimal")] pub(crate) mod auxiliary; #[cfg(feature = "minimal")] @@ -36,4 +39,4 @@ pub mod replace_subtree_root; pub mod indexed_tree; #[cfg(feature = "minimal")] -pub use get::{QueryItemOrSumReturnType, MAX_REFERENCE_HOPS}; +pub use get::QueryItemOrSumReturnType; diff --git a/grovedb/src/operations/proof/generate.rs b/grovedb/src/operations/proof/generate.rs index 9d176506d..ccc61659b 100644 --- a/grovedb/src/operations/proof/generate.rs +++ b/grovedb/src/operations/proof/generate.rs @@ -1840,7 +1840,7 @@ impl GroveDb { // multi-layer accounting (if any) reflects the consumed // slots. let limit_u64 = path_query.query.limit.map(|l| l as u64); - let prove_result = cost_return_on_error!( + let mut prove_result = cost_return_on_error!( &mut cost, subtree .prove_count_offset_on_range( @@ -1862,6 +1862,79 @@ impl GroveDb { e ))) ); + // This short-circuit returns before the regular reference rewrite + // below, so resolve ordinary user references here and emit the + // same combined-reference node family. Indexed secondary rows do + // not use this path: their logical origin is the indexed primary + // and their dedicated envelope resolves them separately. + for op in prove_result.ops.iter_mut() { + let node = match op { + Op::Push(node) | Op::PushInverted(node) => node, + _ => continue, + }; + let Node::KVValueHashFeatureType(key, value, _, feature_type) = node else { + continue; + }; + let elem = match Element::deserialize(value, grove_version) { + Ok(element) => element.into_underlying(), + Err(_) => continue, + }; + let (Element::Reference(reference_path, ..) + | Element::ReferenceWithSumItem(reference_path, ..)) = elem + else { + continue; + }; + let absolute_path = cost_return_on_error_no_add!( + cost, + path_from_reference_path_type( + reference_path, + &path.to_vec(), + Some(key.as_slice()), + ) + .map_err(Error::from) + ); + let referenced_elem = cost_return_on_error!( + &mut cost, + self.follow_reference( + absolute_path.as_slice().into(), + true, + None, + grove_version, + ) + ); + let serialized_referenced_elem = cost_return_on_error_no_add!( + cost, + referenced_elem + .serialize(grove_version) + .map_err(|_| Error::CorruptedData("unable to serialize element".into())) + ); + let reference_element_hash = value_hash(value).unwrap_add_cost(&mut cost); + *node = match feature_type { + TreeFeatureType::ProvableCountedAndProvableSummedMerkNode(count, sum) => { + Node::KVRefValueHashCountSum( + key.to_owned(), + serialized_referenced_elem, + reference_element_hash, + *count, + *sum, + ) + } + TreeFeatureType::ProvableCountedMerkNode(count) => Node::KVRefValueHashCount( + key.to_owned(), + serialized_referenced_elem, + reference_element_hash, + *count, + ), + other => { + return Err(Error::CorruptedData(format!( + "count-offset proof: reference row {} carries non-count feature type \ + {other:?}", + hex::encode(key) + ))) + .wrap_with_cost(cost); + } + }; + } let mut serialized = Vec::with_capacity(128); encode_into(prove_result.ops.iter(), &mut serialized); // Apply consumed limit slots to the outer accounting. diff --git a/grovedb/src/operations/proof/indexed_axis/envelope.rs b/grovedb/src/operations/proof/indexed_axis/envelope.rs index 4f420b1b7..c96064f45 100644 --- a/grovedb/src/operations/proof/indexed_axis/envelope.rs +++ b/grovedb/src/operations/proof/indexed_axis/envelope.rs @@ -9,6 +9,8 @@ use bincode::{Decode, Encode}; use grovedb_element::indexed::IndexAxis; use grovedb_merk::tree::CryptoHash; +use crate::IndexedAxisEntry; + /// Per-ancestor attestation for chaining the cidx/psit/pcpsit layer /// composition during verification. /// @@ -29,7 +31,7 @@ use grovedb_merk::tree::CryptoHash; /// The carried list is the *canonical* axes list of the ancestor /// (sorted by tag ascending, 1..=3 entries) with each tag mapped to /// the secondary's root hash at proof time. -#[derive(Encode, Decode, Debug, Clone)] +#[derive(Encode, Decode, Debug, Clone, PartialEq, Eq)] pub enum AncestorAttestation { /// Regular tree ancestor. NotIndexed, @@ -41,6 +43,88 @@ pub enum AncestorAttestation { MultiAxis(Vec<(u8, [u8; 32])>), } +/// How a resolved target node's serialized element bytes are bound to +/// the value hash committed by its parent Merk. +/// +/// References bind to the next node in the witness chain. Tree variants +/// carry the child commitment needed to authenticate their serialized +/// element bytes. Simple item-like values commit as `H(value)`. +#[derive(Encode, Decode, Debug, Clone, PartialEq, Eq)] +pub enum IndexedTargetCommitment { + /// Item-like value whose committed value hash is `H(value)`. + Simple, + /// Ordinary Merk-backed or non-Merk tree, committed as + /// `combine_hash(H(value), child_root_hash)`. + Layered([u8; 32]), + /// PCIT / PSIT, committed as + /// `combine_hash_three(H(value), primary_root_hash, + /// secondary_root_hash)`. + IndexedSingle { + /// Root hash of the indexed tree's primary Merk. + primary_root_hash: [u8; 32], + /// Root hash of its only secondary Merk. + secondary_root_hash: [u8; 32], + }, + /// PCPSIT, committed as + /// `combine_hash_three(H(value), primary_root_hash, + /// axes_digest(axes))`. + IndexedMulti { + /// Root hash of the indexed tree's primary Merk. + primary_root_hash: [u8; 32], + /// Canonical `(axis_tag, secondary_root_hash)` list. + axes: Vec<(u8, [u8; 32])>, + }, + /// Reference node. Its committed value hash is reconstructed from + /// `H(value)` and the terminal target's committed value hash. + Reference, +} + +/// Root authentication carried only after an indexed row leaves its immediate +/// primary node by following an ordinary GroveDB reference. +/// +/// Direct primary values are already authenticated by the secondary row's +/// combined target hash. Reference-chain nodes live elsewhere in the grove, so +/// they retain the ordinary layer proof needed to authenticate their location +/// and stored commitment against the reconstructed GroveDB root. +#[derive(Encode, Decode, Debug, Clone, PartialEq, Eq)] +pub struct IndexedTargetAuthentication { + /// Single-key Merk proof per segment of the derived qualified path. + pub layer_proofs: Vec>, + /// Composition metadata for the ancestors above the node's parent Merk. + pub ancestor_attestations: Vec, +} + +/// One commitment-authenticated node in a resolved indexed-axis target chain. +/// +/// The first node's committed value hash is bound directly by the canonical +/// secondary reference row. Nodes reached after following a reference retain +/// root authentication because they live outside that immediate row binding; +/// their paths are derived from the primary key and serialized reference values +/// rather than repeated in the wire format. +#[derive(Encode, Decode, Debug, Clone, PartialEq, Eq)] +pub struct IndexedTargetNodeWitness { + /// Serialized element bytes committed by this node. + pub value: Vec, + /// Commitment shape for this node. + pub commitment: IndexedTargetCommitment, + /// `None` for the row-authenticated immediate primary node; `Some` for + /// nodes reached by following an ordinary reference out of that row. + pub authentication: Option, +} + +/// Shape-complete compact witness that starts at the immediate primary row +/// and follows any ordinary GroveDB references to the terminal value. +/// +/// Authentication starts at the secondary row's combined-reference hash. +/// Ordinary direct values therefore carry no repeated primary inclusion proof; +/// only rows whose immediate primary value is itself a reference pay for the +/// external chain's root authentication. +#[derive(Encode, Decode, Debug, Clone, PartialEq, Eq)] +pub struct IndexedTargetWitness { + /// Immediate primary node followed by zero or more reference targets. + pub nodes: Vec, +} + /// Wire-format envelope for a range / top-k / arbitrary-query proof /// over an indexed-tree's per-axis secondary index. #[derive(Encode, Decode, Debug)] @@ -79,6 +163,9 @@ pub struct IndexedAxisRangeProof { pub target_is_pcpsit: bool, /// Encoded Merk range proof for the per-axis secondary. pub secondary_proof: Vec, + /// One immediate-primary/terminal-resolution witness per returned + /// secondary row, in the secondary proof's result order. + pub target_witnesses: Vec, /// Echoed query limit (preserves `None`-vs-`Some(0)` semantics). pub requested_limit: Option, /// Echoed iteration direction. `false` = ascending, `true` = @@ -114,6 +201,8 @@ pub struct IndexedAxisPaginatedProof { /// `prove_count_offset_on_range`-produced `Vec` stream (every /// axis's secondary carries a provable count). pub secondary_proof: Vec, + /// Same as [`IndexedAxisRangeProof::target_witnesses`]. + pub target_witnesses: Vec, /// Echoed pagination parameters. pub requested_k: u16, /// Echoed offset. @@ -165,12 +254,12 @@ pub struct IndexedAxisAggregateProof { /// [`IndexedAxisQueryResult::entries`]. #[derive(Debug, Clone, PartialEq, Eq)] pub enum AxisEntries { - /// Count-axis entries: `(count_value, original_key)`. - Count(Vec<(u64, Vec)>), - /// Sum-axis entries: `(sum_value, original_key)`. - Sum(Vec<(i64, Vec)>), - /// Avg-axis entries: `(avg_fixed_point_i128, original_key)`. - Avg(Vec<(i128, Vec)>), + /// Count-axis entries. + Count(Vec>), + /// Sum-axis entries. + Sum(Vec>), + /// Avg-axis entries. + Avg(Vec>), } impl AxisEntries { @@ -197,9 +286,11 @@ impl AxisEntries { /// item of a `k = 1` page, used by rank verification. pub fn first_original_key(&self) -> Option<&[u8]> { match self { - AxisEntries::Count(entries) => entries.first().map(|(_, key)| key.as_slice()), - AxisEntries::Sum(entries) => entries.first().map(|(_, key)| key.as_slice()), - AxisEntries::Avg(entries) => entries.first().map(|(_, key)| key.as_slice()), + AxisEntries::Count(entries) => { + entries.first().map(|entry| entry.primary_key.as_slice()) + } + AxisEntries::Sum(entries) => entries.first().map(|entry| entry.primary_key.as_slice()), + AxisEntries::Avg(entries) => entries.first().map(|entry| entry.primary_key.as_slice()), } } diff --git a/grovedb/src/operations/proof/indexed_axis/generate.rs b/grovedb/src/operations/proof/indexed_axis/generate.rs index 432f8e849..cce9b9243 100644 --- a/grovedb/src/operations/proof/indexed_axis/generate.rs +++ b/grovedb/src/operations/proof/indexed_axis/generate.rs @@ -8,10 +8,19 @@ use grovedb_costs::{ cost_return_on_error, cost_return_on_error_no_add, CostResult, CostsExt, OperationCost, }; +use std::collections::HashSet; + use grovedb_element::indexed::IndexAxis; use grovedb_merk::{ element::get::ElementFetchFromStorageExtensions, - proofs::{encode_into, query::QueryItem as MerkQueryItemForRange, Query as MerkQuery}, + proofs::{ + encode_into, + query::{ + verify_count_offset_on_range_proof, QueryItem as MerkQueryItemForRange, + QueryProofVerify, + }, + Query as MerkQuery, + }, }; use grovedb_path::{SubtreePath, SubtreePathBuilder}; use grovedb_query::{AggregateFold, QueryItem as MerkQueryItem}; @@ -23,9 +32,9 @@ use crate::{util::TxRef, Element, Error, GroveDb, Transaction, TransactionArg}; use super::{ verify::{count_aggregate_inner_range, sum_aggregate_inner_range}, AncestorAttestation, IndexedAxisAggregateProof, IndexedAxisPaginatedProof, - IndexedAxisRangeProof, + IndexedAxisRangeProof, IndexedTargetCommitment, IndexedTargetNodeWitness, IndexedTargetWitness, }; -use crate::operations::proof::AxisDescentProof; +use crate::operations::{proof::AxisDescentProof, MAX_REFERENCE_HOPS}; /// Build the per-ancestor attestation list for a path of length N: the /// list has length N-1 (one entry per intermediate layer). For each @@ -198,6 +207,359 @@ fn build_layer_proofs<'db>( Ok(layer_proofs).wrap_with_cost(cost) } +/// Build the commitment-shape attestation for one target node. +fn build_target_commitment<'db>( + grovedb: &'db GroveDb, + element: &Element, + qualified_path: &[Vec], + transaction: &'db Transaction, + batch: &'db StorageBatch, + grove_version: &GroveVersion, +) -> CostResult { + let mut cost = OperationCost::default(); + let underlying = element.underlying(); + + if underlying.is_reference() { + return Ok(IndexedTargetCommitment::Reference).wrap_with_cost(cost); + } + + let target_path_owned: SubtreePathBuilder> = + SubtreePathBuilder::owned_from_iter(qualified_path.iter().cloned()); + let target_path = SubtreePath::from(&target_path_owned); + + match underlying { + Element::ProvableCountIndexedTree(_, secondary_root_key, ..) => { + let primary = cost_return_on_error!( + &mut cost, + grovedb.open_transactional_merk_at_path( + target_path.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (primary_root_hash, _, _) = cost_return_on_error!( + &mut cost, + primary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + let secondary = cost_return_on_error!( + &mut cost, + grovedb.open_indexed_secondary_at_path( + target_path, + IndexAxis::Count, + secondary_root_key.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (secondary_root_hash, _, _) = cost_return_on_error!( + &mut cost, + secondary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + Ok(IndexedTargetCommitment::IndexedSingle { + primary_root_hash, + secondary_root_hash, + }) + .wrap_with_cost(cost) + } + Element::ProvableSumIndexedTree(_, secondary_root_key, ..) => { + let primary = cost_return_on_error!( + &mut cost, + grovedb.open_transactional_merk_at_path( + target_path.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (primary_root_hash, _, _) = cost_return_on_error!( + &mut cost, + primary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + let secondary = cost_return_on_error!( + &mut cost, + grovedb.open_indexed_secondary_at_path( + target_path, + IndexAxis::Sum, + secondary_root_key.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (secondary_root_hash, _, _) = cost_return_on_error!( + &mut cost, + secondary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + Ok(IndexedTargetCommitment::IndexedSingle { + primary_root_hash, + secondary_root_hash, + }) + .wrap_with_cost(cost) + } + Element::ProvableCountProvableSumIndexedTree(_, _, _, axes, _) => { + let primary = cost_return_on_error!( + &mut cost, + grovedb.open_transactional_merk_at_path( + target_path.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (primary_root_hash, _, _) = cost_return_on_error!( + &mut cost, + primary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + let mut axis_hashes = Vec::with_capacity(axes.len()); + for (tag, secondary_root_key) in axes { + let axis = cost_return_on_error_no_add!( + cost, + IndexAxis::try_from_tag(*tag).map_err(|e| Error::CorruptedData(format!( + "indexed target witness: invalid PCPSIT axis tag: {e}" + ))) + ); + let secondary = cost_return_on_error!( + &mut cost, + grovedb.open_indexed_secondary_at_path( + target_path.clone(), + axis, + secondary_root_key.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (secondary_root_hash, _, _) = cost_return_on_error!( + &mut cost, + secondary + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + axis_hashes.push((*tag, secondary_root_hash)); + } + Ok(IndexedTargetCommitment::IndexedMulti { + primary_root_hash, + axes: axis_hashes, + }) + .wrap_with_cost(cost) + } + _ if underlying.is_any_tree() => { + let child_merk = cost_return_on_error!( + &mut cost, + grovedb.open_transactional_merk_at_path( + target_path.clone(), + transaction, + Some(batch), + grove_version, + ) + ); + let (merk_root_hash, _, _) = cost_return_on_error!( + &mut cost, + child_merk + .root_hash_key_and_aggregate_data() + .map_err(Error::MerkError) + ); + let child_root_hash = if underlying.uses_non_merk_data_storage() { + grovedb.compute_non_merk_child_hash( + underlying, + target_path, + transaction, + merk_root_hash, + ) + } else { + merk_root_hash + }; + Ok(IndexedTargetCommitment::Layered(child_root_hash)).wrap_with_cost(cost) + } + _ => Ok(IndexedTargetCommitment::Simple).wrap_with_cost(cost), + } +} + +/// Carry the immediate primary node and every ordinary reference hop until +/// the terminal value. +/// +/// The secondary row already commits the immediate node's exact stored value +/// hash. Carrying a full GroveDB inclusion proof for every direct primary value +/// therefore re-proves data the row already authenticates and makes top-k +/// proofs grow by hundreds of bytes per result. The compact witness omits that +/// redundant proof. If the primary is itself a reference, nodes reached outside +/// the immediate row retain root authentication; their paths are derived from +/// authenticated reference elements rather than repeated in the wire format. +fn build_indexed_target_witness<'db>( + grovedb: &'db GroveDb, + indexed_path: &[Vec], + primary_key: &[u8], + transaction: &'db Transaction, + batch: &'db StorageBatch, + grove_version: &GroveVersion, +) -> CostResult { + let mut cost = OperationCost::default(); + let mut qualified_path = indexed_path.to_vec(); + qualified_path.push(primary_key.to_vec()); + let mut visited = HashSet::new(); + let mut nodes = Vec::new(); + + for _ in 0..=MAX_REFERENCE_HOPS { + if !visited.insert(qualified_path.clone()) { + return Err(Error::CyclicReference).wrap_with_cost(cost); + } + let Some((key, parent_segments)) = qualified_path.split_last() else { + return Err(Error::CorruptedPath( + "indexed target witness resolved an empty path".to_string(), + )) + .wrap_with_cost(cost); + }; + let parent_slices: Vec<&[u8]> = parent_segments.iter().map(Vec::as_slice).collect(); + let parent_path: SubtreePath<&[u8]> = parent_slices.as_slice().into(); + let parent_merk = cost_return_on_error!( + &mut cost, + grovedb.open_transactional_merk_at_path( + parent_path, + transaction, + Some(batch), + grove_version, + ) + ); + let (element, _) = cost_return_on_error!( + &mut cost, + Element::get_with_value_hash(&parent_merk, key, true, grove_version) + .map_err(Error::MerkError) + ); + let value = cost_return_on_error_no_add!( + cost, + element + .serialize(grove_version) + .map_err(Error::ElementError) + ); + let commitment = cost_return_on_error!( + &mut cost, + build_target_commitment( + grovedb, + &element, + &qualified_path, + transaction, + batch, + grove_version, + ) + ); + let authentication = if nodes.is_empty() { + None + } else { + let layer_proofs = cost_return_on_error!( + &mut cost, + build_layer_proofs( + grovedb, + &qualified_path, + transaction, + batch, + grove_version, + "indexed reference-chain target", + ) + ); + let ancestor_attestations = cost_return_on_error!( + &mut cost, + build_ancestor_attestations( + grovedb, + &qualified_path, + transaction, + batch, + grove_version, + "indexed reference-chain target", + ) + ); + Some(super::IndexedTargetAuthentication { + layer_proofs, + ancestor_attestations, + }) + }; + + let next_path = match element.underlying() { + Element::Reference(reference_path, ..) + | Element::ReferenceWithSumItem(reference_path, ..) => { + let parent_builder: SubtreePathBuilder> = + SubtreePathBuilder::owned_from_iter(parent_segments.iter().cloned()); + Some(cost_return_on_error_no_add!( + cost, + reference_path + .clone() + .absolute_qualified_path(parent_builder, key) + .map(|p| p.to_vec()) + .map_err(Error::ElementError) + )) + } + _ => None, + }; + + nodes.push(IndexedTargetNodeWitness { + value, + commitment, + authentication, + }); + match next_path { + Some(next) => qualified_path = next, + None => return Ok(IndexedTargetWitness { nodes }).wrap_with_cost(cost), + } + } + + Err(Error::ReferenceLimit).wrap_with_cost(cost) +} + +fn primary_key_from_secondary_key(axis: IndexAxis, key: &[u8]) -> Result, Error> { + let prefix_len = match axis { + IndexAxis::Count | IndexAxis::Sum => 8, + IndexAxis::Avg => 16, + }; + if key.len() < prefix_len { + return Err(Error::CorruptedData(format!( + "indexed-axis secondary key shorter than its {prefix_len}-byte sort prefix" + ))); + } + Ok(key[prefix_len..].to_vec()) +} + +fn build_target_witnesses<'db>( + grovedb: &'db GroveDb, + indexed_path: &[Vec], + axis: IndexAxis, + secondary_keys: impl IntoIterator>, + transaction: &'db Transaction, + batch: &'db StorageBatch, + grove_version: &GroveVersion, +) -> CostResult, Error> { + let mut cost = OperationCost::default(); + let mut witnesses = Vec::new(); + for secondary_key in secondary_keys { + let primary_key = cost_return_on_error_no_add!( + cost, + primary_key_from_secondary_key(axis, &secondary_key) + ); + witnesses.push(cost_return_on_error!( + &mut cost, + build_indexed_target_witness( + grovedb, + indexed_path, + &primary_key, + transaction, + batch, + grove_version, + ) + )); + } + Ok(witnesses).wrap_with_cost(cost) +} + /// Path-keys-driven variant of `read_queried_axis_info`. For PCPSIT, also /// opens each non-queried axis's secondary to capture its root hash. /// @@ -856,6 +1218,7 @@ impl GroveDb { ); let descending = !secondary_query.left_to_right; let requested_limit = limit; + let verification_query = secondary_query.clone(); let sec_result = cost_return_on_error!( &mut cost, secondary_merk @@ -864,6 +1227,26 @@ impl GroveDb { "indexed-axis range proof: secondary range proof: {e}" ))) ); + let (_, verified) = cost_return_on_error!( + &mut cost, + verification_query + .execute_proof(&sec_result.proof, limit, !descending, 0) + .map_err(|e| Error::CorruptedData(format!( + "indexed-axis range proof: replay generated secondary proof: {e}" + ))) + ); + let target_witnesses = cost_return_on_error!( + &mut cost, + build_target_witnesses( + self, + &path_keys, + axis, + verified.result_set.into_iter().map(|entry| entry.key), + transaction, + batch, + grove_version, + ) + ); Ok(IndexedAxisRangeProof { axis_tag: axis.tag(), @@ -873,6 +1256,7 @@ impl GroveDb { other_axes_root_hashes, target_is_pcpsit, secondary_proof: sec_result.proof, + target_witnesses, requested_limit, descending, }) @@ -1009,6 +1393,31 @@ impl GroveDb { ); let mut serialized = Vec::with_capacity(128); encode_into(prove_result.ops.iter(), &mut serialized); + let verified = cost_return_on_error!( + &mut cost, + verify_count_offset_on_range_proof( + &serialized, + &inner_range, + offset, + Some(k as u64), + !descending, + ) + .map_err(|e| Error::CorruptedData(format!( + "indexed-axis paginated proof: replay generated secondary proof: {e}" + ))) + ); + let target_witnesses = cost_return_on_error!( + &mut cost, + build_target_witnesses( + self, + &path_keys, + axis, + verified.returned_items.into_iter().map(|entry| entry.key), + transaction, + batch, + grove_version, + ) + ); Ok(IndexedAxisPaginatedProof { axis_tag: axis.tag(), @@ -1018,6 +1427,7 @@ impl GroveDb { other_axes_root_hashes, target_is_pcpsit, secondary_proof: serialized, + target_witnesses, requested_k: k, requested_offset: offset, descending, @@ -1333,6 +1743,84 @@ impl GroveDb { ) } }; + let secondary_keys = match &axis_query.traversal { + AxisTraversal::RankedPage { k, offset } => { + let inner_range = MerkQueryItemForRange::RangeFull(std::ops::RangeFull); + cost_return_on_error!( + &mut cost, + verify_count_offset_on_range_proof( + &secondary_proof, + &inner_range, + *offset, + Some(*k as u64), + !axis_query.descending, + ) + .map_err(|e| Error::CorruptedData(format!( + "axis descent: replay generated paginated proof: {e}" + ))) + ) + .returned_items + .into_iter() + .map(|entry| entry.key) + .collect() + } + AxisTraversal::RankOfKey { .. } => { + let inner_range = MerkQueryItemForRange::RangeFull(std::ops::RangeFull); + let rank_offset = rank.expect("set above for RankOfKey"); + cost_return_on_error!( + &mut cost, + verify_count_offset_on_range_proof( + &secondary_proof, + &inner_range, + rank_offset, + Some(1), + !axis_query.descending, + ) + .map_err(|e| Error::CorruptedData(format!( + "axis descent: replay generated rank proof: {e}" + ))) + ) + .returned_items + .into_iter() + .map(|entry| entry.key) + .collect() + } + AxisTraversal::Bounded { limit, .. } if !secondary_proof.is_empty() => { + let secondary_query = cost_return_on_error_no_add!( + cost, + crate::query::axis_lowering::axis_bounded_merk_query(axis_query) + ); + let left_to_right = secondary_query.left_to_right; + cost_return_on_error!( + &mut cost, + secondary_query + .execute_proof(&secondary_proof, Some(*limit), left_to_right, 0) + .map_err(|e| Error::CorruptedData(format!( + "axis descent: replay generated bounded proof: {e}" + ))) + ) + .1 + .result_set + .into_iter() + .map(|entry| entry.key) + .collect() + } + AxisTraversal::Bounded { .. } | AxisTraversal::AggregateOverValueRange { .. } => { + Vec::new() + } + }; + let target_witnesses = cost_return_on_error!( + &mut cost, + build_target_witnesses( + self, + &path_keys, + axis, + secondary_keys, + transaction, + batch, + grove_version, + ) + ); Ok(AxisDescentProof { axis_tag: axis.tag(), @@ -1341,6 +1829,7 @@ impl GroveDb { primary_root_hash, rank, secondary_proof, + target_witnesses, }) .wrap_with_cost(cost) } diff --git a/grovedb/src/operations/proof/indexed_axis/mod.rs b/grovedb/src/operations/proof/indexed_axis/mod.rs index 7508c14f7..e683e125a 100644 --- a/grovedb/src/operations/proof/indexed_axis/mod.rs +++ b/grovedb/src/operations/proof/indexed_axis/mod.rs @@ -70,7 +70,8 @@ pub(crate) mod verify; pub use envelope::{ AncestorAttestation, AxisEntries, IndexedAxisAggregateProof, IndexedAxisAggregateResult, IndexedAxisPaginatedProof, IndexedAxisPaginatedResult, IndexedAxisQueryResult, - IndexedAxisRangeProof, + IndexedAxisRangeProof, IndexedTargetAuthentication, IndexedTargetCommitment, + IndexedTargetNodeWitness, IndexedTargetWitness, }; use grovedb_element::indexed::IndexAxis; diff --git a/grovedb/src/operations/proof/indexed_axis/verify.rs b/grovedb/src/operations/proof/indexed_axis/verify.rs index 16f0d6328..1541fa5b3 100644 --- a/grovedb/src/operations/proof/indexed_axis/verify.rs +++ b/grovedb/src/operations/proof/indexed_axis/verify.rs @@ -24,14 +24,39 @@ use grovedb_merk::{ use grovedb_query::{AggregateFold, QueryItem as MerkQueryItem}; use grovedb_version::{check_grovedb_v0, version::GroveVersion}; -use crate::{Error, GroveDb}; +use crate::{operations::MAX_REFERENCE_HOPS, Element, Error, GroveDb, IndexedAxisEntry}; use super::{ aggregate_range_out_of_domain, AncestorAttestation, AxisEntries, IndexedAxisAggregateProof, IndexedAxisAggregateResult, IndexedAxisPaginatedProof, IndexedAxisPaginatedResult, - IndexedAxisQueryResult, IndexedAxisRangeProof, + IndexedAxisQueryResult, IndexedAxisRangeProof, IndexedTargetCommitment, IndexedTargetWitness, }; +#[derive(Debug)] +pub(crate) struct ProvenAxisRow { + ordering_value: T, + primary_key: Vec, + row_bytes: Vec, + row_value_hash: CryptoHash, +} + +#[derive(Debug)] +pub(crate) enum ProvenAxisRows { + Count(Vec>), + Sum(Vec>), + Avg(Vec>), +} + +impl ProvenAxisRows { + pub(crate) fn empty_for_axis(axis: IndexAxis) -> Self { + match axis { + IndexAxis::Count => Self::Count(Vec::new()), + IndexAxis::Sum => Self::Sum(Vec::new()), + IndexAxis::Avg => Self::Avg(Vec::new()), + } + } +} + /// Walk the verifier-side ancestor chain (depths `last_idx - 1` down to /// `0`) and return the final reconstructed root hash. Returns the /// outer GroveDB root hash on success. @@ -296,6 +321,372 @@ fn execute_single_key_proof( Ok((value, root_hash, proved.proof)) } +struct VerifiedTargetNode { + value_bytes: Vec, + element: Element, + recorded_value_hash: Option, +} + +fn verify_indexed_target_witness( + witness: &IndexedTargetWitness, + indexed_path: &[&[u8]], + primary_key: &[u8], + grove_root_hash: &CryptoHash, + grove_version: &GroveVersion, +) -> Result<(Element, CryptoHash, Element), Error> { + if witness.nodes.is_empty() { + return Err(Error::CorruptedData( + "indexed target witness contains no nodes".to_string(), + )); + } + if witness.nodes.len() > MAX_REFERENCE_HOPS + 1 { + return Err(Error::ReferenceLimit); + } + + let mut current_path: Vec> = indexed_path + .iter() + .map(|segment| segment.to_vec()) + .collect(); + current_path.push(primary_key.to_vec()); + + let mut seen_paths = std::collections::HashSet::new(); + let mut verified_nodes = Vec::with_capacity(witness.nodes.len()); + for (index, node) in witness.nodes.iter().enumerate() { + if !seen_paths.insert(current_path.clone()) { + return Err(Error::CyclicReference); + } + let recorded_value_hash = match (index, &node.authentication) { + (0, None) => None, + (0, Some(_)) => { + return Err(Error::CorruptedData( + "indexed target witness redundantly authenticates its immediate primary node" + .to_string(), + )); + } + (_, None) => { + return Err(Error::CorruptedData(format!( + "indexed reference-chain target node {index} has no root authentication" + ))); + } + (_, Some(authentication)) => { + if authentication.layer_proofs.len() != current_path.len() { + return Err(Error::CorruptedData(format!( + "indexed reference-chain target node {index} has {} layer proofs for a \ + {}-segment path", + authentication.layer_proofs.len(), + current_path.len() + ))); + } + let key = current_path.last().expect("initial path is non-empty"); + let deepest = authentication.layer_proofs.len() - 1; + let (proved_value, node_parent_root, recorded_value_hash) = + execute_single_key_proof( + &authentication.layer_proofs[deepest], + key, + "indexed reference-chain target", + )?; + if proved_value != node.value { + return Err(Error::CorruptedData(format!( + "indexed reference-chain target node {index} value differs from its root \ + proof" + ))); + } + let path_slices: Vec<&[u8]> = current_path.iter().map(Vec::as_slice).collect(); + let reconstructed_root = walk_ancestor_chain( + &authentication.layer_proofs, + &authentication.ancestor_attestations, + &path_slices, + node_parent_root, + "indexed reference-chain target", + )?; + if reconstructed_root != *grove_root_hash { + return Err(Error::CorruptedData(format!( + "indexed reference-chain target node {index} reconstructs GroveDB root \ + {}, expected {}", + hex::encode(reconstructed_root), + hex::encode(grove_root_hash) + ))); + } + Some(recorded_value_hash) + } + }; + let element = Element::deserialize(&node.value, grove_version).map_err(|e| { + Error::CorruptedData(format!( + "indexed target witness node {index} contains an invalid element: {e}" + )) + })?; + + match element.underlying() { + Element::Reference(reference_path, ..) + | Element::ReferenceWithSumItem(reference_path, ..) => { + if index + 1 == witness.nodes.len() { + return Err(Error::CorruptedData( + "indexed target witness terminates at a reference".to_string(), + )); + } + let Some((key, parent_segments)) = current_path.split_last() else { + return Err(Error::CorruptedData( + "indexed target witness reference has an empty path".to_string(), + )); + }; + let parent_builder: grovedb_path::SubtreePathBuilder> = + grovedb_path::SubtreePathBuilder::owned_from_iter( + parent_segments.iter().cloned(), + ); + current_path = reference_path + .clone() + .absolute_qualified_path(parent_builder, key) + .map_err(Error::ElementError)? + .to_vec(); + } + _ if index + 1 != witness.nodes.len() => { + return Err(Error::CorruptedData(format!( + "indexed target witness continues after terminal node {index}" + ))); + } + _ => {} + } + verified_nodes.push(VerifiedTargetNode { + value_bytes: node.value.clone(), + element, + recorded_value_hash, + }); + } + + let mut committed_hashes = vec![[0u8; 32]; verified_nodes.len()]; + for index in (0..verified_nodes.len()).rev() { + let wire_node = &witness.nodes[index]; + let verified = &verified_nodes[index]; + let serialized_hash = value_hash(&verified.value_bytes).value().to_owned(); + let underlying = verified.element.underlying(); + let expected_hash = match &wire_node.commitment { + IndexedTargetCommitment::Simple => { + if underlying.is_reference() || underlying.is_any_tree() { + return Err(Error::CorruptedData(format!( + "indexed target witness node {index} claims a simple commitment for {}", + underlying.type_str() + ))); + } + if index + 1 != verified_nodes.len() { + return Err(Error::CorruptedData(format!( + "indexed target witness continues after terminal node {index}" + ))); + } + serialized_hash + } + IndexedTargetCommitment::Layered(child_root_hash) => { + if !underlying.is_any_tree() || underlying.is_indexed_tree() { + return Err(Error::CorruptedData(format!( + "indexed target witness node {index} claims a layered commitment for {}", + underlying.type_str() + ))); + } + if index + 1 != verified_nodes.len() { + return Err(Error::CorruptedData(format!( + "indexed target witness continues after terminal node {index}" + ))); + } + combine_hash(&serialized_hash, child_root_hash) + .value() + .to_owned() + } + IndexedTargetCommitment::IndexedSingle { + primary_root_hash, + secondary_root_hash, + } => { + if !matches!( + underlying, + Element::ProvableCountIndexedTree(..) | Element::ProvableSumIndexedTree(..) + ) { + return Err(Error::CorruptedData(format!( + "indexed target witness node {index} claims a single-axis indexed \ + commitment for {}", + underlying.type_str() + ))); + } + if index + 1 != verified_nodes.len() { + return Err(Error::CorruptedData(format!( + "indexed target witness continues after terminal node {index}" + ))); + } + combine_hash_three(&serialized_hash, primary_root_hash, secondary_root_hash) + .value() + .to_owned() + } + IndexedTargetCommitment::IndexedMulti { + primary_root_hash, + axes, + } => { + let Element::ProvableCountProvableSumIndexedTree(_, _, _, configured_axes, _) = + underlying + else { + return Err(Error::CorruptedData(format!( + "indexed target witness node {index} claims a multi-axis indexed \ + commitment for {}", + underlying.type_str() + ))); + }; + if axes.len() != configured_axes.len() + || axes + .iter() + .zip(configured_axes) + .any(|((got_tag, _), (want_tag, _))| got_tag != want_tag) + { + return Err(Error::CorruptedData(format!( + "indexed target witness node {index} axes do not match its PCPSIT element" + ))); + } + if index + 1 != verified_nodes.len() { + return Err(Error::CorruptedData(format!( + "indexed target witness continues after terminal node {index}" + ))); + } + let digest = axes_digest(axes).value().to_owned(); + combine_hash_three(&serialized_hash, primary_root_hash, &digest) + .value() + .to_owned() + } + IndexedTargetCommitment::Reference => { + match underlying { + Element::Reference(..) | Element::ReferenceWithSumItem(..) => {} + _ => { + return Err(Error::CorruptedData(format!( + "indexed target witness node {index} claims a reference commitment \ + for {}", + underlying.type_str() + ))); + } + } + let Some(terminal_hash) = committed_hashes.last() else { + return Err(Error::CorruptedData( + "indexed target witness terminates at a reference".to_string(), + )); + }; + combine_hash(&serialized_hash, terminal_hash) + .value() + .to_owned() + } + }; + if let Some(recorded_value_hash) = verified.recorded_value_hash + && expected_hash != recorded_value_hash + { + return Err(Error::CorruptedData(format!( + "indexed reference-chain target node {index} commitment mismatch: computed {}, \ + root proof records {}", + hex::encode(expected_hash), + hex::encode(recorded_value_hash) + ))); + } + committed_hashes[index] = expected_hash; + } + + let immediate = verified_nodes.first().expect("checked non-empty"); + let terminal = verified_nodes.last().expect("checked non-empty"); + if terminal.element.underlying().is_reference() { + return Err(Error::CorruptedData( + "indexed target witness did not resolve to a terminal value".to_string(), + )); + } + Ok(( + immediate.element.clone(), + committed_hashes[0], + terminal.element.clone().into_underlying(), + )) +} + +fn resolve_axis_rows( + axis: IndexAxis, + rows: Vec>, + witnesses: &[IndexedTargetWitness], + indexed_path: &[&[u8]], + grove_root_hash: &CryptoHash, + grove_version: &GroveVersion, +) -> Result>, Error> { + if rows.len() != witnesses.len() { + return Err(Error::CorruptedData(format!( + "indexed-axis proof returned {} secondary rows but carries {} target witnesses", + rows.len(), + witnesses.len() + ))); + } + let mut entries = Vec::with_capacity(rows.len()); + for (row, witness) in rows.into_iter().zip(witnesses) { + let (immediate, immediate_value_hash, terminal) = verify_indexed_target_witness( + witness, + indexed_path, + &row.primary_key, + grove_root_hash, + grove_version, + )?; + let (count, sum) = immediate.count_sum_value_or_default(); + let expected_row = + grovedb_element::canonical_axis_reference(axis, &row.primary_key, count, sum) + .map_err(Error::ElementError)?; + let expected_row_bytes = expected_row + .serialize(grove_version) + .map_err(Error::ElementError)?; + if row.row_bytes != expected_row_bytes { + return Err(Error::CorruptedData(format!( + "indexed-axis secondary row for primary key {} is not the canonical one-hop \ + reference", + hex::encode(&row.primary_key) + ))); + } + let row_hash = value_hash(&row.row_bytes).value().to_owned(); + let expected_secondary_value_hash = combine_hash(&row_hash, &immediate_value_hash) + .value() + .to_owned(); + if row.row_value_hash != expected_secondary_value_hash { + return Err(Error::CorruptedData(format!( + "indexed-axis secondary reference for primary key {} is stale or bound to the \ + wrong immediate primary value hash", + hex::encode(&row.primary_key) + ))); + } + entries.push(IndexedAxisEntry { + ordering_value: row.ordering_value, + primary_key: row.primary_key, + value: terminal, + }); + } + Ok(entries) +} + +pub(crate) fn resolve_indexed_axis_rows( + rows: ProvenAxisRows, + witnesses: &[IndexedTargetWitness], + indexed_path: &[&[u8]], + grove_root_hash: &CryptoHash, + grove_version: &GroveVersion, +) -> Result { + match rows { + ProvenAxisRows::Count(rows) => Ok(AxisEntries::Count(resolve_axis_rows( + IndexAxis::Count, + rows, + witnesses, + indexed_path, + grove_root_hash, + grove_version, + )?)), + ProvenAxisRows::Sum(rows) => Ok(AxisEntries::Sum(resolve_axis_rows( + IndexAxis::Sum, + rows, + witnesses, + indexed_path, + grove_root_hash, + grove_version, + )?)), + ProvenAxisRows::Avg(rows) => Ok(AxisEntries::Avg(resolve_axis_rows( + IndexAxis::Avg, + rows, + witnesses, + indexed_path, + grove_root_hash, + grove_version, + )?)), + } +} + impl GroveDb { /// Verify an `IndexedAxisRangeProof`-shaped top-k proof (full range, /// limit = `expected_k`, direction = `expected_descending`). @@ -342,7 +733,7 @@ impl GroveDb { let mut full_range = MerkQuery::new(); full_range.insert_all(); full_range.left_to_right = !envelope.descending; - verify_indexed_axis_range_inner(envelope, full_range, expected_axis, path) + verify_indexed_axis_range_inner(envelope, full_range, expected_axis, path, grove_version) } /// Verify an `IndexedAxisRangeProof`-shaped arbitrary-query proof. @@ -389,7 +780,13 @@ impl GroveDb { expected_descending, envelope.descending ))); } - verify_indexed_axis_range_inner(envelope, secondary_query, expected_axis, path) + verify_indexed_axis_range_inner( + envelope, + secondary_query, + expected_axis, + path, + grove_version, + ) } /// Verify an `IndexedAxisPaginatedProof`-shaped paginated proof. @@ -444,7 +841,7 @@ impl GroveDb { expected_offset, envelope.requested_offset ))); } - verify_indexed_axis_paginated_inner(envelope, expected_axis, path) + verify_indexed_axis_paginated_inner(envelope, expected_axis, path, grove_version) } /// Verify a rank-of-key proof produced by @@ -491,9 +888,9 @@ impl GroveDb { ))); } let yielded_key: &[u8] = match &result.entries { - AxisEntries::Count(v) if v.len() == 1 => &v[0].1, - AxisEntries::Sum(v) if v.len() == 1 => &v[0].1, - AxisEntries::Avg(v) if v.len() == 1 => &v[0].1, + AxisEntries::Count(v) if v.len() == 1 => &v[0].primary_key, + AxisEntries::Sum(v) if v.len() == 1 => &v[0].primary_key, + AxisEntries::Avg(v) if v.len() == 1 => &v[0].primary_key, other => { return Err(Error::CorruptedData(format!( "indexed-axis rank proof: expected exactly one yielded entry at the rank \ @@ -609,6 +1006,7 @@ fn verify_indexed_axis_range_inner( secondary_query: MerkQuery, axis: IndexAxis, path: &[&[u8]], + grove_version: &GroveVersion, ) -> Result { if envelope.layer_proofs.len() != path.len() { return Err(Error::CorruptedData(format!( @@ -639,7 +1037,7 @@ fn verify_indexed_axis_range_inner( )) })?; - let entries = decode_axis_entries_from_result_set(axis, &sec_result.result_set)?; + let rows = decode_axis_entries_from_result_set(axis, &sec_result.result_set)?; let initial_root = verify_deepest_layer( &envelope.layer_proofs, @@ -659,6 +1057,13 @@ fn verify_indexed_axis_range_inner( initial_root, "indexed-axis range proof", )?; + let entries = resolve_indexed_axis_rows( + rows, + &envelope.target_witnesses, + path, + &root_hash, + grove_version, + )?; Ok(IndexedAxisQueryResult { root_hash, entries }) } @@ -667,6 +1072,7 @@ fn verify_indexed_axis_paginated_inner( envelope: IndexedAxisPaginatedProof, axis: IndexAxis, path: &[&[u8]], + grove_version: &GroveVersion, ) -> Result { if envelope.layer_proofs.len() != path.len() { return Err(Error::CorruptedData(format!( @@ -701,7 +1107,7 @@ fn verify_indexed_axis_paginated_inner( "indexed-axis paginated proof: secondary count-offset proof failed to verify: {e}" )) })?; - let entries = + let rows = decode_axis_entries_from_count_offset_items(axis, &count_offset_result.returned_items)?; let (secondary_root_hash, skipped) = (count_offset_result.root_hash, count_offset_result.skipped); @@ -724,6 +1130,13 @@ fn verify_indexed_axis_paginated_inner( initial_root, "indexed-axis paginated proof", )?; + let entries = resolve_indexed_axis_rows( + rows, + &envelope.target_witnesses, + path, + &root_hash, + grove_version, + )?; Ok(IndexedAxisPaginatedResult { root_hash, @@ -820,10 +1233,10 @@ fn verify_indexed_axis_aggregate_inner( pub(crate) fn decode_axis_entries_from_result_set( axis: IndexAxis, result_set: &[grovedb_merk::proofs::query::ProvedKeyOptionalValue], -) -> Result { +) -> Result { match axis { IndexAxis::Count => { - let mut entries: Vec<(u64, Vec)> = Vec::with_capacity(result_set.len()); + let mut entries = Vec::with_capacity(result_set.len()); for proved in result_set { if proved.key.len() < 8 { return Err(Error::CorruptedData(format!( @@ -833,15 +1246,21 @@ pub(crate) fn decode_axis_entries_from_result_set( } let mut count_bytes = [0u8; 8]; count_bytes.copy_from_slice(&proved.key[..8]); - entries.push(( - decode_count_sort_key(&count_bytes), - proved.key[8..].to_vec(), - )); + entries.push(ProvenAxisRow { + ordering_value: decode_count_sort_key(&count_bytes), + primary_key: proved.key[8..].to_vec(), + row_bytes: proved.value.clone().ok_or_else(|| { + Error::CorruptedData( + "indexed-axis secondary proof omitted a returned row value".to_string(), + ) + })?, + row_value_hash: proved.proof, + }); } - Ok(AxisEntries::Count(entries)) + Ok(ProvenAxisRows::Count(entries)) } IndexAxis::Sum => { - let mut entries: Vec<(i64, Vec)> = Vec::with_capacity(result_set.len()); + let mut entries = Vec::with_capacity(result_set.len()); for proved in result_set { if proved.key.len() < 8 { return Err(Error::CorruptedData(format!( @@ -851,12 +1270,21 @@ pub(crate) fn decode_axis_entries_from_result_set( } let mut sum_bytes = [0u8; 8]; sum_bytes.copy_from_slice(&proved.key[..8]); - entries.push((decode_sum_sort_key(&sum_bytes), proved.key[8..].to_vec())); + entries.push(ProvenAxisRow { + ordering_value: decode_sum_sort_key(&sum_bytes), + primary_key: proved.key[8..].to_vec(), + row_bytes: proved.value.clone().ok_or_else(|| { + Error::CorruptedData( + "indexed-axis secondary proof omitted a returned row value".to_string(), + ) + })?, + row_value_hash: proved.proof, + }); } - Ok(AxisEntries::Sum(entries)) + Ok(ProvenAxisRows::Sum(entries)) } IndexAxis::Avg => { - let mut entries: Vec<(i128, Vec)> = Vec::with_capacity(result_set.len()); + let mut entries = Vec::with_capacity(result_set.len()); for proved in result_set { if proved.key.len() < 16 { return Err(Error::CorruptedData(format!( @@ -866,12 +1294,18 @@ pub(crate) fn decode_axis_entries_from_result_set( } let mut avg_bytes = [0u8; 16]; avg_bytes.copy_from_slice(&proved.key[..16]); - entries.push(( - grovedb_element::indexed::decode_avg_sort_key(&avg_bytes), - proved.key[16..].to_vec(), - )); + entries.push(ProvenAxisRow { + ordering_value: grovedb_element::indexed::decode_avg_sort_key(&avg_bytes), + primary_key: proved.key[16..].to_vec(), + row_bytes: proved.value.clone().ok_or_else(|| { + Error::CorruptedData( + "indexed-axis secondary proof omitted a returned row value".to_string(), + ) + })?, + row_value_hash: proved.proof, + }); } - Ok(AxisEntries::Avg(entries)) + Ok(ProvenAxisRows::Avg(entries)) } } } @@ -879,10 +1313,10 @@ pub(crate) fn decode_axis_entries_from_result_set( pub(crate) fn decode_axis_entries_from_count_offset_items( axis: IndexAxis, items: &[grovedb_merk::proofs::query::CountOffsetReturnedItem], -) -> Result { +) -> Result { match axis { IndexAxis::Count => { - let mut entries: Vec<(u64, Vec)> = Vec::with_capacity(items.len()); + let mut entries = Vec::with_capacity(items.len()); for it in items { if it.key.len() < 8 { return Err(Error::CorruptedData(format!( @@ -892,12 +1326,17 @@ pub(crate) fn decode_axis_entries_from_count_offset_items( } let mut count_bytes = [0u8; 8]; count_bytes.copy_from_slice(&it.key[..8]); - entries.push((decode_count_sort_key(&count_bytes), it.key[8..].to_vec())); + entries.push(ProvenAxisRow { + ordering_value: decode_count_sort_key(&count_bytes), + primary_key: it.key[8..].to_vec(), + row_bytes: it.value.clone(), + row_value_hash: it.value_hash, + }); } - Ok(AxisEntries::Count(entries)) + Ok(ProvenAxisRows::Count(entries)) } IndexAxis::Avg => { - let mut entries: Vec<(i128, Vec)> = Vec::with_capacity(items.len()); + let mut entries = Vec::with_capacity(items.len()); for it in items { if it.key.len() < 16 { return Err(Error::CorruptedData(format!( @@ -907,15 +1346,17 @@ pub(crate) fn decode_axis_entries_from_count_offset_items( } let mut avg_bytes = [0u8; 16]; avg_bytes.copy_from_slice(&it.key[..16]); - entries.push(( - grovedb_element::indexed::decode_avg_sort_key(&avg_bytes), - it.key[16..].to_vec(), - )); + entries.push(ProvenAxisRow { + ordering_value: grovedb_element::indexed::decode_avg_sort_key(&avg_bytes), + primary_key: it.key[16..].to_vec(), + row_bytes: it.value.clone(), + row_value_hash: it.value_hash, + }); } - Ok(AxisEntries::Avg(entries)) + Ok(ProvenAxisRows::Avg(entries)) } IndexAxis::Sum => { - let mut entries: Vec<(i64, Vec)> = Vec::with_capacity(items.len()); + let mut entries = Vec::with_capacity(items.len()); for it in items { if it.key.len() < 8 { return Err(Error::CorruptedData(format!( @@ -925,9 +1366,14 @@ pub(crate) fn decode_axis_entries_from_count_offset_items( } let mut sum_bytes = [0u8; 8]; sum_bytes.copy_from_slice(&it.key[..8]); - entries.push((decode_sum_sort_key(&sum_bytes), it.key[8..].to_vec())); + entries.push(ProvenAxisRow { + ordering_value: decode_sum_sort_key(&sum_bytes), + primary_key: it.key[8..].to_vec(), + row_bytes: it.value.clone(), + row_value_hash: it.value_hash, + }); } - Ok(AxisEntries::Sum(entries)) + Ok(ProvenAxisRows::Sum(entries)) } } } diff --git a/grovedb/src/operations/proof/mod.rs b/grovedb/src/operations/proof/mod.rs index 0f3aeec78..ea0268b6c 100644 --- a/grovedb/src/operations/proof/mod.rs +++ b/grovedb/src/operations/proof/mod.rs @@ -402,6 +402,9 @@ pub struct AxisDescentProof { /// Merk range proof for `Bounded`, an aggregate-on-range proof for /// `AggregateOverValueRange`. pub secondary_proof: Vec, + /// One immediate-primary/terminal-resolution witness per returned + /// secondary row, in proof order. Aggregate traversals carry none. + pub target_witnesses: Vec, } impl AxisDescentProof { diff --git a/grovedb/src/operations/proof/verify.rs b/grovedb/src/operations/proof/verify.rs index 7510b3cdf..756d50b1c 100644 --- a/grovedb/src/operations/proof/verify.rs +++ b/grovedb/src/operations/proof/verify.rs @@ -21,7 +21,10 @@ use crate::operations::proof::util::{ }; use crate::{ operations::proof::{ - indexed_axis::AxisEntries, + indexed_axis::{ + verify::{resolve_indexed_axis_rows, ProvenAxisRows}, + AxisEntries, IndexedTargetWitness, + }, util::{ProvedPathKeyOptionalValue, ProvedPathKeyValues}, AxisDescentProof, GroveDBProof, GroveDBProofV0, GroveDBProofV1, LayerProof, MerkOnlyLayerProof, ProofBytes, ProveOptions, @@ -56,6 +59,19 @@ pub(crate) enum AxisWalkResult { }, /// `RankOfKey`: the attested 0-based rank of the queried key. Rank { rank: u64 }, + /// Internal-only state held until the surrounding V1 walk finishes. + PendingEntries { + rows: ProvenAxisRows, + witnesses: Vec, + skipped: Option, + }, + /// Internal-only rank counterpart of [`Self::PendingEntries`]. + PendingRank { + rows: ProvenAxisRows, + witnesses: Vec, + rank: u64, + expected_key: Vec, + }, /// `AggregateOverValueRange`: the attested aggregate over the value range. Aggregate { value: i128 }, /// Sum-budget window: the matched `(key, value)` pairs, their net @@ -509,6 +525,55 @@ impl GroveDb { 0, grove_version, )?; + for outcome in &mut axis_outcomes { + let placeholder = AxisWalkResult::Aggregate { value: 0 }; + outcome.result = match std::mem::replace(&mut outcome.result, placeholder) { + AxisWalkResult::PendingEntries { + rows, + witnesses, + skipped, + } => { + let path_slices: Vec<&[u8]> = outcome.path.iter().map(Vec::as_slice).collect(); + let entries = resolve_indexed_axis_rows( + rows, + &witnesses, + &path_slices, + &root_hash, + grove_version, + )?; + AxisWalkResult::Entries { entries, skipped } + } + AxisWalkResult::PendingRank { + rows, + witnesses, + rank, + expected_key, + } => { + let path_slices: Vec<&[u8]> = outcome.path.iter().map(Vec::as_slice).collect(); + let entries = resolve_indexed_axis_rows( + rows, + &witnesses, + &path_slices, + &root_hash, + grove_version, + )?; + let yielded_key = entries.first_original_key().map(<[u8]>::to_vec); + if yielded_key.as_deref() != Some(expected_key.as_slice()) { + return Err(Error::InvalidProof( + query.clone(), + format!( + "axis descent: the entry at rank {rank} is {:?}, not the queried \ + key {}", + yielded_key.map(hex::encode), + hex::encode(expected_key), + ), + )); + } + AxisWalkResult::Rank { rank } + } + resolved => resolved, + }; + } Ok((root_hash, result, axis_outcomes)) } @@ -629,11 +694,10 @@ impl GroveDb { // normal traversal (own_count = 0) and not surfaced via // the merk's `returned_items`. If one appears here, the // proof was forged. - // • **Reference / ReferenceWithSumItem** — would need the - // regular flow's reference post-pass to dereference the - // target; we don't run that on the count-offset - // short-circuit, so a raw reference here would be returned - // verbatim. Reject. + // • **Raw Reference / ReferenceWithSumItem** — the prover runs + // the ordinary reference rewrite on this short-circuit, so an + // honest proof surfaces the dereferenced target here. A raw + // reference is therefore malformed. // • **Non-empty tree** — V1 strict-mode would require a // `KVValueHashFeatureTypeWithChildHash` proof node here; // accepting one without that would silently bypass the @@ -684,15 +748,14 @@ impl GroveDb { ))); } if inner.is_reference() { - return Err(Error::NotSupported(format!( - "count-offset paginated proofs do not yet support \ - Reference / ReferenceWithSumItem return values (key {}); the \ - regular flow's reference post-pass isn't applied on the \ - count-offset short-circuit, so an accepted reference here \ - would surface the raw Element::Reference rather than the \ - dereferenced target", - hex::encode(&item.key) - ))); + return Err(Error::InvalidProof( + query.clone(), + format!( + "count-offset paginated proof surfaced a raw Reference / \ + ReferenceWithSumItem at key {} after the prover's reference rewrite", + hex::encode(&item.key) + ), + )); } // Empty-tree value-hash equality check (defense-in-depth on // top of the merk-level KV→KVValueHash forgery guard). @@ -833,7 +896,7 @@ impl GroveDb { use crate::operations::proof::indexed_axis::verify::{ count_aggregate_inner_range, decode_axis_entries_from_count_offset_items, decode_axis_entries_from_result_set, recompute_axis_binding_digest, - sum_aggregate_inner_range, + sum_aggregate_inner_range, ProvenAxisRows, }; // Envelope gate: the axis descent is a V4 acceptance rule. @@ -885,6 +948,16 @@ impl GroveDb { .to_string(), )); } + if matches!( + axis_query.traversal, + AxisTraversal::AggregateOverValueRange { .. } + ) && !payload.target_witnesses.is_empty() + { + return Err(Error::InvalidProof( + query.clone(), + "an aggregate axis descent must not carry target witnesses".to_string(), + )); + } // 1. Verify the secondary proof for the query's traversal, // recomputing the secondary root hash. @@ -905,12 +978,12 @@ impl GroveDb { format!("axis descent: secondary count-offset proof failed: {e}"), ) })?; - let entries = - decode_axis_entries_from_count_offset_items(axis, &res.returned_items)?; + let rows = decode_axis_entries_from_count_offset_items(axis, &res.returned_items)?; ( res.root_hash, - AxisWalkResult::Entries { - entries, + AxisWalkResult::PendingEntries { + rows, + witnesses: payload.target_witnesses.clone(), skipped: Some(res.skipped), }, ) @@ -951,21 +1024,16 @@ impl GroveDb { ), )); } - let entries = - decode_axis_entries_from_count_offset_items(axis, &res.returned_items)?; - let yielded_key = entries.first_original_key().map(|k| k.to_vec()); - if yielded_key.as_deref() != Some(key.as_slice()) { - return Err(Error::InvalidProof( - query.clone(), - format!( - "axis descent: the entry at rank {rank} is {:?}, not the queried \ - key {}", - yielded_key.map(hex::encode), - hex::encode(key), - ), - )); - } - (res.root_hash, AxisWalkResult::Rank { rank }) + let rows = decode_axis_entries_from_count_offset_items(axis, &res.returned_items)?; + ( + res.root_hash, + AxisWalkResult::PendingRank { + rows, + witnesses: payload.target_witnesses.clone(), + rank, + expected_key: key.clone(), + }, + ) } AxisTraversal::Bounded { limit, .. } => { if payload.secondary_proof.is_empty() { @@ -979,8 +1047,9 @@ impl GroveDb { // preimage. ( NULL_HASH, - AxisWalkResult::Entries { - entries: AxisEntries::empty_for_axis(axis), + AxisWalkResult::PendingEntries { + rows: ProvenAxisRows::empty_for_axis(axis), + witnesses: payload.target_witnesses.clone(), skipped: None, }, ) @@ -1004,11 +1073,12 @@ impl GroveDb { format!("axis descent: secondary range proof failed: {e}"), ) })?; - let entries = decode_axis_entries_from_result_set(axis, &res.result_set)?; + let rows = decode_axis_entries_from_result_set(axis, &res.result_set)?; ( root, - AxisWalkResult::Entries { - entries, + AxisWalkResult::PendingEntries { + rows, + witnesses: payload.target_witnesses.clone(), skipped: None, }, ) diff --git a/grovedb/src/operations/replace_subtree_root.rs b/grovedb/src/operations/replace_subtree_root.rs index 3395f71b3..75a4fc487 100644 --- a/grovedb/src/operations/replace_subtree_root.rs +++ b/grovedb/src/operations/replace_subtree_root.rs @@ -99,9 +99,10 @@ impl GroveDb { cost_return_on_error!( &mut cost, - self.propagate_changes_with_transaction( + self.propagate_changes_with_transaction_refreshing_indexed_row( merk_cache, path, + key, tx.as_ref(), &batch, grove_version, diff --git a/grovedb/src/query_result_type.rs b/grovedb/src/query_result_type.rs index 6c4deca72..f5978da41 100644 --- a/grovedb/src/query_result_type.rs +++ b/grovedb/src/query_result_type.rs @@ -17,6 +17,17 @@ use crate::{ Element, Error, }; +/// One resolved entry returned by every non-aggregate indexed-axis read. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct IndexedAxisEntry { + /// Axis ordering value decoded from the secondary-key prefix. + pub ordering_value: T, + /// Primary key decoded from the secondary-key suffix. + pub primary_key: Vec, + /// Primary value after applying ordinary GroveDB reference resolution. + pub value: Element, +} + #[derive(Copy, Clone)] /// Query result type pub enum QueryResultType { diff --git a/grovedb/src/tests/axis_descent_proof_tests.rs b/grovedb/src/tests/axis_descent_proof_tests.rs index 06722e0ca..17cdc32af 100644 --- a/grovedb/src/tests/axis_descent_proof_tests.rs +++ b/grovedb/src/tests/axis_descent_proof_tests.rs @@ -137,7 +137,7 @@ mod tests { .expect("prove axis path query") } - fn entries_as_sum(entries: &AxisEntries) -> &[(i64, Vec)] { + fn entries_as_sum(entries: &AxisEntries) -> &[crate::IndexedAxisEntry] { match entries { AxisEntries::Sum(entries) => entries, other => panic!("expected sum entries, got {other:?}"), @@ -1841,7 +1841,7 @@ mod tests { #[test] fn verify_grovedb_holds_across_count_secondary_mutations() { - // The PCPS count secondary — payload = SumItem(count_value) — + // The PCPS count secondary — canonical reference row with sum = count_value — // must satisfy verify_grovedb's primary<->secondary walk through // inserts, count changes, and deletes; the expected-payload // check in verify_indexed_axis_content is what would flag a diff --git a/grovedb/src/tests/batch_indexed_fresh_create_tests.rs b/grovedb/src/tests/batch_indexed_fresh_create_tests.rs index 3709fe6ca..81a9021f3 100644 --- a/grovedb/src/tests/batch_indexed_fresh_create_tests.rs +++ b/grovedb/src/tests/batch_indexed_fresh_create_tests.rs @@ -97,7 +97,7 @@ mod tests { .expect("populate"); assert_same_root(&one, &split, gv); - assert_eq!( + assert_axis_entries_eq!( one.indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), @@ -219,7 +219,7 @@ mod tests { .expect("sequential"); } assert_same_root(&one, &split, gv); - assert_eq!( + assert_axis_entries_eq!( one.indexed_count_top_k([TEST_LEAF, b"t", b"cidx"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), @@ -462,5 +462,8 @@ mod tests { db.indexed_count_top_k(path, 5, true, None, gv) .unwrap() .expect("count top_k") + .into_iter() + .map(|entry| (entry.ordering_value, entry.primary_key)) + .collect() } } diff --git a/grovedb/src/tests/batch_indexed_multi_axis_tests.rs b/grovedb/src/tests/batch_indexed_multi_axis_tests.rs index 25a52699d..af2b40613 100644 --- a/grovedb/src/tests/batch_indexed_multi_axis_tests.rs +++ b/grovedb/src/tests/batch_indexed_multi_axis_tests.rs @@ -250,7 +250,7 @@ mod tests { .indexed_sum_top_k(path.as_ref(), 10, true, None, gv) .unwrap() .expect("sum top_k"); - assert_eq!( + assert_axis_entries_eq!( by_sum, vec![ (30, b"a".to_vec()), @@ -266,7 +266,7 @@ mod tests { .indexed_count_top_k(path.as_ref(), 10, false, None, gv) .unwrap() .expect("count top_k"); - assert_eq!( + assert_axis_entries_eq!( by_count, vec![(1, b"a".to_vec()), (1, b"b".to_vec()), (1, b"c".to_vec())], "the count axis must be populated too, not just the sum axis" @@ -277,7 +277,10 @@ mod tests { .indexed_avg_top_k(path.as_ref(), 10, true, None, gv) .unwrap() .expect("avg top_k"); - let avg_keys: Vec> = by_avg.iter().map(|(_, k)| k.clone()).collect(); + let avg_keys: Vec> = by_avg + .iter() + .map(|entry| entry.primary_key.clone()) + .collect(); assert_eq!( avg_keys, vec![b"a".to_vec(), b"c".to_vec(), b"b".to_vec()], @@ -308,7 +311,7 @@ mod tests { .expect("batch write into single-axis PCPSIT"); let path = [TEST_LEAF, b"idx"]; - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k(path.as_ref(), 10, true, None, gv) .unwrap() .expect("sum top_k"), @@ -369,14 +372,14 @@ mod tests { .unwrap() .expect("update sum only"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k(path.as_ref(), 10, true, None, gv) .unwrap() .expect("sum top_k"), vec![(99, b"a".to_vec())], "the sum axis must reflect the new sum, with no stale row left behind" ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k(path.as_ref(), 10, true, None, gv) .unwrap() .expect("count top_k"), @@ -388,7 +391,7 @@ mod tests { .unwrap() .expect("avg top_k"); assert_eq!(by_avg.len(), 1, "the avg axis must not have a stale row"); - assert_eq!(by_avg[0].1, b"a".to_vec()); + assert_eq!(by_avg[0].primary_key, b"a".to_vec()); assert_clean(&db, gv); } @@ -440,7 +443,7 @@ mod tests { .unwrap() .expect("sum top_k") .into_iter() - .map(|(_, k)| k) + .map(|entry| entry.primary_key) .collect::>(), ), ( @@ -449,7 +452,7 @@ mod tests { .unwrap() .expect("count top_k") .into_iter() - .map(|(_, k)| k) + .map(|entry| entry.primary_key) .collect::>(), ), ( @@ -458,7 +461,7 @@ mod tests { .unwrap() .expect("avg top_k") .into_iter() - .map(|(_, k)| k) + .map(|entry| entry.primary_key) .collect::>(), ), ] { @@ -510,7 +513,7 @@ mod tests { .unwrap() .expect("batch delete from PSIT"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 10, true, None, gv) .unwrap() .expect("sum top_k"), @@ -563,13 +566,13 @@ mod tests { .len(), 1 ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 10, true, None, gv) .unwrap() .expect("psit"), vec![(12, b"y".to_vec())] ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"idx"].as_ref(), 10, true, None, gv) .unwrap() .expect("pcpsit sum"), @@ -602,7 +605,7 @@ mod tests { .unwrap() .expect("batch with negative sums"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"idx"].as_ref(), 10, false, None, gv) .unwrap() .expect("ascending sum top_k"), @@ -668,14 +671,14 @@ mod tests { .unwrap() .expect("batch write into the nested PCPSIT"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"mid", b"idx"].as_ref(), 10, true, None, gv) .unwrap() .expect("nested sum top_k"), vec![(15, b"a".to_vec())], "the nested PCPSIT's sum axis must be mirrored through the intermediate level" ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"mid", b"idx"].as_ref(), 10, true, None, gv) .unwrap() .expect("nested count top_k"), @@ -807,7 +810,7 @@ mod tests { .unwrap() .expect("insert_if_not_exists into an indexed primary"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"idx"].as_ref(), 10, true, None, gv) .unwrap() .expect("sum top_k"), @@ -854,7 +857,7 @@ mod tests { .unwrap() .expect("sum top_k"); assert_eq!(by_sum.len(), n as usize, "every entry must be indexed"); - let sums: Vec = by_sum.iter().map(|(s, _)| *s).collect(); + let sums: Vec = by_sum.iter().map(|entry| entry.ordering_value).collect(); let mut sorted = sums.clone(); sorted.sort_unstable(); assert_eq!(sums, sorted, "the sum axis must come back ascending"); @@ -1229,10 +1232,19 @@ mod tests { .unwrap() .expect("avg top_k"); assert_eq!( - avg_before, avg_after, + avg_before[0].ordering_value, avg_after[0].ordering_value, "the avg sort key is unchanged by (1,5) -> (2,10)" ); - assert_eq!( + assert_eq!(avg_before[0].primary_key, avg_after[0].primary_key); + assert_ne!( + avg_before[0].value, avg_after[0].value, + "the canonical reference row must resolve the refreshed primary value" + ); + assert!(matches!( + avg_after[0].value, + Element::CountSumTree(_, 2, 10, _) + )); + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"idx"].as_ref(), 5, true, None, gv) .unwrap() .expect("sum top_k"), @@ -1402,6 +1414,80 @@ mod tests { } } + /// Every direct non-Merk append rewrites its parent element's commitment + /// without changing the PCIT ordering value. The canonical reference row + /// must therefore be refreshed in place for all four APIs. + #[test] + fn every_direct_non_merk_append_refreshes_its_indexed_row() { + let gv = GroveVersion::latest(); + + let pcit_with_child = |key: &[u8], child: Element| { + let db = make_test_grovedb(gv); + make_pcit(&db, b"cidx", gv); + db.insert_into_count_indexed_tree([TEST_LEAF, b"cidx"].as_ref(), key, child, None, gv) + .unwrap() + .expect("insert non-Merk indexed child"); + db + }; + + let db = pcit_with_child(b"mmr", Element::empty_mmr_tree()); + db.mmr_tree_append( + [TEST_LEAF, b"cidx"].as_ref(), + b"mmr", + b"leaf".to_vec(), + None, + gv, + ) + .unwrap() + .expect("MMR append"); + assert_clean(&db, gv); + + let db = pcit_with_child( + b"bulk", + Element::empty_bulk_append_tree(4).expect("bulk tree"), + ); + db.bulk_append( + [TEST_LEAF, b"cidx"].as_ref(), + b"bulk", + b"value".to_vec(), + None, + gv, + ) + .unwrap() + .expect("bulk append"); + assert_clean(&db, gv); + + let db = pcit_with_child(b"dense", Element::empty_dense_tree(4)); + db.dense_tree_insert( + [TEST_LEAF, b"cidx"].as_ref(), + b"dense", + b"value".to_vec(), + None, + gv, + ) + .unwrap() + .expect("dense insert"); + assert_clean(&db, gv); + + let db = pcit_with_child( + b"commitment", + Element::empty_commitment_tree(4).expect("commitment tree"), + ); + db.commitment_tree_insert_raw( + [TEST_LEAF, b"cidx"].as_ref(), + b"commitment", + [1u8; 32], + [2u8; 32], + [3u8; 32], + vec![0u8; 216], + None, + gv, + ) + .unwrap() + .expect("commitment insert"); + assert_clean(&db, gv); + } + /// A deep write *under* a child of an indexed primary keeps the secondary /// in sync on its own, through both the generic and the batch path. /// @@ -1427,7 +1513,7 @@ mod tests { ) .unwrap() .expect("child count tree"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), @@ -1445,7 +1531,7 @@ mod tests { ) .unwrap() .expect("a deep generic write under the child is allowed"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), @@ -1466,7 +1552,7 @@ mod tests { ) .unwrap() .expect("the same write through the batch path"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), diff --git a/grovedb/src/tests/batch_indexed_overwrite_tests.rs b/grovedb/src/tests/batch_indexed_overwrite_tests.rs index f0d355ad1..0ea691c6e 100644 --- a/grovedb/src/tests/batch_indexed_overwrite_tests.rs +++ b/grovedb/src/tests/batch_indexed_overwrite_tests.rs @@ -146,7 +146,7 @@ mod tests { root_before, "the refused batch must not have moved any state" ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), @@ -220,7 +220,7 @@ mod tests { let db = make_test_grovedb(gv); make_psit_with_entry(&db, b"psit", gv); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 5, true, None, gv) .unwrap() .expect("top_k"), diff --git a/grovedb/src/tests/count_offset_paginated_tests.rs b/grovedb/src/tests/count_offset_paginated_tests.rs index 964e0f66b..925cd7f18 100644 --- a/grovedb/src/tests/count_offset_paginated_tests.rs +++ b/grovedb/src/tests/count_offset_paginated_tests.rs @@ -841,13 +841,11 @@ mod tests { // `merk/src/proofs/query/count_offset/tests.rs`) covers the // verifier symmetric. - /// Prover-side rejection for `Reference` in-range entries. Earlier - /// drafts returned the raw `Element::Reference` bytes verbatim - /// because the count-offset short-circuit doesn't run the regular - /// flow's reference post-pass. The prover now refuses to emit - /// these. + /// The generic count-offset short-circuit runs the same ordinary reference + /// rewrite as regular GroveDB proofs, so verified results contain the + /// resolved target rather than raw reference bytes. #[test] - fn rejects_count_offset_with_reference_entry() { + fn count_offset_resolves_reference_result() { let v = GroveVersion::latest(); let db = make_test_grovedb(v); db.insert( @@ -903,13 +901,23 @@ mod tests { vec![b"counts".to_vec()], SizedQuery::new(q, Some(2), Some(1)), ); - let result = db.prove_query(&path_query, None, v).unwrap(); - let err = result.expect_err("prover must reject Reference in-range entry"); - let msg = format!("{}", err); - assert!( - msg.contains("Reference"), - "prover rejection should mention Reference; got {}", - msg + let proof = db + .prove_query(&path_query, None, v) + .unwrap() + .expect("prove count-offset page with reference"); + let (_, rows) = GroveDb::verify_query_raw(&proof, &path_query, v) + .expect("generic GroveDB verification should resolve the reference"); + assert_eq!(rows.len(), 2); + assert_eq!(rows[0].key, b"b".to_vec()); + assert_eq!( + Element::deserialize(&rows[0].value, v).expect("resolved element"), + Element::new_item(b"target_value".to_vec()), + "count-offset proof must surface the referenced target value" + ); + assert_eq!(rows[1].key, b"c".to_vec()); + assert_eq!( + Element::deserialize(&rows[1].value, v).expect("direct element"), + Element::new_item(b"v_c".to_vec()) ); } @@ -1004,9 +1012,9 @@ mod tests { // ──────── Forged-proof tests for verifier defense-in-depth ──────── // - // The merk-level prover now refuses to emit NonCounted-wrapped / - // Reference / non-empty-tree in-range entries (see the three - // `rejects_count_offset_with_*` tests above). That makes the + // The merk-level prover refuses to emit NonCounted-wrapped and non-empty + // tree in-range entries. References are rewritten to resolved-value proof + // nodes before encoding. That makes the remaining // GroveDB-layer defense-in-depth checks in // `run_count_offset_layer_dispatch` (verify.rs ~537-566) unreachable // by **honest** proofs. To keep those branches exercised — they're @@ -1246,11 +1254,9 @@ mod tests { ); } - /// Defense-in-depth: a forged proof that surfaces a Reference - /// element in `returned_items` must be rejected as `NotSupported` - /// mentioning "Reference" — the count-offset short-circuit doesn't - /// run the regular flow's reference post-pass, so accepting one - /// would surface a raw `Element::Reference` to the caller. + /// Defense-in-depth: a forged proof that surfaces a raw Reference in + /// `returned_items` must be rejected as malformed. Honest count-offset + /// proofs rewrite references to resolved-value nodes before encoding. #[test] fn verifier_rejects_forged_reference_returned_item() { use crate::reference_path::ReferencePathType; @@ -1265,8 +1271,8 @@ mod tests { let result = GroveDb::verify_query_raw(&tampered, &path_query, v); let err = result.expect_err("forged Reference return must be rejected"); assert!( - matches!(err, crate::Error::NotSupported(ref msg) if msg.contains("Reference")), - "forged Reference return should reject as NotSupported mentioning Reference; got {:?}", + matches!(err, crate::Error::InvalidProof(_, ref msg) if msg.contains("Reference")), + "forged Reference return should reject as InvalidProof mentioning Reference; got {:?}", err, ); } diff --git a/grovedb/src/tests/coverage_batch_indexed_tests.rs b/grovedb/src/tests/coverage_batch_indexed_tests.rs index 8caec6bbc..b86e238b2 100644 --- a/grovedb/src/tests/coverage_batch_indexed_tests.rs +++ b/grovedb/src/tests/coverage_batch_indexed_tests.rs @@ -618,7 +618,7 @@ mod tests { db.apply_batch(ops, None, None, grove_version) .unwrap() .expect("fresh PSIT create + populate in one batch is supported"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 5, true, None, grove_version) .unwrap() .expect("sum top_k"), @@ -648,7 +648,7 @@ mod tests { db.apply_batch(ops, None, None, grove_version) .unwrap() .expect("fresh PCPSIT create + populate in one batch is supported"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k( [TEST_LEAF, b"pcpsit"].as_ref(), 5, diff --git a/grovedb/src/tests/coverage_lib_paths_tests.rs b/grovedb/src/tests/coverage_lib_paths_tests.rs index 4ce3dc783..c13a82ae6 100644 --- a/grovedb/src/tests/coverage_lib_paths_tests.rs +++ b/grovedb/src/tests/coverage_lib_paths_tests.rs @@ -283,6 +283,7 @@ mod tests { merk_cache, sp, initial_deferred, + None, &tx, &batch, grove_version, @@ -539,6 +540,7 @@ mod tests { merk_cache, root_path, Some((ZERO_HASH, Some(b"stale".to_vec()))), + None, &tx, &batch, grove_version, diff --git a/grovedb/src/tests/coverage_misc_tests.rs b/grovedb/src/tests/coverage_misc_tests.rs index acc99ec15..f91e3cb75 100644 --- a/grovedb/src/tests/coverage_misc_tests.rs +++ b/grovedb/src/tests/coverage_misc_tests.rs @@ -412,20 +412,20 @@ mod tests { ) .storage_cost .added_bytes; - // +8 bytes of primary key => +8 bytes in each of the two rows the - // mirror writes. - assert_eq!(k16_count - k8_count, 16); + // +8 bytes of primary key appears in each old/new secondary key + // and in the canonical sibling-reference payload. + assert_eq!(k16_count - k8_count, 3 * 8); // A 8-byte primary + the avg axis and a 16-byte primary + the // count axis both produce a 24-byte secondary key, and since // issue #806 EVERY axis is a dual-aggregate // ProvableCountProvableSumTree (the count axis mirrors its // count_value into the sum half), the merk-node aggregate width - // is identical across axes. What is left is only the payload - // shape: avg stores an `ItemWithSumItem` (12 bytes at worst sum - // width) where count stores a `SumItem` (11) — the mirror - // estimator sizes each axis's row with its REAL payload shape. - assert_eq!(k8_avg - k16_count, 1); - assert_eq!(k8_avg - k8_count, 16 + 1); + // is identical across axes. Both rows now use the same canonical + // `ReferenceWithSumItem` shape. The avg row's shorter primary key + // saves eight payload bytes, while its wider sort prefix adds eight + // bytes to each of the old/new secondary keys. + assert_eq!(k16_count - k8_avg, 8); + assert_eq!(k8_avg - k8_count, 2 * 8); let count_axis = narrow; let sizes = EstimatedLayerSizes::AllItems(8, 100, None); @@ -443,15 +443,18 @@ mod tests { assert_eq!(mirror_cost(sizes, &[]), OperationCost::default()); } - /// The secondary key width saturates at `u8::MAX`, so primary keys - /// that would overflow price identically. + /// The secondary key width saturates at `u8::MAX`, but the canonical + /// reference payload still carries the complete primary key. #[test] - fn indexed_secondary_mirror_key_width_saturates_at_u8_max() { + fn indexed_secondary_mirror_reference_payload_grows_after_key_width_saturates() { let axes = [IndexAxis::Avg]; - assert_eq!( - mirror_cost(EstimatedLayerSizes::AllItems(250, 100, None), &axes), - mirror_cost(EstimatedLayerSizes::AllItems(255, 100, None), &axes), - "both 250+16 and 255+16 clamp to a 255-byte secondary key" + let key_250 = mirror_cost(EstimatedLayerSizes::AllItems(250, 100, None), &axes); + let key_255 = mirror_cost(EstimatedLayerSizes::AllItems(255, 100, None), &axes); + assert_eq!(key_250.seek_count, key_255.seek_count); + assert_eq!(key_250.hash_node_calls, key_255.hash_node_calls); + assert!( + key_255.storage_cost.added_bytes > key_250.storage_cost.added_bytes, + "the secondary key is clamped in both estimates, but the reference payload grows" ); } } diff --git a/grovedb/src/tests/coverage_round7_tests.rs b/grovedb/src/tests/coverage_round7_tests.rs index 3cf0a6353..dec9f6359 100644 --- a/grovedb/src/tests/coverage_round7_tests.rs +++ b/grovedb/src/tests/coverage_round7_tests.rs @@ -798,6 +798,7 @@ mod tests { other_axes_root_hashes: vec![], target_is_pcpsit: false, secondary_proof: vec![], + target_witnesses: vec![], requested_limit: Some(1), descending: true, }; @@ -823,6 +824,7 @@ mod tests { other_axes_root_hashes: vec![], target_is_pcpsit: false, secondary_proof: vec![], + target_witnesses: vec![], requested_k: 1, requested_offset: 0, descending: true, @@ -1696,7 +1698,7 @@ mod tests { assert_eq!(top.len(), 2); // The partial batch must have mirrored the DERIVED count of the // new child into the secondary, ahead of the existing "a" (3). - assert_eq!(top, vec![(9, b"b".to_vec()), (3, b"a".to_vec())]); + assert_axis_entries_eq!(top, vec![(9, b"b".to_vec()), (3, b"a".to_vec())]); assert_verify_passes(&db, grove_version); } @@ -1765,7 +1767,7 @@ mod tests { .unwrap() .expect("top-k"); assert_eq!(top.len(), 1); - assert_eq!(top[0].1, b"x".to_vec()); + assert_eq!(top[0].primary_key, b"x".to_vec()); assert_verify_passes(&db, grove_version); } @@ -2206,8 +2208,8 @@ mod tests { }; // Descending starting at offset 1: skip "e"(9), then take "d"(7), "c"(5). assert_eq!(entries.len(), 2); - assert_eq!(entries[0].0, 7); - assert_eq!(entries[1].0, 5); + assert_eq!(entries[0].ordering_value, 7); + assert_eq!(entries[1].ordering_value, 5); } /// PSIT arbitrary query round trip. @@ -2236,8 +2238,8 @@ mod tests { _ => panic!("expected sum"), }; assert_eq!(entries.len(), 2); - assert_eq!(entries[0].0, 10); - assert_eq!(entries[1].0, 5); + assert_eq!(entries[0].ordering_value, 10); + assert_eq!(entries[1].ordering_value, 5); } /// L4019-4028: ops_at_level_above (level above) exists but the diff --git a/grovedb/src/tests/indexed_axis_nested_and_bounds_tests.rs b/grovedb/src/tests/indexed_axis_nested_and_bounds_tests.rs index 090fb754f..5c0124033 100644 --- a/grovedb/src/tests/indexed_axis_nested_and_bounds_tests.rs +++ b/grovedb/src/tests/indexed_axis_nested_and_bounds_tests.rs @@ -39,7 +39,7 @@ mod tests { assert!(issues.is_empty(), "verify_grovedb issues: {issues:?}"); } - fn count_entries(entries: &AxisEntries) -> &Vec<(u64, Vec)> { + fn count_entries(entries: &AxisEntries) -> &Vec> { match entries { AxisEntries::Count(v) => v, other => panic!("expected count entries, got {other:?}"), @@ -144,7 +144,7 @@ mod tests { db.root_hash(None, gv).unwrap().expect("root hash"), "the reconstructed root must equal the live GroveDB root" ); - assert_eq!( + assert_axis_entries_eq!( count_entries(&result.entries), &vec![(1u64, b"y".to_vec()), (1u64, b"x".to_vec())], "descending count order, ties broken by descending key" @@ -239,9 +239,12 @@ mod tests { db.root_hash(None, gv).unwrap().expect("root hash"), "the reconstructed root must equal the live GroveDB root" ); - assert_eq!( - result.entries, - AxisEntries::Sum(vec![(9i64, b"y".to_vec()), (-4i64, b"x".to_vec())]), + let AxisEntries::Sum(entries) = &result.entries else { + panic!("expected sum entries"); + }; + assert_axis_entries_eq!( + entries.as_slice(), + [(9i64, b"y".to_vec()), (-4i64, b"x".to_vec())], "descending sum order, negatives sorting below positives" ); } @@ -339,7 +342,7 @@ mod tests { db.root_hash(None, gv).unwrap().expect("root hash"), "the reconstructed root must equal the live GroveDB root" ); - assert_eq!( + assert_axis_entries_eq!( count_entries(&result.entries), &vec![(1u64, b"y".to_vec()), (1u64, b"x".to_vec())], ); diff --git a/grovedb/src/tests/indexed_axis_offset_proof_tests.rs b/grovedb/src/tests/indexed_axis_offset_proof_tests.rs index e81988bdf..e5be83dc4 100644 --- a/grovedb/src/tests/indexed_axis_offset_proof_tests.rs +++ b/grovedb/src/tests/indexed_axis_offset_proof_tests.rs @@ -88,7 +88,7 @@ mod tests { } } - fn entries_as_sum(entries: &AxisEntries) -> &[(i64, Vec)] { + fn entries_as_sum(entries: &AxisEntries) -> &[crate::IndexedAxisEntry] { match entries { AxisEntries::Sum(v) => v.as_slice(), other => panic!("expected sum entries, got {:?}", other), @@ -193,7 +193,7 @@ mod tests { ) .expect("verify"); assert_eq!(result.skipped, 3); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[ (70i64, b"g".to_vec()), @@ -218,7 +218,7 @@ mod tests { ) .expect("verify"); assert_eq!(result.skipped, 4); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[(50i64, b"e".to_vec()), (60, b"f".to_vec())] ); @@ -252,7 +252,7 @@ mod tests { ) .expect("verify"); assert_eq!(result.skipped, 8); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[(20i64, b"b".to_vec()), (10, b"a".to_vec())], "the page is the walk's tail, shorter than k" @@ -373,7 +373,7 @@ mod tests { ) .expect("verify 4th biggest"); assert_eq!(result.skipped, 3); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[(70i64, b"g".to_vec())], "rank 4 descending of sums 10..100 is g(70)" @@ -407,7 +407,7 @@ mod tests { ) .expect("verify rank window"); assert_eq!(result.skipped, rank_zero_based as u64); - assert_eq!(entries_as_sum(&result.entries), &[(*sum, key.to_vec())]); + assert_axis_entries_eq!(entries_as_sum(&result.entries), &[(*sum, key.to_vec())]); } } @@ -458,7 +458,7 @@ mod tests { ) .expect("verify mid-tie ascending"); assert_eq!(result.skipped, 4); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[ (50i64, b"t_c".to_vec()), @@ -484,7 +484,7 @@ mod tests { ) .expect("verify mid-tie descending"); assert_eq!(result.skipped, 3); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[ (50i64, b"t_d".to_vec()), @@ -538,7 +538,7 @@ mod tests { ) .expect("verify"); assert_eq!(result.skipped, 1); - assert_eq!( + assert_axis_entries_eq!( entries_as_sum(&result.entries), &[ (5i64, b"b".to_vec()), diff --git a/grovedb/src/tests/indexed_axis_paginated_cost_tests.rs b/grovedb/src/tests/indexed_axis_paginated_cost_tests.rs index 793953362..30187f1d7 100644 --- a/grovedb/src/tests/indexed_axis_paginated_cost_tests.rs +++ b/grovedb/src/tests/indexed_axis_paginated_cost_tests.rs @@ -685,8 +685,9 @@ mod tests { ) .unwrap() .expect("legacy linear read"); - assert_eq!( - counted.entries, legacy, + assert_axis_entries_eq!( + counted.entries, + legacy, "counted and legacy diverge at offset={offset} k={k} \ descending={descending}" ); @@ -743,7 +744,7 @@ mod tests { in_tx .entries .iter() - .map(|(_, key)| key.clone()) + .map(|entry| entry.primary_key.clone()) .collect::>(), vec![ b"k0000025".to_vec(), @@ -838,8 +839,9 @@ mod tests { } = linear.expect("three runs happened"); let linear_rows = linear_rows.expect("legacy linear read"); - assert_eq!( - counted_page.entries, linear_rows, + assert_axis_entries_eq!( + counted_page.entries, + linear_rows, "counted and linear paths diverged at n={n} k={k} offset={offset}" ); assert_eq!( diff --git a/grovedb/src/tests/indexed_axis_proof_tests.rs b/grovedb/src/tests/indexed_axis_proof_tests.rs index cfbbed8c0..b7b533d15 100644 --- a/grovedb/src/tests/indexed_axis_proof_tests.rs +++ b/grovedb/src/tests/indexed_axis_proof_tests.rs @@ -14,6 +14,7 @@ #[cfg(test)] mod tests { use grovedb_element::indexed::IndexAxis; + use grovedb_element::reference_path::ReferencePathType; use grovedb_merk::proofs::query::AggregateFold; use grovedb_merk::proofs::{query::QueryItem as MerkQueryItem, Query as MerkQuery}; use grovedb_version::version::GroveVersion; @@ -111,21 +112,21 @@ mod tests { db.root_hash(None, grove_version).unwrap().expect("root") } - fn entries_as_count(entries: &AxisEntries) -> &[(u64, Vec)] { + fn entries_as_count(entries: &AxisEntries) -> &[crate::IndexedAxisEntry] { match entries { AxisEntries::Count(v) => v.as_slice(), other => panic!("expected count entries, got {:?}", other), } } - fn entries_as_sum(entries: &AxisEntries) -> &[(i64, Vec)] { + fn entries_as_sum(entries: &AxisEntries) -> &[crate::IndexedAxisEntry] { match entries { AxisEntries::Sum(v) => v.as_slice(), other => panic!("expected sum entries, got {:?}", other), } } - fn entries_as_avg(entries: &AxisEntries) -> &[(i128, Vec)] { + fn entries_as_avg(entries: &AxisEntries) -> &[crate::IndexedAxisEntry] { match entries { AxisEntries::Avg(v) => v.as_slice(), other => panic!("expected avg entries, got {:?}", other), @@ -136,6 +137,199 @@ mod tests { // PCIT × count axis (compat with PCIT proof family) // ================================================================= + #[test] + fn indexed_rows_resolve_primary_references_in_reads_and_proofs() { + use crate::operations::proof::indexed_axis::{ + IndexedAxisRangeProof, IndexedTargetCommitment, + }; + + let grove_version = GroveVersion::latest(); + let db = make_test_grovedb(grove_version); + db.insert( + [TEST_LEAF].as_ref(), + b"pcit", + Element::empty_provable_count_indexed_tree(), + None, + None, + grove_version, + ) + .unwrap() + .expect("create PCIT"); + db.insert_into_count_indexed_tree( + [TEST_LEAF, b"pcit"].as_ref(), + b"target", + Element::new_item(b"canonical-value".to_vec()), + None, + grove_version, + ) + .unwrap() + .expect("insert canonical target"); + db.insert_into_count_indexed_tree( + [TEST_LEAF, b"pcit"].as_ref(), + b"middle", + Element::new_reference(ReferencePathType::SiblingReference(b"target".to_vec())), + None, + grove_version, + ) + .unwrap() + .expect("insert middle primary reference"); + db.insert_into_count_indexed_tree( + [TEST_LEAF, b"pcit"].as_ref(), + b"alias", + Element::new_reference(ReferencePathType::SiblingReference(b"middle".to_vec())), + None, + grove_version, + ) + .unwrap() + .expect("insert primary reference"); + + let path: &[&[u8]] = &[TEST_LEAF, b"pcit"]; + let direct = db + .indexed_count_top_k(path, 3, false, None, grove_version) + .unwrap() + .expect("direct indexed read"); + assert_eq!(direct[0].primary_key, b"alias".to_vec()); + assert_eq!( + direct[0].value, + Element::new_item(b"canonical-value".to_vec()) + ); + + let proof = db + .prove_indexed_count_top_k(path, 3, false, None, grove_version) + .unwrap() + .expect("prove indexed reference result"); + let verified = GroveDb::verify_indexed_count_top_k(&proof, path, 3, false, grove_version) + .expect("verify indexed reference result"); + let entries = entries_as_count(&verified.entries); + assert_eq!(entries[0].primary_key, b"alias".to_vec()); + assert_eq!( + entries[0].value, + Element::new_item(b"canonical-value".to_vec()) + ); + assert_eq!(verified.root_hash, root_hash(&db, grove_version)); + + let paginated_proof = db + .prove_indexed_count_top_k_paginated(path, 3, 0, false, None, grove_version) + .unwrap() + .expect("prove paginated indexed reference result"); + let paginated = GroveDb::verify_indexed_count_top_k_paginated( + &paginated_proof, + path, + 3, + 0, + false, + grove_version, + ) + .expect("verify paginated indexed reference result"); + let paginated_entries = entries_as_count(&paginated.entries); + assert_eq!(paginated_entries[0].primary_key, b"alias".to_vec()); + assert_eq!( + paginated_entries[0].value, + Element::new_item(b"canonical-value".to_vec()) + ); + + let config = bincode::config::standard(); + let (mut tampered, _): (IndexedAxisRangeProof, _) = + bincode::decode_from_slice(&proof, config).expect("decode range envelope"); + assert!(matches!( + tampered.target_witnesses[0].nodes[0].commitment, + IndexedTargetCommitment::Reference + )); + assert_eq!(tampered.target_witnesses[0].nodes.len(), 3); + assert!(tampered.target_witnesses[0].nodes[0] + .authentication + .is_none()); + assert!(tampered.target_witnesses[0].nodes[1..] + .iter() + .all(|node| node.authentication.is_some())); + tampered.target_witnesses[0].nodes[0].commitment = IndexedTargetCommitment::Simple; + let tampered = bincode::encode_to_vec(&tampered, config).expect("encode tampered proof"); + assert!( + GroveDb::verify_indexed_count_top_k(&tampered, path, 3, false, grove_version).is_err(), + "changing a reference witness to a simple commitment must invalidate the proof" + ); + + let (mut tampered, _): (IndexedAxisRangeProof, _) = + bincode::decode_from_slice(&proof, config).expect("decode range envelope"); + let direct_witness = tampered + .target_witnesses + .iter_mut() + .find(|witness| witness.nodes.len() == 1) + .expect("direct target witness"); + let terminal = direct_witness + .nodes + .last_mut() + .expect("terminal target node"); + terminal.value = Element::new_item(b"forged-value".to_vec()) + .serialize(grove_version) + .expect("serialize forged value"); + let tampered = bincode::encode_to_vec(&tampered, config).expect("encode tampered proof"); + assert!( + GroveDb::verify_indexed_count_top_k(&tampered, path, 3, false, grove_version).is_err(), + "changing compact witness value bytes must break the secondary row's hash binding" + ); + + let (mut tampered, _): (IndexedAxisRangeProof, _) = + bincode::decode_from_slice(&proof, config).expect("decode range envelope"); + let extra_terminal = tampered.target_witnesses[0] + .nodes + .last() + .expect("terminal target node") + .clone(); + tampered.target_witnesses[0].nodes.push(extra_terminal); + let tampered = bincode::encode_to_vec(&tampered, config).expect("encode tampered proof"); + assert!( + GroveDb::verify_indexed_count_top_k(&tampered, path, 3, false, grove_version).is_err(), + "compact witnesses must reject nodes appended after the terminal value" + ); + } + + /// Pin the compact-witness property that motivated reference-backed rows: + /// adding a result must not add another root-to-primary inclusion proof. + #[test] + fn indexed_target_witness_size_stays_compact_as_k_grows() { + let grove_version = GroveVersion::latest(); + let db = make_test_grovedb(grove_version); + let keys: Vec> = (0..32) + .map(|i| format!("entry-{i:02}").into_bytes()) + .collect(); + let entries: Vec<(&[u8], u64)> = keys + .iter() + .enumerate() + .map(|(i, key)| (key.as_slice(), (i + 1) as u64)) + .collect(); + build_pcit(&db, grove_version, &entries); + let path: &[&[u8]] = &[TEST_LEAF, b"pcit"]; + + let proof_4 = db + .prove_indexed_count_top_k(path, 4, true, None, grove_version) + .unwrap() + .expect("prove k=4"); + let proof_16 = db + .prove_indexed_count_top_k(path, 16, true, None, grove_version) + .unwrap() + .expect("prove k=16"); + GroveDb::verify_indexed_count_top_k(&proof_16, path, 16, true, grove_version) + .expect("compact proof verifies"); + + let marginal_per_row = (proof_16.len() - proof_4.len()) / 12; + eprintln!( + "compact indexed proof sizes: k=4 {} bytes, k=16 {} bytes, ~{} bytes/extra row", + proof_4.len(), + proof_16.len(), + marginal_per_row + ); + assert!( + proof_16.len() < 4096, + "k=16 proof regressed to {} bytes; compact target witnesses must stay below 4 KiB", + proof_16.len() + ); + assert!( + marginal_per_row < 256, + "compact target witnesses grew by ~{marginal_per_row} bytes per row" + ); + } + #[test] fn pcit_indexed_axis_top_k_descending_round_trip() { let grove_version = GroveVersion::latest(); @@ -153,7 +347,7 @@ mod tests { let result = GroveDb::verify_indexed_count_top_k(&proof, path, 3, true, grove_version) .expect("verify"); let entries = entries_as_count(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (12u64, b"bob".to_vec()), @@ -181,7 +375,7 @@ mod tests { let result = GroveDb::verify_indexed_count_top_k(&proof, path, 3, false, grove_version) .expect("verify"); let entries = entries_as_count(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (1u64, b"carol".to_vec()), @@ -218,7 +412,7 @@ mod tests { // Descending paged after skipping 2: c(3 was top-3? no — desc top-2 = f(6), e(5); // after skip-2 of f,e → d(4), c(3). let entries = entries_as_count(&result.entries); - assert_eq!(entries, &[(4u64, b"d".to_vec()), (3u64, b"c".to_vec())]); + assert_axis_entries_eq!(entries, &[(4u64, b"d".to_vec()), (3u64, b"c".to_vec())]); assert_eq!(result.skipped, 2); assert_eq!(result.root_hash, root_hash(&db, grove_version)); } @@ -268,7 +462,7 @@ mod tests { .expect("verify"); let entries = entries_as_count(&result.entries); // Ascending all: 1,5,10. - assert_eq!( + assert_axis_entries_eq!( entries, &[ (1u64, b"a".to_vec()), @@ -299,7 +493,7 @@ mod tests { let result = GroveDb::verify_indexed_sum_top_k(&proof, path, 3, true, grove_version) .expect("verify"); let entries = entries_as_sum(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (10i64, b"c".to_vec()), @@ -327,7 +521,7 @@ mod tests { let result = GroveDb::verify_indexed_sum_top_k(&proof, path, 4, true, grove_version) .expect("verify"); let entries = entries_as_sum(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (50i64, b"d".to_vec()), @@ -367,7 +561,7 @@ mod tests { .expect("verify"); // Descending after skip-2: 6,5 skipped → d(4), c(3) returned. let entries = entries_as_sum(&result.entries); - assert_eq!(entries, &[(4i64, b"d".to_vec()), (3, b"c".to_vec())]); + assert_axis_entries_eq!(entries, &[(4i64, b"d".to_vec()), (3, b"c".to_vec())]); assert_eq!(result.skipped, 2); } @@ -452,8 +646,8 @@ mod tests { // of original_key for ties (b/c/a → c, b, a). let entries = entries_as_count(&result.entries); assert_eq!(entries.len(), 3); - for (c, _) in entries { - assert_eq!(*c, 1); + for entry in entries { + assert_eq!(entry.ordering_value, 1); } assert_eq!(result.root_hash, root_hash(&db, grove_version)); } @@ -480,7 +674,7 @@ mod tests { let result = GroveDb::verify_indexed_sum_top_k(&proof, path, 3, true, grove_version) .expect("verify"); let entries = entries_as_sum(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (20i64, b"c".to_vec()), @@ -515,7 +709,7 @@ mod tests { let result = GroveDb::verify_indexed_avg_top_k(&proof, path, 3, true, grove_version) .expect("verify"); let entries = entries_as_avg(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (20i128 * SCALE, b"c".to_vec()), @@ -602,8 +796,8 @@ mod tests { // Skip-1 then take-2 → c, b. let entries = entries_as_count(&result.entries); assert_eq!(entries.len(), 2); - for (c, _) in entries { - assert_eq!(*c, 1); + for entry in entries { + assert_eq!(entry.ordering_value, 1); } assert_eq!(result.skipped, 1); } @@ -631,7 +825,7 @@ mod tests { assert_eq!(entries.len(), 2); const SCALE: i128 = grovedb_element::indexed::AVG_FIXED_POINT_SCALE; // Descending top-3 by avg: d(30), c(20), b(10). After skip-1 → c, b. - assert_eq!( + assert_axis_entries_eq!( entries, &[(20i128 * SCALE, b"c".to_vec()), (10 * SCALE, b"b".to_vec())] ); @@ -884,7 +1078,7 @@ mod tests { let result = GroveDb::verify_indexed_count_top_k(&proof, path, 2, true, grove_version) .expect("verify"); let entries = entries_as_count(&result.entries); - assert_eq!(entries, &[(9u64, b"b".to_vec()), (4, b"a".to_vec())]); + assert_axis_entries_eq!(entries, &[(9u64, b"b".to_vec()), (4, b"a".to_vec())]); assert_eq!(result.root_hash, root_hash(&db, grove_version)); } @@ -1818,7 +2012,7 @@ mod tests { let result = GroveDb::verify_indexed_count_query(&proof, path, q, Some(3), grove_version) .expect("verify"); let entries = entries_as_count(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (10u64, b"c".to_vec()), @@ -1844,7 +2038,7 @@ mod tests { let result = GroveDb::verify_indexed_sum_query(&proof, path, q, Some(3), grove_version) .expect("verify"); let entries = entries_as_sum(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (-3i64, b"a".to_vec()), @@ -1955,7 +2149,7 @@ mod tests { let result = GroveDb::verify_indexed_sum_top_k(&proof, path, 3, true, grove_version) .expect("verify"); let entries = entries_as_sum(&result.entries); - assert_eq!( + assert_axis_entries_eq!( entries, &[ (15i64, b"c".to_vec()), @@ -1991,15 +2185,34 @@ mod tests { #[test] fn axis_entries_helpers() { - let c = AxisEntries::Count(vec![(1u64, b"a".to_vec())]); + let c = AxisEntries::Count(vec![crate::IndexedAxisEntry { + ordering_value: 1u64, + primary_key: b"a".to_vec(), + value: Element::new_item(b"a".to_vec()), + }]); assert_eq!(c.len(), 1); assert!(!c.is_empty()); let empty_c = AxisEntries::Count(vec![]); assert_eq!(empty_c.len(), 0); assert!(empty_c.is_empty()); - let s = AxisEntries::Sum(vec![(1i64, b"a".to_vec()), (2i64, b"b".to_vec())]); + let s = AxisEntries::Sum(vec![ + crate::IndexedAxisEntry { + ordering_value: 1i64, + primary_key: b"a".to_vec(), + value: Element::new_item(b"a".to_vec()), + }, + crate::IndexedAxisEntry { + ordering_value: 2i64, + primary_key: b"b".to_vec(), + value: Element::new_item(b"b".to_vec()), + }, + ]); assert_eq!(s.len(), 2); - let a = AxisEntries::Avg(vec![(1i128, b"a".to_vec())]); + let a = AxisEntries::Avg(vec![crate::IndexedAxisEntry { + ordering_value: 1i128, + primary_key: b"a".to_vec(), + value: Element::new_item(b"a".to_vec()), + }]); assert_eq!(a.len(), 1); } @@ -2487,7 +2700,7 @@ mod tests { .expect("prove"); let result = GroveDb::verify_indexed_count_top_k(&proof, path, 1, true, grove_version) .expect("verify"); - assert_eq!(entries_as_count(&result.entries), &[(99u64, b"b".to_vec())]); + assert_axis_entries_eq!(entries_as_count(&result.entries), &[(99u64, b"b".to_vec())]); } // ---------- top_k proof-bytes / path rejections ---------- @@ -2671,7 +2884,7 @@ mod tests { let result = GroveDb::verify_indexed_count_top_k_paginated(&proof, path, 2, 1, false, grove_version) .expect("verify"); - assert_eq!( + assert_axis_entries_eq!( entries_as_count(&result.entries), &[(2u64, b"b".to_vec()), (3u64, b"c".to_vec())] ); @@ -2760,9 +2973,9 @@ mod tests { .expect("verify"); let got = entries_as_count(&result.entries); assert_eq!(got.len(), 3); - assert_eq!(got[0].0, 3); - assert_eq!(got[1].0, 2); - assert_eq!(got[2].0, 1); + assert_eq!(got[0].ordering_value, 3); + assert_eq!(got[1].ordering_value, 2); + assert_eq!(got[2].ordering_value, 1); assert_eq!(result.skipped, 6); } @@ -2804,7 +3017,7 @@ mod tests { GroveDb::verify_indexed_count_top_k_paginated(&proof, path, 2, 1, true, grove_version) .expect("verify"); // Descending: d(4), c(3), b(2), a(1). Skip 1 (d), take 2: c, b. - assert_eq!( + assert_axis_entries_eq!( entries_as_count(&result.entries), &[(3u64, b"c".to_vec()), (2u64, b"b".to_vec())] ); @@ -3110,7 +3323,14 @@ mod tests { .expect("verify"); let got = entries_as_count(&result.entries); assert_eq!(got.len(), 1); - assert_eq!(got[0], (5u64, b"alice".to_vec())); + assert_eq!(got[0].ordering_value, 5u64); + assert_eq!(got[0].primary_key, b"alice".to_vec()); + assert_eq!( + got[0].value, + db.get_raw(path.into(), b"alice", None, grove_version) + .unwrap() + .expect("primary value") + ); } #[test] @@ -3155,7 +3375,7 @@ mod tests { .expect("verify"); let got = entries_as_count(&result.entries); assert_eq!(got.len(), 1); - assert_eq!(got[0].0, 2); + assert_eq!(got[0].ordering_value, 2); } // ---------- post-mutation / cross-check / scale ---------- @@ -3178,8 +3398,8 @@ mod tests { let got = entries_as_count(&result.entries); assert_eq!(got.len(), 2); // c(3) and a(1) remain. - assert_eq!(got[0].1, b"c".to_vec()); - assert_eq!(got[1].1, b"a".to_vec()); + assert_eq!(got[0].primary_key, b"c".to_vec()); + assert_eq!(got[1].primary_key, b"a".to_vec()); } #[test] @@ -3236,8 +3456,8 @@ mod tests { .expect("verify"); let got = entries_as_count(&result.entries); assert_eq!(got.len(), 10); - assert_eq!(got[0].0, 29); - assert_eq!(got[9].0, 20); + assert_eq!(got[0].ordering_value, 29); + assert_eq!(got[9].ordering_value, 20); } // ---------- triple-nested cidx ---------- @@ -3290,7 +3510,7 @@ mod tests { .expect("verify"); let got = entries_as_count(&result.entries); assert_eq!(got.len(), 4); - assert_eq!(got[0].0, 7); + assert_eq!(got[0].ordering_value, 7); assert_eq!(result.root_hash, root_hash(&db, grove_version)); } diff --git a/grovedb/src/tests/indexed_reference_row_tests.rs b/grovedb/src/tests/indexed_reference_row_tests.rs new file mode 100644 index 000000000..bf6e49d5b --- /dev/null +++ b/grovedb/src/tests/indexed_reference_row_tests.rs @@ -0,0 +1,305 @@ +//! Focused corruption coverage for canonical indexed-secondary rows. +//! +//! Each test changes one part of the stored row and asserts that +//! `verify_grovedb` reports the corresponding sentinel. Keeping these cases +//! separate makes the operator-facing diagnostics part of the regression +//! surface, rather than only checking that some unspecified error occurred. + +#[cfg(test)] +mod tests { + use grovedb_element::{indexed::IndexAxis, reference_path::ReferencePathType}; + use grovedb_merk::element::{ + get::ElementFetchFromStorageExtensions, insert::ElementInsertToStorageExtensions, + }; + use grovedb_path::SubtreePath; + use grovedb_storage::{Storage, StorageBatch}; + use grovedb_version::version::GroveVersion; + + use crate::{ + operations::indexed_tree::make_axis_secondary_key, + tests::{make_test_grovedb, TEST_LEAF}, + Element, GroveDb, + }; + + fn pcit_with_one_entry(grove_version: &GroveVersion) -> crate::tests::TempGroveDb { + let db = make_test_grovedb(grove_version); + db.insert( + [TEST_LEAF].as_ref(), + b"cidx", + Element::empty_provable_count_indexed_tree(), + None, + None, + grove_version, + ) + .unwrap() + .expect("create PCIT"); + db.insert_into_count_indexed_tree( + [TEST_LEAF, b"cidx"].as_ref(), + b"a", + Element::new_item(b"v".to_vec()), + None, + grove_version, + ) + .unwrap() + .expect("insert entry"); + db + } + + /// Replace the canonical row while choosing the target value hash it is + /// combined with. The honest primary hash isolates row-shape corruption; + /// an explicit hash isolates stale commitment detection. + fn overwrite_row( + db: &GroveDb, + secondary_key: &[u8], + row: Element, + target_hash: Option<[u8; 32]>, + grove_version: &GroveVersion, + ) { + let tx = db.start_transaction(); + let batch = StorageBatch::new(); + let path_segments: [&[u8]; 2] = [TEST_LEAF, b"cidx".as_ref()]; + let path: SubtreePath<&[u8]> = (&path_segments).into(); + + let secondary_root_key = { + let parent_merk = db + .open_transactional_merk_at_path( + [TEST_LEAF].as_ref().into(), + &tx, + Some(&batch), + grove_version, + ) + .unwrap() + .expect("open parent"); + let cidx = Element::get(&parent_merk, b"cidx", true, grove_version) + .unwrap() + .expect("cidx element"); + match cidx.underlying() { + Element::ProvableCountIndexedTree(_, secondary_root, ..) => secondary_root.clone(), + other => panic!("not a PCIT element: {other:?}"), + } + }; + + { + let mut secondary = db + .open_indexed_secondary_at_path( + path, + IndexAxis::Count, + secondary_root_key, + &tx, + Some(&batch), + grove_version, + ) + .unwrap() + .expect("open secondary"); + let bind_to = target_hash.unwrap_or_else(|| { + let primary = db + .open_transactional_merk_at_path( + [TEST_LEAF, b"cidx".as_ref()].as_ref().into(), + &tx, + Some(&batch), + grove_version, + ) + .unwrap() + .expect("open primary"); + primary + .get_value_hash( + b"a", + true, + None::<&fn(&[u8], &GroveVersion) -> _>, + grove_version, + ) + .unwrap() + .expect("read value hash") + .expect("entry present") + }); + row.insert_reference(&mut secondary, secondary_key, bind_to, None, grove_version) + .unwrap() + .expect("write row"); + } + + db.db + .commit_multi_context_batch(batch, Some(&tx)) + .unwrap() + .expect("commit batch"); + tx.commit().expect("commit transaction"); + } + + fn sentinel_path(kind: &str) -> Vec> { + vec![ + TEST_LEAF.to_vec(), + b"cidx".to_vec(), + format!("__cidx_{kind}__").into_bytes(), + b"a".to_vec(), + ] + } + + fn assert_only_issue(db: &GroveDb, kind: &str, grove_version: &GroveVersion) { + let issues = db.verify_grovedb(None, false, true, grove_version).unwrap(); + let want = sentinel_path(kind); + // Rewriting the secondary directly deliberately leaves the parent + // indexed element's recorded secondary root stale, so the general + // verifier also reports the structural mismatch at `[.../cidx]`. + // Among indexed-row diagnostics, however, this fixture must produce + // exactly the sentinel named by the test. + let row_issue_paths = issues + .keys() + .filter(|path| { + path.get(2) + .is_some_and(|segment| segment.starts_with(b"__cidx_")) + }) + .collect::>(); + assert_eq!( + row_issue_paths.len(), + 1, + "expected exactly `{}`, got {:?}", + String::from_utf8_lossy(&want[2]), + row_issue_paths + .iter() + .map(|path| path + .iter() + .map(|segment| String::from_utf8_lossy(segment).to_string()) + .collect::>()) + .collect::>() + ); + assert!( + issues.contains_key(&want), + "expected `{}`, got {:?}", + String::from_utf8_lossy(&want[2]), + issues.keys().collect::>() + ); + } + + #[test] + fn healthy_tree_stores_a_canonical_reference_row() { + let grove_version = GroveVersion::latest(); + let db = pcit_with_one_entry(grove_version); + let issues = db.verify_grovedb(None, false, true, grove_version).unwrap(); + assert!(issues.is_empty(), "healthy tree reported: {issues:?}"); + + let tx = db.start_transaction(); + let batch = StorageBatch::new(); + let path_segments: [&[u8]; 2] = [TEST_LEAF, b"cidx".as_ref()]; + let secondary_root_key = { + let parent = db + .open_transactional_merk_at_path( + [TEST_LEAF].as_ref().into(), + &tx, + Some(&batch), + grove_version, + ) + .unwrap() + .unwrap(); + match Element::get(&parent, b"cidx", true, grove_version) + .unwrap() + .unwrap() + .underlying() + { + Element::ProvableCountIndexedTree(_, secondary_root, ..) => secondary_root.clone(), + other => panic!("not a PCIT: {other:?}"), + } + }; + let secondary = db + .open_indexed_secondary_at_path( + (&path_segments).into(), + IndexAxis::Count, + secondary_root_key, + &tx, + Some(&batch), + grove_version, + ) + .unwrap() + .unwrap(); + let key = make_axis_secondary_key(IndexAxis::Count, 1, 0, b"a"); + let row = Element::get(&secondary, key.as_slice(), true, grove_version) + .unwrap() + .expect("row present"); + assert_eq!( + row, + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"a".to_vec()), + Some(1), + 1, + ) + ); + } + + #[test] + fn legacy_placeholder_row_has_a_specific_sentinel() { + let grove_version = GroveVersion::latest(); + let db = pcit_with_one_entry(grove_version); + let key = make_axis_secondary_key(IndexAxis::Count, 1, 0, b"a"); + overwrite_row(&db, &key, Element::new_sum_item(1), None, grove_version); + assert_only_issue(&db, "secondary_legacy_or_non_reference_row", grove_version); + } + + #[test] + fn wrong_reference_path_has_a_specific_sentinel() { + let grove_version = GroveVersion::latest(); + let db = pcit_with_one_entry(grove_version); + let key = make_axis_secondary_key(IndexAxis::Count, 1, 0, b"a"); + overwrite_row( + &db, + &key, + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"somewhere-else".to_vec()), + Some(1), + 1, + ), + None, + grove_version, + ); + assert_only_issue(&db, "secondary_reference_path_mismatch", grove_version); + } + + #[test] + fn wrong_reference_hop_budget_has_a_specific_sentinel() { + let grove_version = GroveVersion::latest(); + let db = pcit_with_one_entry(grove_version); + let key = make_axis_secondary_key(IndexAxis::Count, 1, 0, b"a"); + overwrite_row( + &db, + &key, + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"a".to_vec()), + Some(2), + 1, + ), + None, + grove_version, + ); + assert_only_issue(&db, "secondary_reference_hop_mismatch", grove_version); + } + + #[test] + fn wrong_reference_sum_has_a_specific_sentinel() { + let grove_version = GroveVersion::latest(); + let db = pcit_with_one_entry(grove_version); + let key = make_axis_secondary_key(IndexAxis::Count, 1, 0, b"a"); + overwrite_row( + &db, + &key, + Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"a".to_vec()), + Some(1), + 99, + ), + None, + grove_version, + ); + assert_only_issue(&db, "secondary_reference_sum_mismatch", grove_version); + } + + #[test] + fn stale_target_hash_has_a_specific_sentinel() { + let grove_version = GroveVersion::latest(); + let db = pcit_with_one_entry(grove_version); + let key = make_axis_secondary_key(IndexAxis::Count, 1, 0, b"a"); + let canonical = Element::new_reference_with_sum_item_with_hops( + ReferencePathType::SiblingReference(b"a".to_vec()), + Some(1), + 1, + ); + overwrite_row(&db, &key, canonical, Some([0xEE; 32]), grove_version); + assert_only_issue(&db, "secondary_stale_target_hash", grove_version); + } +} diff --git a/grovedb/src/tests/indexed_tree_secondary_drift_tests.rs b/grovedb/src/tests/indexed_tree_secondary_drift_tests.rs index b0e360543..cc19ded48 100644 --- a/grovedb/src/tests/indexed_tree_secondary_drift_tests.rs +++ b/grovedb/src/tests/indexed_tree_secondary_drift_tests.rs @@ -288,7 +288,7 @@ mod tests { .indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, false, None, gv) .unwrap() .expect("top_k"); - assert_eq!( + assert_axis_entries_eq!( pristine_listing, vec![(1u64, b"a".to_vec()), (1u64, b"b".to_vec())], "baseline listing" @@ -371,7 +371,7 @@ mod tests { issue_keys(&db, gv).contains(&primary_orphan_issue), "deleting the row for 'a' must be reported as a primary orphan first" ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, false, None, gv) .unwrap() .expect("top_k"), @@ -385,7 +385,7 @@ mod tests { .expect("reconcile reinserts the missing row"); assert_clean(&db, gv); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, false, None, gv) .unwrap() .expect("top_k"), @@ -427,7 +427,7 @@ mod tests { issue_keys(&db, gv).contains(&mismatch_issue), "moving 'a' to count 7 must be reported as a count mismatch first" ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, true, None, gv) .unwrap() .expect("top_k"), @@ -441,7 +441,7 @@ mod tests { .expect("reconcile moves the row back"); assert_clean(&db, gv); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, false, None, gv) .unwrap() .expect("top_k"), @@ -571,7 +571,7 @@ mod tests { .unwrap() .expect("populate small"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, true, None, gv) .unwrap() .expect("top_k"), @@ -587,7 +587,7 @@ mod tests { &[secondary_key(3, b"big")], gv, ); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, true, None, gv) .unwrap() .expect("top_k"), @@ -599,7 +599,7 @@ mod tests { .unwrap() .expect("reconcile rebuilds from the primary"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, true, None, gv) .unwrap() .expect("top_k"), @@ -770,7 +770,7 @@ mod tests { ) .unwrap() .expect("reinsert"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 10, false, None, gv) .unwrap() .expect("top_k"), @@ -846,7 +846,7 @@ mod tests { // Detecting malformed rows in the skipped region is // `verify_grovedb`'s job (asserted above) and the collect loops' // (asserted below for every shape that reads the row itself). - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k_paginated([TEST_LEAF, b"pcit"].as_ref(), 1, 1, true, None, gv) .unwrap() .expect("counted skip passes the malformed row without decoding it") @@ -883,7 +883,7 @@ mod tests { // Ascending top-k stops before reaching the malformed row, so the // well-formed prefix of the index still reads cleanly — the error above // is the decoder refusing a specific row, not the query failing wholesale. - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"pcit"].as_ref(), 2, false, None, gv) .unwrap() .expect("the first two rows are well formed"), @@ -1324,7 +1324,7 @@ mod tests { // The iterator path reads the physical keyspace and is untouched // by the dangling root key. - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k_paginated([TEST_LEAF, b"pcit"].as_ref(), 2, 0, false, None, gv) .unwrap() .expect("offset-0 path reads physical rows") diff --git a/grovedb/src/tests/indexed_tree_security_regression_tests.rs b/grovedb/src/tests/indexed_tree_security_regression_tests.rs index 97ac00276..a81895d17 100644 --- a/grovedb/src/tests/indexed_tree_security_regression_tests.rs +++ b/grovedb/src/tests/indexed_tree_security_regression_tests.rs @@ -105,7 +105,7 @@ fn psit_rejects_big_sum_tree_at_i64_boundary() { ) .unwrap() .expect("populate the control child so its sum derives to 42"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_sum_top_k([b"psit".as_slice()].as_ref(), 10, true, None, grove_version,) .unwrap() .unwrap(), @@ -151,7 +151,7 @@ fn delete_tree_rejects_declared_type_mismatch() { ) .unwrap(); assert!(matches!(result, Err(Error::InvalidBatchOperation(_)))); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 10, true, None, grove_version,) .unwrap() .unwrap(), @@ -409,7 +409,7 @@ fn batch_count_changes_remove_all_old_secondary_rows_first() { ) .unwrap() .expect("update both children"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k( [TEST_LEAF, b"cidx"].as_ref(), 10, @@ -488,7 +488,7 @@ fn derived_counts_order_the_secondary_index() { ) .unwrap() .expect("top_k"); - let order: Vec> = top.iter().map(|(_, k)| k.clone()).collect(); + let order: Vec> = top.iter().map(|entry| entry.primary_key.clone()).collect(); assert_eq!( order, vec![b"b".to_vec(), b"c".to_vec(), b"a".to_vec()], @@ -574,7 +574,7 @@ fn batch_rejects_rootless_aggregate_child_under_indexed_primary() { ) .unwrap() .expect("top_k"); - assert_eq!( + assert_axis_entries_eq!( top, vec![(9, b"b".to_vec())], "the derived count must reach the secondary index" diff --git a/grovedb/src/tests/mod.rs b/grovedb/src/tests/mod.rs index bafe8cfa7..8e4f69a77 100644 --- a/grovedb/src/tests/mod.rs +++ b/grovedb/src/tests/mod.rs @@ -1,5 +1,39 @@ //! Tests +/// Compare the ordering/key projection in tests whose purpose is index order, +/// while still asserting that every returned value has been resolved past any +/// ordinary reference. Tests of value identity use `IndexedAxisEntry` directly. +macro_rules! assert_axis_entries_eq { + ($actual:expr, $expected:expr $(,)?) => {{ + let actual_entries = &$actual; + assert!( + actual_entries + .iter() + .all(|entry| !entry.value.underlying().is_reference()), + "indexed-axis results must contain terminal values, not references" + ); + let actual_pairs: Vec<_> = actual_entries + .iter() + .map(|entry| (entry.ordering_value, entry.primary_key.clone())) + .collect(); + assert_eq!(actual_pairs.as_slice(), $expected); + }}; + ($actual:expr, $expected:expr, $($message:tt)+) => {{ + let actual_entries = &$actual; + assert!( + actual_entries + .iter() + .all(|entry| !entry.value.underlying().is_reference()), + "indexed-axis results must contain terminal values, not references" + ); + let actual_pairs: Vec<_> = actual_entries + .iter() + .map(|entry| (entry.ordering_value, entry.primary_key.clone())) + .collect(); + assert_eq!(actual_pairs.as_slice(), $expected, $($message)+); + }}; +} + pub mod common; mod query_tests; @@ -60,6 +94,7 @@ mod indexed_axis_nested_and_bounds_tests; mod indexed_axis_offset_proof_tests; mod indexed_axis_paginated_cost_tests; mod indexed_axis_proof_tests; +mod indexed_reference_row_tests; mod indexed_tree_secondary_drift_tests; mod indexed_tree_security_regression_tests; mod is_empty_tree_tests; diff --git a/grovedb/src/tests/provable_count_indexed_tree_tests.rs b/grovedb/src/tests/provable_count_indexed_tree_tests.rs index ff97d1c81..26fa1da24 100644 --- a/grovedb/src/tests/provable_count_indexed_tree_tests.rs +++ b/grovedb/src/tests/provable_count_indexed_tree_tests.rs @@ -289,6 +289,12 @@ mod tests { Element::ProvableCountIndexedTree(_, _, c, _) => assert_eq!(c, 1), other => panic!("expected count=1 PCIT, got {:?}", other), } + let indexed = db + .indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 1, false, None, grove_version) + .unwrap() + .expect("indexed read after same-count overwrite"); + assert_eq!(indexed[0].primary_key, b"row".to_vec()); + assert_eq!(indexed[0].value, Element::new_item(b"second".to_vec())); assert_verify_passes(&db, grove_version); } @@ -821,7 +827,7 @@ mod tests { ) .unwrap(); result.expect("fresh PCIT create + populate in one batch is supported"); - assert_eq!( + assert_axis_entries_eq!( db.indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 5, true, None, grove_version) .unwrap() .expect("count top_k"), @@ -974,7 +980,7 @@ mod tests { .indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 3, true, None, grove_version) .unwrap() .expect("top-k descending"); - assert_eq!( + assert_axis_entries_eq!( top3, vec![ (20u64, b"eve".to_vec()), @@ -988,7 +994,7 @@ mod tests { .indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 2, false, None, grove_version) .unwrap() .expect("top-k ascending"); - assert_eq!( + assert_axis_entries_eq!( bottom2, vec![(1u64, b"carol".to_vec()), (5u64, b"alice".to_vec())] ); @@ -1039,7 +1045,7 @@ mod tests { ) .unwrap() .expect("page 1"); - assert_eq!( + assert_axis_entries_eq!( page1.entries, vec![(20u64, b"eve".to_vec()), (12u64, b"bob".to_vec())] ); @@ -1056,7 +1062,7 @@ mod tests { ) .unwrap() .expect("page 2"); - assert_eq!( + assert_axis_entries_eq!( page2.entries, vec![(7u64, b"dave".to_vec()), (5u64, b"alice".to_vec())] ); @@ -1073,7 +1079,7 @@ mod tests { ) .unwrap() .expect("page 3"); - assert_eq!(page3.entries, vec![(1u64, b"carol".to_vec())]); + assert_axis_entries_eq!(page3.entries, vec![(1u64, b"carol".to_vec())]); // Offset beyond total → empty. let beyond = db @@ -1128,7 +1134,7 @@ mod tests { ) .unwrap() .expect("range"); - assert_eq!( + assert_axis_entries_eq!( in_range, vec![ (5u64, b"alice".to_vec()), @@ -1150,7 +1156,7 @@ mod tests { ) .unwrap() .expect("range desc"); - assert_eq!( + assert_axis_entries_eq!( in_range_desc, vec![ (12u64, b"bob".to_vec()), @@ -1172,7 +1178,7 @@ mod tests { ) .unwrap() .expect("exact"); - assert_eq!(exact, vec![(12u64, b"bob".to_vec())]); + assert_axis_entries_eq!(exact, vec![(12u64, b"bob".to_vec())]); // lo > hi: empty. let empty = db diff --git a/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs b/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs index 864134821..2d177ce65 100644 --- a/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs +++ b/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs @@ -1069,7 +1069,7 @@ mod tests { ) .unwrap() .expect("top-k count"); - assert_eq!( + assert_axis_entries_eq!( top3, vec![ (5u64, b"carol".to_vec()), @@ -1099,7 +1099,7 @@ mod tests { ) .unwrap() .expect("range"); - assert_eq!( + assert_axis_entries_eq!( in_range, vec![ (2u64, b"alice".to_vec()), @@ -1156,7 +1156,7 @@ mod tests { ) .unwrap() .expect("top-k sum"); - assert_eq!( + assert_axis_entries_eq!( top3, vec![ (100i64, b"bob".to_vec()), @@ -1176,7 +1176,7 @@ mod tests { ) .unwrap() .expect("asc"); - assert_eq!( + assert_axis_entries_eq!( asc3, vec![ (-25i64, b"carol".to_vec()), @@ -1206,7 +1206,7 @@ mod tests { ) .unwrap() .expect("sum range"); - assert_eq!( + assert_axis_entries_eq!( in_range, vec![ (0i64, b"dave".to_vec()), @@ -1268,7 +1268,7 @@ mod tests { ) .unwrap() .expect("top-k avg"); - assert_eq!( + assert_axis_entries_eq!( top3, vec![ (25 * AVG_SCALE, b"bob".to_vec()), @@ -1288,7 +1288,7 @@ mod tests { ) .unwrap() .expect("asc avg"); - assert_eq!( + assert_axis_entries_eq!( asc3, vec![ (-5 * AVG_SCALE, b"carol".to_vec()), @@ -1318,7 +1318,7 @@ mod tests { ) .unwrap() .expect("avg range"); - assert_eq!( + assert_axis_entries_eq!( in_range, vec![ (0i128, b"dave".to_vec()), @@ -1340,7 +1340,7 @@ mod tests { ) .unwrap() .expect("exact"); - assert_eq!(exact, vec![(3 * AVG_SCALE, b"eve".to_vec())]); + assert_axis_entries_eq!(exact, vec![(3 * AVG_SCALE, b"eve".to_vec())]); // lo > hi: empty. let empty = db @@ -1394,7 +1394,7 @@ mod tests { ) .unwrap() .expect("page 1"); - assert_eq!( + assert_axis_entries_eq!( page1.entries, vec![ (25 * AVG_SCALE, b"bob".to_vec()), @@ -1414,7 +1414,7 @@ mod tests { ) .unwrap() .expect("page 2"); - assert_eq!( + assert_axis_entries_eq!( page2.entries, vec![(3 * AVG_SCALE, b"eve".to_vec()), (0, b"dave".to_vec())] ); @@ -1461,7 +1461,7 @@ mod tests { ) .unwrap() .expect("zero only"); - assert_eq!(zero_only, vec![(0i128, b"dave".to_vec())]); + assert_axis_entries_eq!(zero_only, vec![(0i128, b"dave".to_vec())]); } #[test] @@ -1489,7 +1489,7 @@ mod tests { .unwrap() .expect("asc"); // Both share avg = 7*SCALE; tie-break by original_key ascending. - assert_eq!( + assert_axis_entries_eq!( asc, vec![ (7 * AVG_SCALE, b"aaa".to_vec()), @@ -1609,7 +1609,7 @@ mod tests { ) .unwrap() .expect("count"); - assert_eq!(by_count, vec![(1u64, b"row".to_vec())]); + assert_axis_entries_eq!(by_count, vec![(1u64, b"row".to_vec())]); let by_sum = db .indexed_sum_top_k( [TEST_LEAF, b"pcpsit"].as_ref(), @@ -1620,7 +1620,7 @@ mod tests { ) .unwrap() .expect("sum"); - assert_eq!(by_sum, vec![(17i64, b"row".to_vec())]); + assert_axis_entries_eq!(by_sum, vec![(17i64, b"row".to_vec())]); let by_avg = db .indexed_avg_top_k( [TEST_LEAF, b"pcpsit"].as_ref(), @@ -1632,7 +1632,7 @@ mod tests { .unwrap() .expect("avg"); // avg = floor(17 * SCALE / 1) = 17 * SCALE. - assert_eq!(by_avg, vec![(17 * AVG_SCALE, b"row".to_vec())]); + assert_axis_entries_eq!(by_avg, vec![(17 * AVG_SCALE, b"row".to_vec())]); } // ----------------------------------------------------------------- diff --git a/grovedb/src/tests/provable_sum_indexed_tree_tests.rs b/grovedb/src/tests/provable_sum_indexed_tree_tests.rs index 324382ab6..c1e103fd2 100644 --- a/grovedb/src/tests/provable_sum_indexed_tree_tests.rs +++ b/grovedb/src/tests/provable_sum_indexed_tree_tests.rs @@ -824,7 +824,7 @@ mod tests { .indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 3, true, None, grove_version) .unwrap() .expect("top-k desc"); - assert_eq!( + assert_axis_entries_eq!( top3, vec![ (100i64, b"frank".to_vec()), @@ -838,7 +838,7 @@ mod tests { .indexed_sum_top_k([TEST_LEAF, b"psit"].as_ref(), 2, false, None, grove_version) .unwrap() .expect("bottom-2"); - assert_eq!( + assert_axis_entries_eq!( bottom2, vec![(-7i64, b"carol".to_vec()), (-1, b"eve".to_vec())] ); @@ -887,7 +887,7 @@ mod tests { ) .unwrap() .expect("asc"); - assert_eq!( + assert_axis_entries_eq!( asc, vec![ (-42i64, b"neg".to_vec()), @@ -917,7 +917,7 @@ mod tests { ) .unwrap() .expect("page 1"); - assert_eq!( + assert_axis_entries_eq!( page1.entries, vec![(100i64, b"frank".to_vec()), (12, b"bob".to_vec())] ); @@ -933,7 +933,7 @@ mod tests { ) .unwrap() .expect("page 2"); - assert_eq!( + assert_axis_entries_eq!( page2.entries, vec![(5i64, b"alice".to_vec()), (0, b"dave".to_vec())] ); @@ -991,7 +991,7 @@ mod tests { ) .unwrap() .expect("range"); - assert_eq!( + assert_axis_entries_eq!( in_range, vec![ (-1i64, b"eve".to_vec()), @@ -1014,7 +1014,7 @@ mod tests { ) .unwrap() .expect("desc"); - assert_eq!( + assert_axis_entries_eq!( desc, vec![ (12i64, b"bob".to_vec()), @@ -1037,7 +1037,7 @@ mod tests { ) .unwrap() .expect("exact"); - assert_eq!(exact, vec![(12i64, b"bob".to_vec())]); + assert_axis_entries_eq!(exact, vec![(12i64, b"bob".to_vec())]); // lo > hi: empty. let empty = db @@ -1105,7 +1105,7 @@ mod tests { ) .unwrap() .expect("neg range"); - assert_eq!(neg, vec![(-7i64, b"carol".to_vec()), (-1, b"eve".to_vec())]); + assert_axis_entries_eq!(neg, vec![(-7i64, b"carol".to_vec()), (-1, b"eve".to_vec())]); } #[test] diff --git a/grovedb/src/tests/verify_grovedb_indexed_tests.rs b/grovedb/src/tests/verify_grovedb_indexed_tests.rs index e7e0a936b..157608220 100644 --- a/grovedb/src/tests/verify_grovedb_indexed_tests.rs +++ b/grovedb/src/tests/verify_grovedb_indexed_tests.rs @@ -1781,7 +1781,7 @@ mod tests { .indexed_count_top_k([TEST_LEAF, b"cidx"].as_ref(), 15, true, None, grove_version) .unwrap() .expect("top-k over distinct derived counts"); - assert_eq!( + assert_axis_entries_eq!( top, (0..15u64) .rev() @@ -1849,7 +1849,7 @@ mod tests { ) .unwrap() .expect("ascending top-k over tied derived counts"); - assert_eq!( + assert_axis_entries_eq!( asc, (0..10) .map(|i| (SHARED_COUNT, format!("k{:02}", i).into_bytes())) diff --git a/merk/src/proofs/query/count_offset/emit.rs b/merk/src/proofs/query/count_offset/emit.rs index 392e2245c..5a7093865 100644 --- a/merk/src/proofs/query/count_offset/emit.rs +++ b/merk/src/proofs/query/count_offset/emit.rs @@ -248,8 +248,7 @@ where // Reject value shapes the count-offset proof flow does not yet // support, so the prover surfaces an explicit `NotSupported` // instead of producing a proof that silently diverges from regular - // GroveDB query semantics. Three cases, each pinned to a finding - // in the PR review: + // GroveDB query semantics. Two cases remain: // // • **NonCounted-wrapped in-range entry** (`own_struct == 0`): // regular GroveDB returns the NonCounted item's value; the @@ -258,12 +257,6 @@ where // Silently dropping it would be a correctness divergence — we // reject upfront instead. // - // • **Reference / ReferenceWithSumItem** in-range entry: regular - // GroveDB's reference post-pass dereferences these into the - // target's value bytes. The count-offset short-circuit returns - // before that post-pass, so a verified result would contain - // the raw `Element::Reference` rather than the target. Reject. - // // • **Non-empty tree** in-range entry: V1 strict-mode requires a // `KVValueHashFeatureTypeWithChildHash` proof node for these, // which the count-offset prover doesn't emit. The verifier @@ -271,8 +264,10 @@ where // time saves the work of producing an honest-but-unverifiable // proof. // - // Lifting any of these is straightforward future work: emit the - // appropriate node variant and update the verifier symmetrically. + // References are intentionally allowed: GroveDB rewrites their returned + // nodes with the resolved target before encoding. Lifting either remaining + // restriction requires emitting the appropriate node variant and updating + // the verifier symmetrically. if is_in_range { if own_struct == 0 { return Err(Error::InvalidProofError( @@ -288,17 +283,6 @@ where match Element::deserialize(value_bytes, grove_version) { Ok(elem) => { let inner = elem.into_underlying(); - if inner.is_reference() { - return Err(Error::InvalidProofError( - "count-offset paginated proofs do not yet support \ - Reference / ReferenceWithSumItem in-range entries — the regular \ - flow's reference post-pass isn't applied on the count-offset \ - short-circuit, so a verified result would expose the raw \ - Element::Reference rather than the dereferenced target" - .to_string(), - )) - .wrap_with_cost(cost); - } if inner.is_non_empty_tree() { return Err(Error::InvalidProofError( "count-offset paginated proofs do not yet support non-empty tree \ diff --git a/merk/src/proofs/query/count_offset/verify.rs b/merk/src/proofs/query/count_offset/verify.rs index 0870242d0..c4901d720 100644 --- a/merk/src/proofs/query/count_offset/verify.rs +++ b/merk/src/proofs/query/count_offset/verify.rs @@ -73,16 +73,17 @@ pub struct CountOffsetReturnedItem { /// The matched key. pub key: Vec, /// The element's serialized value bytes, as emitted by the prover. - /// GroveDB's reference-resolution post-pass (mirroring the regular - /// count-tree proof flow) operates on this byte stream — reference - /// dereferencing happens at the GroveDB layer, not here. + /// For a `KVRefValueHash*` node these are the resolved target's bytes: + /// GroveDB rewrites the raw reference node before proof encoding. pub value: Vec, /// The value-hash the proof's merk node committed for this entry. /// For `KVCount` nodes this is `H(value)` (the Item-flavored value /// hash). For `KVValueHashFeatureType` / `KVValueHash` it is the /// value-hash carried explicitly in the proof — which for /// tree-flavored entries is `combine_hash(H(value), child_root)` - /// (or `combine_hash(H(value), NULL_HASH)` for empty trees). + /// (or `combine_hash(H(value), NULL_HASH)` for empty trees). For + /// `KVRefValueHash*` it is recomputed as + /// `combine_hash(H(reference), H(resolved_target))`. /// /// Callers building `ProvedPathKeyOptionalValue` must surface this /// value (not recompute via `value_hash(value)`) so downstream @@ -173,7 +174,9 @@ pub fn verify_count_offset_on_range_proof( | Node::KVValueHashFeatureType(_, _, _, _) | Node::HashWithCountAndSum(_, _, _, _, _) | Node::KVDigestCountSum(_, _, _, _) - | Node::KVCountSum(_, _, _, _) => Ok(()), + | Node::KVCountSum(_, _, _, _) + | Node::KVRefValueHashCount(_, _, _, _) + | Node::KVRefValueHashCountSum(_, _, _, _, _) => Ok(()), other => Err(Error::InvalidProofError(format!( "unexpected node type in count-offset proof: {}", other @@ -245,6 +248,8 @@ fn aggregate_of_proof_tree_node(tree: &ProofTree) -> Result { Node::HashWithCountAndSum(_, _, _, c, _) => Ok(*c), Node::KVDigestCountSum(_, _, c, _) => Ok(*c), Node::KVCountSum(_, _, c, _) => Ok(*c), + Node::KVRefValueHashCount(_, _, _, c) => Ok(*c), + Node::KVRefValueHashCountSum(_, _, _, c, _) => Ok(*c), Node::KVValueHashFeatureType(_, _, _, ft) => match ft { TreeFeatureType::ProvableCountedMerkNode(c) => Ok(*c), TreeFeatureType::ProvableCountedSummedMerkNode(c, _) => Ok(*c), @@ -264,7 +269,7 @@ fn aggregate_of_proof_tree_node(tree: &ProofTree) -> Result { Node::KVValueHash(..) => Ok(0), // Truly unreachable: the `execute_with_options` allowlist // earlier in `verify_count_offset_on_range_proof` rejects any - // node kind that isn't one of the eight matched above before + // node kind that isn't one of the ten matched above before // this function is ever called. Keeping the arm as // `unreachable!()` is both correct (it would only ever fire // if the allowlist were widened without updating this @@ -399,6 +404,8 @@ fn verify_count_offset_shape( // Dual-axis (PCPS) per-element variants. Node::KVDigestCountSum(key, _, _, _) => key.as_slice(), Node::KVCountSum(key, _, _, _) => key.as_slice(), + Node::KVRefValueHashCount(key, _, _, _) => key.as_slice(), + Node::KVRefValueHashCountSum(key, _, _, _, _) => key.as_slice(), // Reaching here would require: // - the `execute_with_options` allowlist accepted a node // that doesn't carry a key (only `HashWithCount` / @@ -679,6 +686,48 @@ fn classify_self<'a>( value_hash: *vh, }) } + Node::KVRefValueHashCount(key, value, reference_element_hash, _) => { + if !in_range { + return Err(Error::InvalidProofError( + "count-offset proof: KVRefValueHashCount at an out-of-range position" + .to_string(), + )); + } + if own_count != 1 { + return Err(Error::InvalidProofError(format!( + "count-offset proof: KVRefValueHashCount at own_count={} (expected 1)", + own_count + ))); + } + let target_hash = compute_value_hash(value.as_slice()).unwrap(); + let combined = crate::tree::combine_hash(reference_element_hash, &target_hash).unwrap(); + Ok(BoundaryKind::ValueReturned { + key: key.as_slice(), + value: value.as_slice(), + value_hash: combined, + }) + } + Node::KVRefValueHashCountSum(key, value, reference_element_hash, _, _) => { + if !in_range { + return Err(Error::InvalidProofError( + "count-offset proof: KVRefValueHashCountSum at an out-of-range position" + .to_string(), + )); + } + if own_count != 1 { + return Err(Error::InvalidProofError(format!( + "count-offset proof: KVRefValueHashCountSum at own_count={} (expected 1)", + own_count + ))); + } + let target_hash = compute_value_hash(value.as_slice()).unwrap(); + let combined = crate::tree::combine_hash(reference_element_hash, &target_hash).unwrap(); + Ok(BoundaryKind::ValueReturned { + key: key.as_slice(), + value: value.as_slice(), + value_hash: combined, + }) + } Node::KVValueHash(key, value, _) => { // Non-count fallback. Only legitimate if the prover hit a // raw / unknown element type and fell back to the regular @@ -704,9 +753,9 @@ fn classify_self<'a>( )) } // Same fail-loud reasoning as the per-element switch in - // `verify_count_offset_shape`: only the five allowlisted node - // kinds reach `classify_self`, and the four key-bearing ones - // are handled above. The only way here is a refactor that + // `verify_count_offset_shape`: only allowlisted key-bearing node + // kinds reach `classify_self`, and all of them are handled above. + // The only way here is a refactor that // widens the allowlist without updating this match. _ => unreachable!("classify_self: dispatch unreachable for non-allowlisted node"), }