feat(grovedb,query): allow AggregateCountOnRange as carrier subquery - #663
Merged
Conversation
Allow `AggregateCountOnRange` as a subquery item under an outer `Keys`/ `Range*` carrier query — the natural multi-outer-key extension of the existing single-leg ACOR. The carrier returns one `(outer_key, u64)` pair per matched outer key while leaving the leaf wire format and existing entry points byte-identical. Why: Dash Platform's count-query module needs to answer `SELECT brand, COUNT(*) ... WHERE brand IN (...) GROUP BY brand` against `byBrand / <brand> / color / <ProvableCountTree>` layouts. Today grovedb rejects the nested shape at `Query::validate_*`, forcing callers to either fan out one proof per outer key (k× larger proofs) or fall back to the unproven path. With this change the same GroveDB proof contains all k counts and is verified in one pass. Changes: - `grovedb-query`: split validation into a leaf validator (renamed from the old `validate_aggregate_count_on_range`) and a new carrier validator that requires `Key`/`Range*` outer items, a leaf-ACOR subquery, and no conditional branches. The top-level `validate_aggregate_count_on_range` now dispatches by query shape and returns the leaf inner range either way. - `grovedb`: add `validate_leaf_aggregate_count_on_range` at SizedQuery/PathQuery for entry points that must reject the carrier. No prover changes needed — the existing recursion naturally walks outer items -> subquery_path -> leaf ACOR via `query_items_at_path`. - `grovedb`: add `verify_aggregate_count_query_per_key` returning `(RootHash, Vec<(Vec<u8>, u64)>)`. Leaf queries surface as a single entry with empty key + the same count `verify_aggregate_count_query` returns. The legacy `verify_aggregate_count_query` now strictly validates the leaf shape; carrier queries must use the new entry. - Tests: 10 new carrier-shape validation tests in grovedb-query and 7 end-to-end carrier proof tests in grovedb (two-outer-key happy path, absent-outer-key returns present-only, both-levels rejection, Range-outer carrier, leaf symmetry, non-ACOR rejection, and count forgery rejection through the carrier envelope). Asymptotic complexity for |outer matches| = k, outer tree B, leaf tree C: O(k * (log B + log C)). Out of scope (deferred): conditional-branch carriers, multi-In on prefix (IN x IN), drive-side integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR splits AggregateCountOnRange handling into explicit leaf vs carrier validations, adds query-level ACOR constructors/detectors, gates proof generation to V1 for ACOR, and implements classification-driven V1 verification (leaf-chain and per-key carrier walkers) with extensive carrier-focused tests. ChangesACOR Leaf and Carrier Shape Separation
🎯 4 (Complex) | ⏱️ ~45 minutes
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #663 +/- ##
===========================================
- Coverage 90.76% 90.76% -0.01%
===========================================
Files 184 189 +5
Lines 55791 56407 +616
===========================================
+ Hits 50639 51196 +557
- Misses 5152 5211 +59
🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
grovedb/src/tests/aggregate_count_query_tests.rs (2)
1656-1672: ⚡ Quick winMake the ordering contract observable in the explicit-key test.
The test currently supplies
brand_000thenbrand_001and asserts the same order, so it also passes if results preserve request order rather than canonical lexicographic order. Reversing the input keys would make the ordering guarantee real.Suggested tweak
let path_query = carrier_acor_path_query( - &[b"brand_000", b"brand_001"], + &[b"brand_001", b"brand_000"], QueryItem::RangeAfter(b"color_00499".to_vec()..), );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@grovedb/src/tests/aggregate_count_query_tests.rs` around lines 1656 - 1672, The test's ordering assertion isn't robust because it passes the keys in lex order; change the carrier_acor_path_query call to provide the input keys in reverse (e.g., pass b"brand_001" then b"brand_000") and update the subsequent expectations for results[0].0 and results[1].0 accordingly so the test verifies that GroveDb::verify_aggregate_count_query_per_key (used after prove_query) returns canonical lexicographic ordering rather than preserving request order.
791-796: ⚡ Quick winAssert the rejection reason, not just
InvalidQuery.This passes on any entry-point
InvalidQuery, so a future regression could still look green even if the carrier/leaf-inner validation path changes. Pin the malformed-innerKeyreason here as well.Suggested tightening
- assert!( - matches!(prove_result, Err(crate::Error::InvalidQuery(_))), - "carrier ACOR with malformed leaf-inner Key must be rejected at entry, got {:?}", - prove_result.map(|b| b.len()) - ); + match prove_result { + Err(crate::Error::InvalidQuery(msg)) => assert!( + msg.contains("AggregateCountOnRange may not wrap Key"), + "expected malformed inner ACOR rejection, got: {msg}" + ), + other => panic!( + "carrier ACOR with malformed leaf-inner Key must be rejected at entry, got {:?}", + other.map(|b| b.len()) + ), + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@grovedb/src/tests/aggregate_count_query_tests.rs` around lines 791 - 796, The test currently only checks for Err(crate::Error::InvalidQuery(_)) when calling prove_query; tighten it to assert the specific invalid-query reason for a malformed leaf-inner Key (the carrier ACOR path) by matching the concrete InvalidQuery variant produced (e.g., Err(crate::Error::InvalidQuery(crate::InvalidQueryReason::MalformedLeafInnerKey)) or the exact enum/variant used in your codebase), or by asserting the error's description/message equals the expected malformed-inner-Key reason; update the assert to match that specific variant/message instead of the wildcard underscore.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@grovedb/src/operations/proof/aggregate_count.rs`:
- Around line 95-100: The code currently runs ACOR validation on the inner Query
by calling path_query.query.query.validate_leaf_aggregate_count_on_range(),
which skips SizedQuery pagination checks (limit/offset); change the validation
call to run on the SizedQuery/PathQuery level (e.g., call
validate_leaf_aggregate_count_on_range on path_query.query or otherwise ensure
the SizedQuery wrapper is validated) and keep mapping errors via
crate::query_validation_error_to_invalid_query; apply the same change to the
other aggregate-count verification sites in this file that mirror this pattern
(the later blocks referencing inner_range/path_query and
validate_leaf_aggregate_count_on_range).
- Around line 140-141: The doc comments in aggregate_count.rs claim the returned
vector is "ascending lexicographic order" but the implementation respects the
query's left_to_right direction, so update the docs (in the comments around the
vector/return description in aggregate_count.rs and the similar blocks at the
other noted locations) to say "in query-direction (left_to_right) order" instead
of "ascending lexicographic order", or alternatively change the implementation
to sort the per-key result vector into ascending lexicographic order before
returning; mention or use the existing left_to_right flag in your change so the
behavior is explicit.
---
Nitpick comments:
In `@grovedb/src/tests/aggregate_count_query_tests.rs`:
- Around line 1656-1672: The test's ordering assertion isn't robust because it
passes the keys in lex order; change the carrier_acor_path_query call to provide
the input keys in reverse (e.g., pass b"brand_001" then b"brand_000") and update
the subsequent expectations for results[0].0 and results[1].0 accordingly so the
test verifies that GroveDb::verify_aggregate_count_query_per_key (used after
prove_query) returns canonical lexicographic ordering rather than preserving
request order.
- Around line 791-796: The test currently only checks for
Err(crate::Error::InvalidQuery(_)) when calling prove_query; tighten it to
assert the specific invalid-query reason for a malformed leaf-inner Key (the
carrier ACOR path) by matching the concrete InvalidQuery variant produced (e.g.,
Err(crate::Error::InvalidQuery(crate::InvalidQueryReason::MalformedLeafInnerKey))
or the exact enum/variant used in your codebase), or by asserting the error's
description/message equals the expected malformed-inner-Key reason; update the
assert to match that specific variant/message instead of the wildcard
underscore.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f96586f0-6339-4473-8c8c-6234f3da9593
📒 Files selected for processing (5)
grovedb-query/src/query.rsgrovedb/src/lib.rsgrovedb/src/operations/proof/aggregate_count.rsgrovedb/src/query/mod.rsgrovedb/src/tests/aggregate_count_query_tests.rs
- Move ACOR construction/validation out of `grovedb-query/src/query.rs`
(it had grown past 1500 lines) into a focused
`grovedb-query/src/aggregate_count.rs` with its own `impl Query` block
and tests. `query.rs` shrinks from ~1580 to ~910 lines.
- Per CodeRabbit major: route verifier validation through the
PathQuery layer so `SizedQuery::limit`/`offset` are rejected up
front for both leaf and carrier shapes. New tests pin the
pagination rejection.
- V0 proof envelopes can't carry the carrier shape (they're produced
only by pre-feature grove versions); the new `verify_v0_with_classification`
rejects carrier classification with `InvalidProof("only supported on V1
proof envelopes")` instead of attempting a never-emitted carrier walk.
Removes the unused `verify_v0_carrier_layer` / `verify_v0_subquery_path`.
- Per CodeRabbit minor: ordering doc on
`verify_aggregate_count_query_per_key` now says "query-direction
order" (matches the merk walker) instead of asserting strict
ascending lex, since the carrier honors `Query::left_to_right`.
- Per CodeRabbit nitpick: pin the malformed-leaf-inner-Key rejection
message in the carrier path so future refactors can't silently
re-route the error.
- New end-to-end tests: V0+carrier rejection, long subquery_path
(2 intermediate single-key layers), corrupted outer-layer merk
bytes, undecodable envelope, legacy `verify_aggregate_count_query`
rejection of carrier shapes, carrier pagination rejection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Patch coverage was 89.86% (just under codecov's 90% target). The biggest gaps were untested error paths in the carrier verifier and a few carrier-validator branches the top-level dispatcher masks. New tests: grovedb (carrier proof verifier error paths): - `acor_carrier_missing_outer_lower_layer_is_rejected` — drops one `lower_layers[outer_key]` entry from a real envelope; verifier must surface "missing lower layer for outer key". - `acor_carrier_missing_subquery_path_layer_is_rejected` — drops the subquery_path "color" layer; exercises `verify_v1_subquery_path`'s missing-layer branch. - `acor_carrier_non_merk_proof_bytes_is_rejected` — swaps a `ProofBytes::Merk(...)` layer for `ProofBytes::MMR(...)`; the `expect_merk_bytes` helper rejects. grovedb-query (direct carrier-validator branches): - `validate_carrier_acor_direct_rejects_missing_subquery` - `validate_carrier_acor_direct_rejects_acor_outer_item` - `validate_carrier_acor_direct_rejects_range_full_outer` These call `validate_carrier_aggregate_count_on_range` directly to hit branches the top-level dispatcher routes around (the dispatcher classifies first and sends ACOR-outer-item shapes to the leaf validator). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rely V0 proof envelopes (`GroveDBProofV0` / `MerkOnlyLayerProof`) predate the ACOR feature — ACOR was introduced in #656, well after V1 envelopes became the default for grove versions used by Dash Platform v12+. The V0+ACOR combination is impossible in any deployed Platform release, so the previous compatibility shims are dead code. This change makes that explicit: - Prover entry: `prove_query_non_serialized` rejects ACOR queries paired with grove versions that dispatch to V0 with `Error::NotSupported("requires V1 proof envelopes")`. The check fires before any work happens, so callers can't accidentally emit a V0 ACOR proof. - Verifier entries: both `verify_aggregate_count_query` (leaf) and `verify_aggregate_count_query_per_key` (leaf or carrier) refuse `GroveDBProof::V0` envelopes via a shared `require_v1_envelope` helper, surfacing `Error::InvalidProof`. A forged V0 envelope carrying ACOR-looking bytes can't even reach the verification logic. - Removed dead code: `verify_v0_leaf_chain` and `verify_v0_with_classification` are deleted. `MerkOnlyLayerProof` and `GroveDBProofV0` are no longer imported by `aggregate_count.rs`. - Tests updated: `provable_count_tree_works_on_grove_v2_envelope` inverts to `aggregate_count_rejects_grove_v2_envelope`, asserting the prover refuses the combination. `acor_carrier_rejects_v0_envelope` similarly tightens its assertion to expect `NotSupported` from the prover instead of `InvalidProof` from the verifier. The V0 short-circuit in `prove_subqueries` (the v0 path) stays as defense-in-depth — unreachable through the entry gate but a safety net for any future internal caller that bypasses `prove_query_non_serialized`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The carrier-layer verifier was hardcoding `left_to_right=true` when calling `merk::execute_proof`, even though the rest of the carrier classification (`AcorClassification::carrier_left_to_right`) and the prover (`generate_merk_proof(..., query.left_to_right, ...)`) both honor the query's direction. Consequence: a carrier query with `left_to_right=false` would produce a correct multi-key proof emitted in reverse order, but the verifier would feed it through `execute_proof` walking left-to-right and discover only the last matched key (the merk walker terminates as soon as the first "impossible direction" boundary is hit), returning a single result instead of all matches. The single-key descent (`verify_single_key_layer_proof_v0`) is unaffected because it's exactly-one-key, so order doesn't matter. The fix is a one-line change: pass `left_to_right` through to `execute_proof` in `execute_carrier_layer_proof`. Adds a new test `acor_subquery_right_to_left_returns_descending_order` that builds a 3-brand carrier with `left_to_right=false` and asserts results come back in descending lex order (brand_002, brand_001, brand_000), each with the correct per-brand count. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip the "CRITICAL:" framing and the hypothetical "hardcoding true would..." — a short factual note about what would actually go wrong is enough. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
grovedb/src/operations/proof/aggregate_count.rs (1)
261-270: 💤 Low valueTrailing-byte tolerance in proof decoding.
bincode::decode_from_slicereturns the number of bytes consumed but the value is discarded, so any trailing bytes appended to a valid proof are silently ignored. Since the resulting(RootHash, count)is still bound by the cryptographic chain check, this is not a correctness/security issue, but it does make the encoding non-canonical (multiple byte-strings can verify to the same result). If strict canonical-encoding is desired here, consider rejecting proofs whoseconsumed != proof.len().♻️ Optional canonical-encoding tweak
- let (proof, _) = bincode::decode_from_slice(proof, config) - .map_err(|e| Error::CorruptedData(format!("unable to decode proof: {}", e)))?; - Ok(proof) + let (decoded, consumed) = bincode::decode_from_slice(proof, config) + .map_err(|e| Error::CorruptedData(format!("unable to decode proof: {}", e)))?; + if consumed != proof.len() { + return Err(Error::CorruptedData(format!( + "proof has {} trailing bytes after decoding", + proof.len() - consumed + ))); + } + Ok(decoded)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@grovedb/src/operations/proof/aggregate_count.rs` around lines 261 - 270, decode_grovedb_proof currently discards the byte-count returned by bincode::decode_from_slice which allows trailing bytes to be ignored and makes the encoding non-canonical; change decode_grovedb_proof to capture the returned consumed length from bincode::decode_from_slice(proof, config) and if consumed != proof.len() return an Error::CorruptedData (or a new descriptive error) indicating trailing bytes, so only proofs that consume the entire input are accepted.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@grovedb/src/operations/proof/aggregate_count.rs`:
- Around line 261-270: decode_grovedb_proof currently discards the byte-count
returned by bincode::decode_from_slice which allows trailing bytes to be ignored
and makes the encoding non-canonical; change decode_grovedb_proof to capture the
returned consumed length from bincode::decode_from_slice(proof, config) and if
consumed != proof.len() return an Error::CorruptedData (or a new descriptive
error) indicating trailing bytes, so only proofs that consume the entire input
are accepted.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c8ccea73-86cd-40f1-bdfc-f9828934a20a
📒 Files selected for processing (7)
grovedb-query/src/aggregate_count.rsgrovedb-query/src/lib.rsgrovedb-query/src/query.rsgrovedb/src/operations/proof/aggregate_count.rsgrovedb/src/operations/proof/generate.rsgrovedb/src/query/mod.rsgrovedb/src/tests/aggregate_count_query_tests.rs
💤 Files with no reviewable changes (1)
- grovedb-query/src/query.rs
✅ Files skipped from review due to trivial changes (1)
- grovedb-query/src/lib.rs
The "ACOR" / "Acor" shorthand baked count into every identifier and prose mention — but the leaf/carrier shape generalizes to forthcoming aggregate variants (sum, average, …), and "ASOR" / "AAOR" don't make that lineage obvious. This commit drops the abbreviation so future aggregate modules can use parallel naming without inheriting a count-specific prefix. Renames: - `AcorClassification` → `AggregateCountClassification` - `classify_path_query` → `classify_aggregate_count_path_query` - `acor_subquery_*` test functions → `carrier_*` (the `aggregate_count_query_tests` module already scopes them) - `acor_carrier_*` → `carrier_*` - `acor_leaf_*` → `leaf_*` - `acor_per_key_*` → `per_key_*` - `carrier_acor_path_query` helper → `carrier_count_path_query` - `validate_acor_*` → `validate_aggregate_count_*` - `validate_carrier_acor_*` → `validate_carrier_aggregate_count_*` - `make_acor_query` → `make_aggregate_count_query` - `make_leaf_acor_subquery` → `make_leaf_aggregate_count_subquery` - `inner_acor*` → `inner_aggregate_count*` - `bad_inner_acor` → `bad_inner_aggregate_count` Docs/comments/error-message prose: "ACOR" → "aggregate-count" (or the full `AggregateCountOnRange` when referring to the QueryItem). Module docs note that future variants will live in sibling modules (`aggregate_sum`, `aggregate_average`, …) with parallel naming. No behavioral changes — pure rename. All 62 grovedb + 28 grovedb-query aggregate-count tests still pass; workspace tests + clippy clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The single 750-line `aggregate_count.rs` had grown to cover four
distinct concerns: public API, classification, the leaf-only chain
walker, and the per-key carrier walker (plus a pile of shared
helpers). Split into a directory module with one file per concern.
```
grovedb/src/operations/proof/aggregate_count/
├── mod.rs 227 lines — public API (impl GroveDb,
│ verify_aggregate_count_query +
│ verify_aggregate_count_query_per_key),
│ require_v1_envelope, submodule wiring
├── classification.rs 77 lines — AggregateCountClassification struct +
│ classify_aggregate_count_path_query
├── leaf_chain.rs 77 lines — verify_v1_leaf_chain (legacy
│ single-u64 walker)
├── per_key.rs 221 lines — per-key carrier walker
│ (with_classification, _per_key,
│ _carrier_layer, _subquery_path)
└── helpers.rs 270 lines — shared utilities (decode envelope,
verify_count_leaf, expect_merk_bytes,
verify_single_key_layer_proof_v0,
OuterMatch + execute_carrier_layer_proof,
enforce_lower_chain)
```
Visibility is `pub(super)` everywhere — the submodules are crate
implementation detail, only `verify_aggregate_count_query` and
`verify_aggregate_count_query_per_key` are public.
Pure code reorganization — no behavior change. All 62 grovedb +
28 grovedb-query aggregate-count tests still pass; workspace +
clippy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`bincode::decode_from_slice` returns the number of bytes consumed but
we were discarding it, so a valid proof with arbitrary trailing bytes
appended would still decode to the same envelope and verify
successfully. The chain-bound cryptographic check still gives the
same `(RootHash, count)`, so this isn't a correctness/security
problem — but it does mean a single logical proof has infinitely
many byte encodings, which breaks any equality-by-bytes assumption
(caching, deduplication, hashing the proof itself).
`decode_grovedb_proof` now compares `consumed` to `proof.len()` and
rejects with `Error::CorruptedData("trailing bytes after the encoded
envelope")` when they differ.
Test `aggregate_count_proof_with_trailing_bytes_is_rejected` pins
the rejection at both entry points (`verify_aggregate_count_query`
and `verify_aggregate_count_query_per_key`) with a real proof +
one appended zero byte.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original spec explicitly defers the "Range × Range × ACOR" / "IN × IN on prefix" composition: a carrier whose subquery is itself another carrier (rather than a leaf `AggregateCountOnRange`). The carrier validator already rejects this — it calls `validate_leaf_aggregate_count_on_range` on the subquery, which catches the inner carrier's outer items — but we didn't have a test documenting that contract. Two new tests, one per layer: - `grovedb-query::validate_carrier_aggregate_count_rejects_nested_carrier` drives the static validator with `outer.set_subquery(inner_carrier)` and asserts `InvalidOperation`. - `grovedb::rejects_nested_carrier_range_range_aggregate_count` builds a full `PathQuery` with the same shape and asserts both the static `PathQuery::validate_aggregate_count_on_range` and the prover's entry-point gate (`prove_query` → `InvalidQuery`) refuse. If we ever decide to support the nested shape, these tests are the contract that has to change first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an explicit end-to-end test that mirrors the canonical SQL
aggregate-count query
SELECT COUNT(*) FROM t WHERE a = 1 AND b > 4 AND c > 4
against an `(a, b, c)`-indexed grove laid out as
TEST_LEAF / byA / <a_val> / byB / <b_val> / byC / <count_tree>
The mapping to the carrier ACOR shape is:
- `A = 1` is a fixed prefix → `path_query.path`
- `B > 4` is the variable outer dimension → carrier's `RangeAfter`
- per matched `B`, walk `byC` → carrier's `subquery_path`
- `COUNT(C > 4)` is the leaf aggregate-count subquery
The tree contains both `A = 1` and `A = 2` so we also confirm the
fixed prefix actually scopes the result (only `A = 1` rows count).
Expected: two `(b_val, 5)` entries — `b_5` and `b_7` each have 5
`C > c_4` matches.
This shape was already supported via the existing carrier
machinery — the test just makes the SQL → grove mapping obvious
for callers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a carrier `AggregateCountOnRange` query's descent reached an
empty `Element::ProvableCountTree(None, ..)` or
`ProvableCountSumTree(None, ..)`, the V1 prover treated it as a
terminal "tree with no children" result and inserted no
`lower_layer` for that step. The carrier verifier then rejected the
proof at the chain walk with `"carrier aggregate-count proof
missing subquery_path layer for key …"`. The correct behavior is to
return `count = 0` for that outer key — an existing empty leaf
count tree is still a valid match.
The fix is a new, narrow arm in `prove_subqueries_v1` that recurses
into empty `Provable{Count,CountSum}Tree(None, ..)` elements when
both:
1. The surrounding `PathQuery` is an aggregate-count query
(`has_aggregate_count_on_range_anywhere() == true`), and
2. The current level's query has a subquery / matching subquery_path
step on this key.
The recursion opens the empty merk, hits the aggregate-count
short-circuit, runs `prove_aggregate_count_on_range` on the empty
subtree (which returns an empty op stream), and emits a
`LayerProof { merk_proof: ProofBytes::Merk(empty), .. }`. The
verifier's `verify_count_leaf` reads empty bytes as `(NULL_HASH, 0)`,
and the chain check
`combine_hash(H(empty_tree_value), NULL_HASH) == parent_value_hash`
holds because the parent committed the tree element with the same
`NULL_HASH` child root.
Non-aggregate-count queries keep their existing "empty tree =
terminal result" semantics. The narrow arm fires *before* the
general empty-tree arm and only matches on the two provable-count
flavors, so behavior is unchanged for every other tree type.
Adds `carrier_returns_zero_count_for_empty_leaf_subtree` —
inserts `byBrand/brand_000/color` as an empty
`ProvableCountTree`, runs a carrier query, and asserts the
returned vector is `[(b"brand_000", 0)]` with matching root hash.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`query_aggregate_count` is the no-proof counterpart of `verify_aggregate_count_query`: it returns a single `u64` and has no way to surface per-outer-key carrier counts. But the validator it called — `path_query.validate_aggregate_count_on_range` — is the top-level dispatcher that accepts BOTH leaf and carrier shapes. A carrier-shape path query would slip past validation here, then the subsequent `count_aggregate_on_range` call would either error (wrong merk tree type at `path_query.path`) or silently miscompute because the function only ever counts the inner range against the single subtree at `path_query.path`. Same fix we applied to `verify_aggregate_count_query` earlier: switch to `validate_leaf_aggregate_count_on_range`, which rejects the carrier shape up front. The docstring is updated to point carrier-shape callers at `verify_aggregate_count_query_per_key` (the proof-based per-key entry). New test `no_proof_rejects_carrier_shape` constructs a valid carrier path query, sanity-checks that the dispatcher-level validator still accepts it (so the rejection below is specifically from the leaf-only tightening), and asserts the no-proof entry returns `InvalidQuery` with no storage reads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@grovedb/src/operations/proof/aggregate_count/per_key.rs`:
- Around line 132-136: Replace the panic caused by
classification.carrier_subquery_path.as_ref().expect(...) with error
propagation: change to an ok_or_else that returns Error::InvalidProof with a
clear contextual message (e.g. "missing carrier_subquery_path in
classification") so the verifier treats this as a verification failure instead
of aborting; where subsequent fallible calls are made on that path, wrap/map
errors with context using .map_err(|e| Error::CorruptedData(format!("context:
{}", e))) as per the coding guideline so all failures are returned with useful
context.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4a72e90e-06f9-4f51-ad86-08bd46a9d9e6
📒 Files selected for processing (10)
grovedb-query/src/aggregate_count.rsgrovedb/src/operations/get/query.rsgrovedb/src/operations/proof/aggregate_count.rsgrovedb/src/operations/proof/aggregate_count/classification.rsgrovedb/src/operations/proof/aggregate_count/helpers.rsgrovedb/src/operations/proof/aggregate_count/leaf_chain.rsgrovedb/src/operations/proof/aggregate_count/mod.rsgrovedb/src/operations/proof/aggregate_count/per_key.rsgrovedb/src/operations/proof/generate.rsgrovedb/src/tests/aggregate_count_query_tests.rs
💤 Files with no reviewable changes (1)
- grovedb/src/operations/proof/aggregate_count.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- grovedb-query/src/aggregate_count.rs
- grovedb/src/tests/aggregate_count_query_tests.rs
…tion drift `verify_v1_carrier_layer` was `.expect()`-ing that `classification.carrier_subquery_path` is `Some` whenever `carrier_outer_items` is `Some`. That invariant is held today by `classify_aggregate_count_path_query`, but a panic is the wrong failure mode if the invariant ever drifts — a verifier should surface verification failures, not abort. Replace with `ok_or_else` returning `Error::InvalidProof` with a contextual message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the leaf/carrier asymmetry on the no-proof side. Before this
commit:
proof path: verify_aggregate_count_query (leaf only, -> u64)
verify_aggregate_count_query_per_key (leaf or carrier,
-> Vec<(key, u64)>)
no-proof: query_aggregate_count (leaf only, -> u64)
<missing carrier sibling>
A carrier-shape caller that didn't need a proof had to go through
`prove_query` + `verify_aggregate_count_query_per_key` anyway —
paying proof generation, serialization, and chain verification just
to throw the proof bytes away.
`query_aggregate_count_per_key` mirrors the proof-path surface
(returns `Vec<(Vec<u8>, u64)>`) and accepts the same shapes:
- Leaf path query: delegates to `query_aggregate_count` and wraps
as a one-entry vec with an empty key — matches the proof-path
leaf-symmetry contract.
- Carrier path query: opens the carrier subtree via `query_raw`
with a "shallow" path query (carrier's outer items, no subquery)
to enumerate matched outer keys. For each match, walks
`subquery_path` manually and calls `count_aggregate_on_range` on
the leaf merk. Absent outer keys contribute no entry; outer keys
whose leaf is an empty `ProvableCountTree` contribute `(key, 0)`.
Validation is the dispatcher-level
`validate_aggregate_count_on_range` (accepts both shapes); non-ACOR
queries get `InvalidQuery` before any storage reads. Non-tree
matches on the carrier level surface `InvalidQuery` rather than the
merk-level `count_aggregate_on_range` error.
Six tests pin the contract:
- `no_proof_per_key_leaf_matches_single_count` — leaf shape returns
the same `u64` as `query_aggregate_count`, wrapped.
- `no_proof_per_key_carrier_returns_per_outer_count` — two outer
brands, 500 colors each -> two entries.
- `no_proof_per_key_skips_absent_outer_keys` — absent outer key
contributes no entry.
- `no_proof_per_key_empty_leaf_returns_zero` — outer key exists,
leaf is empty `ProvableCountTree` -> `(key, 0)`.
- `no_proof_per_key_matches_proof_path_per_key` — element-for-element
equality with `verify_aggregate_count_query_per_key` on a
non-trivial 3-brand carrier query.
- `no_proof_per_key_rejects_non_aggregate_count_query` — non-ACOR
path queries rejected with `InvalidQuery`.
Reuses the existing `query_aggregate_count_on_range` version gate
since this is another entry point for the same feature, not a
distinct protocol change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
|
Self Reviewed. |
QuantumExplorer
added a commit
to dashpay/platform
that referenced
this pull request
May 15, 2026
…+ bench probe Pulls in dashpay/grovedb#663 ("feat(grovedb,query): allow AggregateCountOnRange as carrier subquery"), merge commit 87554188ad — bumps every pinned `grovedb*` dep across the workspace from a917d92d to 87554188. The new grovedb APIs: - `Query::validate_carrier_aggregate_count_on_range` - `Query::validate_leaf_aggregate_count_on_range` - `GroveDb::query_aggregate_count_per_key` - `GroveDb::verify_aggregate_count_query_per_key` These unblock the chapter 30 G7 shape (`brand IN[...] AND color > floor` with `group_by = [brand]`) at the grovedb layer. Drive wire-up is the next step (lift mode_detection.rs:140 rejection + new path-query builder + new DocumentCountMode variant + per-key verifier wrapper). This commit also adds `probe_carrier_acor` to document_count_worst_case bench as a feasibility smoke test against the existing 100 000-row widget fixture: [carrier-acor] no-proof entries (2): ("brand_000", 499) ("brand_001", 499) [carrier-acor] proof bytes: 4332 B [carrier-acor] verified root_hash: 0x62ee7348f4d28dd9d7cf86a6c725fa8276... [carrier-acor] verified entries (2): ("brand_000", 499) ("brand_001", 499) 4 332 B for k=2, matching the complexity estimate documented in chapter 30 (`O(k · (log B + log C'))`) and ~17 % smaller than the concatenated-leaf-ACOR alternative. Verification: - cargo check --workspace clean - cargo test -p drive --lib drive_document_count_query → 45 tests passing - cargo test -p drive --lib verify → 240 tests passing - All chapter 29 / chapter 30 proof sizes (585 / 1041 / 1327 / 1911 / 1102 / 1381 / 2072 / 2656 B for Q1..Q8) unchanged — bump is purely additive - Carrier-ACOR end-to-end: query → prove → verify all round-trip with root hash matching the chapter's known 0x62ee7348... fixture root Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
4 tasks
QuantumExplorer
added a commit
that referenced
this pull request
May 15, 2026
…0 envelopes V0 (`MerkOnlyLayerProof`) envelopes predate the aggregate-sum feature and cannot legitimately carry an aggregate-sum proof, so the V0 layer walker was unreachable in any honestly-produced proof. Mirror the count-side changes from PR #663: - Convert `aggregate_sum.rs` into a subdirectory mirroring `aggregate_count/`: `mod.rs` (public API + V0 rejection), `helpers.rs` (envelope decode, single-key layer verification, chain enforcement, leaf sum verification), `leaf_chain.rs` (V1 leaf-chain walker). Removes the dead `verify_v0_layer` path. - Add a prover-side V0 gate in `prove_query_non_serialized`: when grove_version dispatches to V0 and the path query carries an `AggregateSumOnRange` anywhere, return `NotSupported` instead of emitting a V0 envelope the verifier would (correctly) reject. - Update tests: replace the V0 round-trip test with a V0-rejection test; broaden the empty-leaf type-confusion test to accept either the V0-rejection or the terminal-type-gate error; remove the now- unreachable V0 missing-lower-layer test (V1 counterpart already pins the missing-layer behavior); refresh stale doc-comments that pointed at `aggregate_sum.rs` line numbers. No carrier-shape support yet — `AggregateSumOnRange` is still leaf-only on the merk side, so there is no `classification.rs` / `per_key.rs` mirror. The leaf-only shape can be extended later in parallel with matching merk-level work, just like the count side did. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QuantumExplorer
added a commit
that referenced
this pull request
May 16, 2026
…SumTree Pulls in PR #666 which adds the `Element::NotCountedOrSummed` wrapper — the third aggregation-suppression wrapper, suppressing BOTH count and sum contributions when inserted into a CountSumTree / ProvableCountSumTree parent — alongside PRs #663 / #664 (aggregate-count carrier improvements) already on develop. PR #666's contract specified four sum-bearing inner types (SumTree, BigSumTree, CountSumTree, ProvableCountSumTree). Because this branch adds `Element::ProvableSumTree` (the new fifth sum-bearing variant), the wrapper's allow-list is widened here to include it, matching the NotSummed allow-list extension we already did on the sum side. ## Conflict resolution ### Element / ElementType discriminants - `Element::ProvableSumTree` stays at variant 17 (our branch). - `Element::NotCountedOrSummed` slides from 17 (develop) to 18. - `NOT_COUNTED_OR_SUMMED_WRAPPER_DISCRIMINANT` becomes 18. - New synthetic twin `NotCountedOrSummedProvableSumTree = 193` (`0xC1`), hand-assigned because base 17 overflows the low-nibble formula — same reason `NotSummedProvableSumTree = 177` is hand-assigned at `0xB1`. Other four twins (196/197/199/202) keep the `0xC0 | base` formula slots. - `ElementType::base()` falls back to a per-variant match for ALL hand-assigned wrapper twins (NotSummed + NotCountedOrSummed). The develop-side `disc & NOT_*_BASE_MASK` paths are replaced because they'd collide with the per-variant slot assignments. ### Allow-list extensions `ProvableSumTree` joins the inner allow-list in every gate: - `Element::new_not_counted_or_summed` constructor. - `Element::validate_wrapper_invariants` (deserialize post-check). - `Element::serialize` pre-check. - `Element::deserialize` post-check. - All matching test data sets and rejection bytes. ### Batch propagation `GroveOp::InsertTreeWithRootHash` carries a new `not_counted_or_summed: bool` field (added by #666). Our branch's `Element::ProvableSumTree` build site was updated to set it from the upstream `not_counted_or_summed` local, matching the other tree variants. ### Test wrapper-byte fix The `deserialize_rejects_long_nested_wrapper_chain_without_recursion` test fills 1024 bytes with the wrapper discriminant. PR #666 used `17`; in this branch it must be `18` because slot 17 is `ProvableSumTree`. ## New tests - `direct_insert_not_counted_or_summed_provable_sum_tree_in_count_sum_tree_excludes_both_axes`: wrapped ProvableSumTree inside a CountSumTree parent — outer aggregate must read `(count=1, sum=7)` (only the unwrapped sum item contributes). - `direct_insert_not_counted_or_summed_provable_sum_tree_in_provable_count_sum_tree_excludes_both_axes`: same scenario but with a ProvableCountSumTree parent. - `constructor_invariants` extended to also accept `new_not_counted_or_summed(new_provable_sum_tree(None))`. ## Verification - `cargo build --workspace`: clean - `cargo clippy --workspace --all-features`: clean - `cargo test --workspace`: 3138 / 0 fail (3136 → 3138, +2 new end-to-end ProvableSumTree+NotCountedOrSummed tests; remaining delta from the develop merge bringing in #663/#664/#666 tests). - `cargo test -p grovedb-element --lib`: 99 / 0 fail - `cargo test -p grovedb --lib not_counted_or_summed`: 11 / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QuantumExplorer
added a commit
that referenced
this pull request
May 17, 2026
* feat(types): add Element::ProvableSumTree variant + NotSummed twin extension
Phase 1 of the ProvableSumTree feature — the missing parallel to
ProvableCountTree that bakes the per-node sum into the node hash, making
aggregate-sum range queries cryptographically verifiable. This commit adds
the types-only foundation (no hash divergence yet — Phase 2 will introduce
node_hash_with_sum and the new proof Node variants).
DISCRIMINANTS
- Element::ProvableSumTree at variant index 17 / bincode discriminant 17
(next free after the NotSummed wrapper byte at 16). This will renumber
to 19 when PR #657 (CountIndexedTree) lands and reclaims 17/18.
- NonCountedProvableSumTree = 0x80 | 17 = 145.
- The NonCounted twin range widened from 0x80..=0x8F (4-bit base) to
0x80..=0x9F (5-bit base) — is_non_counted() now checks the top 3 bits
(& 0xe0 == 0x80) instead of the top 4. Existing twins 128..=142 stay put.
- The NotSummed twin scheme rebases analogously: prefix 0xb0 -> 0xa0,
base mask 0x0F -> 0x1F, family range 0xA0..=0xBF. Existing twins move:
NotSummedSumTree 180 -> 164
NotSummedBigSumTree 181 -> 165
NotSummedCountSumTree 183 -> 167
NotSummedProvableCountSumTree 186 -> 170
Plus the new NotSummedProvableSumTree = 0xa0 | 17 = 177.
Safe because V1 is pre-shipping. is_not_summed() now uses & 0xe0 == 0xa0.
NEW APIS
- ElementType::ProvableSumTree, ElementType::NonCountedProvableSumTree,
ElementType::NotSummedProvableSumTree.
- TreeType::ProvableSumTree (discriminant 11, is_sum_bearing = true,
allows_sum_item = true, inner_node_type = ProvableSumNode).
- NodeType::ProvableSumNode and TreeFeatureType::ProvableSummedMerkNode(i64)
with encode tag byte 7 and a parallel zero_sum() helper alongside
zero_count().
- Element::new_provable_sum_tree*, empty_provable_sum_tree*, plus helpers
(as_provable_sum_tree_value, is_provable_sum_tree).
- Element::NotSummed now accepts ProvableSumTree as a sum-tree inner type
(constructor, serialize, deserialize).
PROOF DISPATCH
ProvableSumTree joins the "provable aggregate parent" family alongside
ProvableCountTree / ProvableCountSumTree in ElementType::proof_node_type:
subtree children use KvValueHashFeatureType and item children use KvCount.
PHASE-1 SCOPE BOUNDARIES
ProvableSumTree behaves identically to SumTree for storage, aggregation,
and hashing in Phase 1. The divergent node_hash_with_sum and the new
proof Node variants (KVSum, KVHashSum, etc.) land in Phase 2.
TreeFeatureType::ProvableSummedMerkNode maps to AggregateData::Sum at the
Element/aggregate level for now; Phase 2 may introduce a dedicated variant
once the hash diverges.
Workspace cargo test --all-features green (1497+ tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(merk): node_hash_with_sum + proof Node variants for ProvableSumTree
Phase 2 of the ProvableSumTree feature — bakes the per-node sum into the
node hash so it becomes cryptographically committed via the parent's
hash chain, parallel to how `node_hash_with_count` commits the count
for `ProvableCountTree`. After this commit a `ProvableSumTree` with the
same {key/value/sum} contents as a plain `SumTree` produces a different
root hash, which is the whole point of the Phase 2 divergence.
Phase 1 (commit c95cf749) was types-only; aggregation, storage, and
hashing all used the SumTree code paths. Phase 2 introduces the new
hash function, the new proof-node variants needed to transport sums
through proofs, and the dispatch wiring on both prover and verifier
sides. Phases 3 (insert/read), 4 (verify_grovedb walk), and 5
(AggregateSumOnRange) remain.
HASH DISPATCH
- `merk::tree::hash::node_hash_with_sum(kv, l, r, i64)` mirrors
`node_hash_with_count` byte-for-byte except the appended 8-byte
field is `i64::to_be_bytes()`. Negative sums hash via their
two's-complement BE form, which is platform-independent.
- New `AggregateData::ProvableSum(i64)` variant. The
`From<TreeFeatureType>` conversion now maps
`ProvableSummedMerkNode(v) -> ProvableSum(v)` (was `Sum(v)` in
Phase 1) so `Tree::hash_for_link` and the commit path can
dispatch through the new arm.
- `Tree::hash_for_link(TreeType::ProvableSumTree)` and both commit
paths (left/right Link::Modified arms) now call
`node_hash_with_sum` when the aggregate is `ProvableSum`.
`Tree::aggregate_data` for `ProvableSummedMerkNode` yields
`ProvableSum` instead of `Sum`.
- Helper updates: `child_aggregate_sum_data_as_i64` /
`child_aggregate_sum_data_as_i128` treat `ProvableSum`
identically to `Sum`; `child_aggregate_count_data_as_u64`
returns 0. `child_ref_and_sum_size` covers the new variant.
- `Link::encode_into` / `decode_into` learn tag byte 7 for
`AggregateData::ProvableSum` (parallel to the existing
`ProvableSummedMerkNode` tag byte 7 in `TreeFeatureType`).
- `grovedb::batch` `InsertTreeWithRootHash` now reconstructs an
`Element::ProvableSumTree` when seeing `AggregateData::ProvableSum`.
PROOF NODE VARIANTS
Five new `Node` enum variants in `grovedb-query/src/proofs/mod.rs`,
mirroring the Count family member-for-member but with `i64` sums:
- `KVSum(key, value, sum)` — sum analogue of `KVCount`
- `KVHashSum(kv_hash, sum)` — analogue of `KVHashCount`
- `KVRefValueHashSum(key, ref_value, ref_elem_hash, sum)`
- `KVDigestSum(key, value_hash, sum)` — analogue of `KVDigestCount`
- `HashWithSum(kv_hash, l, r, sum)` — analogue of `HashWithCount`
`merk::proofs::tree::Tree::hash()` now dispatches each new variant
through `node_hash_with_sum`. `KVValueHashFeatureType` /
`...WithChildHash` handling gains a `ProvableSummedMerkNode` arm so
proof-tree hashes recomputed from a Sum-bearing feature_type match
the Merk-tree side. `aggregate_data()` returns `ProvableSum(sum)`
for `KVSum` and `HashWithSum`; `key()` lists the three key-bearing
new variants alongside their Count counterparts.
`grovedb-element::ProofNodeType` gains `KvSum` and
`KvRefValueHashSum`; `ElementType::proof_node_type` now picks them
when the parent is `ProvableSumTree` (Phase 1 routed Sum-tree
children through the Count dispatch). Subtrees inside ProvableSum
still use `KvValueHashFeatureType` since the feature_type carries
the sum.
Proof generation in `merk/src/proofs/query/mod.rs` adds
`to_kv_sum_node`, `to_kvhash_sum_node`, `to_kvdigest_sum_node`
(parallel to the Count helpers) and an `is_provable_sum_tree`
branch that emits Sum-bearing variants. `chunks.rs`'s
`create_proof_node_for_chunk` dispatches the new ProofNodeType
arms.
GroveDB-side reference post-processing in
`grovedb/src/operations/proof/generate.rs` rewrites the merk-level
`KVValueHashFeatureType(_, _, _, ProvableSummedMerkNode(sum))` to
`KVRefValueHashSum`, mirroring the existing
`KVValueHashFeatureType -> KVRefValueHashCount` path. Both
ref-rewriting loops in that file are updated.
The regular query verifier in `merk/src/proofs/query/verify.rs`
rejects `HashWithSum` at non-aggregate positions (fail-fast,
matching the existing `HashWithCount` guard). `KVSum`,
`KVDigestSum`, and `KVRefValueHashSum` are dispatched via
`execute_node`. `KVHashSum` joins `KVHash` / `KVHashCount` in the
"non-data-bearing on path" branch and in the absence-proof
boundary set.
WIRE FORMAT
Tag bytes 0x30..=0x3D in the previously-unused 0x30..0x3F range:
Push variants (V0 short + V1 wrapper for KV-style large values):
0x30 = KVSum (small), 0x31 = KVSum (large)
0x32 = KVHashSum
0x33 = KVRefValueHashSum (small), 0x34 = KVRefValueHashSum (large)
0x35 = KVDigestSum
0x36 = HashWithSum
PushInverted parallel:
0x37 = KVSum (small), 0x38 = KVSum (large)
0x39 = KVHashSum
0x3a = KVRefValueHashSum (small), 0x3b = KVRefValueHashSum (large)
0x3c = KVDigestSum
0x3d = HashWithSum
0x3e and 0x3f are intentionally reserved.
The on-wire i64 sum uses varint (via `ed::Encode for i64`) for
compactness, matching the Count family. The hash recomputation in
`node_hash_with_sum` uses the fixed 8-byte big-endian form
independently — wire encoding and hash input are deliberately
decoupled. `encoding_length()` and `Decode` arms parallel the Count
family verbatim.
V0 wire format is unchanged. All new tags are V1-only.
TESTS
- `merk::tree::hash` (4): `node_hash_with_sum` differs from
`node_hash` even at sum=0; different sums give different hashes;
`i64::MIN` / `i64::MAX` are distinct; determinism.
- `merk::tree` (2): a `ProvableSummedMerkNode` tree aggregates to
`ProvableSum`, `hash_for_link(ProvableSumTree)` matches
`node_hash_with_sum(...)` and diverges from plain
`Tree::hash()`; mutating a node sum changes the root hash.
- `merk::proofs::tree` (4): forged sums on `HashWithSum`,
`KVSum`, `KVHashSum` change the recomputed node hash;
Phase 1 -> Phase 2 cornerstone — same {key/value/sum} contents
give a different ProvableSumTree hash than a plain SumTree.
- `grovedb-query::proofs::encoding` (4): round-trip every new
variant through `Op::Push` and `Op::PushInverted` at sum
values {`i64::MIN`, -42, -1, 0, 1, 42, `i64::MAX`}; tag-byte
sanity check for all 10 new tags.
- `merk::tree::tree_feature_type`: extended every existing
`AggregateData` test to cover the new `ProvableSum` variant.
Workspace `cargo test --all-features` green: 2881 tests passing,
zero failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(types): NotSummed twin uses explicit per-variant mapping
Phase 1.6 — revert the Phase 1.5 mask widening. The NotSummed family
returns to its original 4-bit prefix `0xb0` / range `0xB0..=0xBF`. The
five legal sum-tree inner types are mapped to twin slots explicitly,
1-at-a-time, instead of via the `prefix | base` bitwise formula:
SumTree (base 4) -> 180 (0xB4)
BigSumTree (base 5) -> 181 (0xB5)
CountSumTree (base 7) -> 183 (0xB7)
ProvableCountSumTree (base 10) -> 186 (0xBA)
ProvableSumTree (base 17) -> 177 (0xB1)
Existing twins return to their original discriminant values; only the
new ProvableSumTree slot is freshly assigned. Pre-shipping V1, so this
discriminant churn is fine.
WHY EXPLICIT MAPPING
`prefix | inner_byte` can only generate twin slots when the inner
discriminant fits in the prefix's complement nibble. For ProvableSumTree
at base 17, the formula `0xb0 | 17` would produce `0xB1` AND then
`disc & 0x0F` would invert it back to base 1 (Reference) — a collision.
Widening the mask to 5 bits in Phase 1.5 rebased every existing twin
discriminant; reverting to per-variant mapping keeps the historical
values stable while still allowing arbitrary new slot assignments.
CONSEQUENCES
- `NOT_SUMMED_TWIN_PREFIX` stays as a const but is now only a family-range
marker, never composed with a base byte.
- `NOT_SUMMED_BASE_MASK` removed — no remaining callers.
- `is_not_summed()` back to `& 0xf0 == 0xb0`.
- `base()` for NotSummed now uses an explicit per-variant match.
- `from_serialized_value` NotSummed branch uses an explicit
`inner_byte → twin_variant` match.
NonCounted is unaffected — it still uses the bitwise formula because all
its bases fit cleanly in the low 5 bits under `0x80`.
Workspace cargo test --all-features green (2881 tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(grovedb): wire ProvableSumTree through insert/read/batch paths
Phase 3 of the ProvableSumTree feature — wires the variant through the
extension traits, cost calculator, reconstruction helper, batch
propagation, and read-path subtree validation so that direct insertion,
nested aggregation, and child-sum mutation behave correctly end-to-end.
Phase 1 (commit c95cf749) added the variant and its twins; Phase 2
(commit 3364f08c) introduced node_hash_with_sum and the proof Node
family. The "behave like SumTree" fallback Phase 1 leaned on covered
most surfaces but several dispatch sites guarded subsequent operations
through explicit per-variant match arms — those sites would silently
drop ProvableSumTree or fail to traverse into it. Phase 3 fills each
in deliberately.
EXTENSION TRAITS — merk/src/element/tree_type.rs
ElementTreeTypeExtensions had six trait methods that enumerated tree
variants explicitly: root_key_and_tree_type_owned, root_key_and_tree_type,
tree_flags_and_type, tree_type, maybe_tree_type, tree_feature_type. Each
was missing its ProvableSumTree arm — get_feature_type was the only one
already wired (Phase 1). Adding the missing arms unblocks callers across
get, batch, and visualize that thread tree types through these helpers
to decide layout, hashing, and aggregate-data extraction.
tree_feature_type now maps ProvableSumTree -> ProvableSummedMerkNode(sum)
explicitly, matching the parallel ProvableCountedMerkNode wiring.
COST CALCULATOR — merk/src/element/costs.rs
get_specialized_cost, the layered_value_byte_cost path in
specialized_costs_for_key_value, the layered_value_defined_cost type
filter, and the value_defined_cost dispatch all enumerated the eight
Merk-tree variants and would have either returned None or mis-sized a
ProvableSumTree element. Added explicit ProvableSumTree => SUM_TREE_COST_SIZE
arms (parity with SumTree as established in Phase 1) and the matching
LayeredValueDefinedCost branch.
RECONSTRUCTION — merk/src/element/reconstruct.rs
ElementReconstructExtensions::reconstruct_with_root_key, used by batch
propagation to rebuild a tree element after a root-key update, returned
None for ProvableSumTree. Added the arm that pulls
aggregate_data.as_sum_i64() into Element::ProvableSumTree(root, sum,
flags); without this, batch operations that mutated a ProvableSumTree
subtree would lose their tree element entirely during the parent's
upward propagation. as_sum_i64 already handles the AggregateData::ProvableSum
case (Phase 2).
BATCH PROPAGATION — grovedb/src/batch/mod.rs
The InsertTreeWithRootHash else-if chain that transcribes a Merk-tree
mutation into the appropriate root-hash-bearing operation enumerated
each tree-element variant explicitly. ProvableSumTree was missing,
so a batch that mutated a ProvableSumTree subtree would fall through
to the CommitmentTree arm — wrong shape entirely. Mirrored the
ProvableCountSumTree arm directly. The accompanying tree-cost match
list above (used by the apply_batch storage-cost callback) was also
missing the variant.
READ-PATH SUBTREE VALIDATION — grovedb/src/operations/get/mod.rs
check_subtree_exists rejected paths whose final segment resolved to a
ProvableSumTree because the variant wasn't in its accepted-tree
match list. This would have broken every query that traversed INTO a
ProvableSumTree.
TESTS — grovedb/src/tests/provable_sum_tree_tests.rs
Ten tests covering Phase 3's externally-observable surface:
- Round-trip insert/read with aggregate-sum tracking.
- Aggregation across mixed positive/negative/zero values + i64::MIN/
i64::MAX extremes.
- Root-hash divergence vs a plain SumTree with identical children
(the Phase 2 cornerstone, verified end-to-end via
open_transactional_merk_at_path).
- Nested ProvableSumTree[A] -> ProvableSumTree[B] aggregate propagation;
mutation of B's children shifts the grovedb root hash.
- Wrapper interactions: NonCounted(ProvableSumTree) contributes 0 to a
CountTree parent; NotSummed(ProvableSumTree) contributes 0 to a
SumTree parent; the wrapped tree's own aggregate is preserved
(verified via get_raw, which retains the wrapper byte).
- Deleting a SumItem child shifts the ProvableSumTree root hash because
the aggregate sum is hash-bound.
- Direct insert of a ProvableSumTree built from an existing template.
The wrapper round-trip tests use db.get_raw rather than db.get because
db.get strips wrappers via into_underlying — by design.
Workspace cargo test --all-features green: 2891 tests passing (was
2881 in Phase 2 + 10 new), zero failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(grovedb): verify_grovedb consistency check for aggregate fields
Phase 4 of the ProvableSumTree feature. The existing
`verify_merk_and_submerks_in_transaction` walk is cryptographically
complete — `combine_hash(value_hash(parent_bytes), inner_merk_root) ==
stored_element_value_hash` catches any byte-level tampering, and for
ProvableSumTree the inner aggregate is bound into the inner Merk's
root_hash via `node_hash_with_sum` (Phase 2). What that walk did not
catch was the *software-consistency* class of drift: a parent
`ProvableSumTree(_, N, _)` whose stored sum field N disagrees with the
inner Merk's actual `aggregate_data()` value M. For provable variants
both N and M are bound into element_value_hash, but they live on disk
independently and could disagree if Phase 3's propagation logic drifts.
For non-provable variants (SumTree, BigSumTree, CountTree, CountSumTree)
the recorded aggregate isn't hash-bound at all, so a pure software bug
in propagation would silently corrupt the tree.
LIB.RS — verify_merk_and_submerks_in_transaction
After the existing cryptographic check, for any tree element whose
inner Merk holds actual data (i.e. excluding the non-Merk-data trees
CommitmentTree/MmrTree/BulkAppendTree/DenseTree, which already
short-circuit via `uses_non_merk_data_storage`), the verifier now opens
the inner Merk, reads its `aggregate_data()`, and compares against the
parent's recorded aggregate field via a new free helper
`aggregate_consistency_labels`. The helper covers all seven aggregate-
bearing tree variants:
- SumTree vs. AggregateData::Sum
- ProvableSumTree vs. AggregateData::ProvableSum
- BigSumTree vs. AggregateData::BigSum
- CountTree vs. AggregateData::Count
- CountSumTree vs. AggregateData::CountAndSum
- ProvableCountTree vs. AggregateData::ProvableCount
- ProvableCountSumTree vs. AggregateData::ProvableCountAndSum
Plus an empty-Merk identity case (NoAggregateData with zero recorded
aggregate matches), and a fallback that reports any variant-shape
mismatch (e.g. ProvableSumTree paired with AggregateData::Count(_)).
VERIFICATIONISSUES SHAPE — placeholder hashes, not type extension
`VerificationIssues` is a private type alias
`HashMap<Vec<Vec<u8>>, (CryptoHash, CryptoHash, CryptoHash)>` whose
shape is consumed by `visualize_verify_grovedb`. To avoid breaking
its callers and the visualize hex output, mismatched aggregates are
packed into deterministic placeholder CryptoHashes via
`blake3(format!("recorded ..."))` and `blake3(format!("inner ..."))`,
slotted into the "expected" and "actual" fields. The "root" slot
reuses the inner-Merk root_hash for path locality. Documented inline.
INTEGRITY WALK TESTS — 7 new tests
A new `integrity_walk_tests` module in `provable_sum_tree_tests.rs`
exercises the verifier end-to-end via two raw-storage tampering
helpers:
- `tamper_value_no_hash_update` decodes the on-disk TreeNode for a
leaf, replaces only its element bytes, re-encodes (leaving the
stored value_hash stale), writes back via the immediate storage
context. Simulates byte-level tampering caught by the SumItem
arm's `value_hash(bytes) != stored_value_hash` check.
- `tamper_parent_element_with_consistent_hashes` splices in fresh
element bytes AND recomputes hash + value_hash to remain
crypto-consistent with the inner Merk's existing root_hash. Used
for aggregate-mismatch scenarios — the crypto check passes,
but the new aggregate-consistency check fires. Offsets into the
on-disk TreeNodeInner encoding are derived from the decoded
`value_as_slice().len()`.
Scenarios covered:
1. Inner SumItem value tamper (different bytes) — crypto check
catches it.
2. Inner SumItem same-length value tamper — crypto check catches
it (assert: hashes, not lengths, are what's verified).
3. Parent ProvableSumTree aggregate mismatch (sum=999 stored vs.
40 actual) — new aggregate-consistency check fires.
4. Clean ProvableSumTree verifies clean (with mixed positive,
negative, zero, and large values).
5. Clean ProvableCountTree verifies clean.
6. Parent ProvableCountTree aggregate mismatch (count=9999 vs. 3)
— sanity check that the generalized helper handles the count
variant too.
7. Reload-after-write determinism: insert, drop the db handle,
reopen, verify_grovedb reports zero issues; the parent's
ProvableSumTree.sum_value field round-trips.
Workspace cargo test --all-features green: 2898 passing
(Phase 3 baseline of 2891 + 7 new), zero failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: AggregateSumOnRange query + proof + verify for ProvableSumTree
Adds the marquee Phase 5 feature for ProvableSumTree: a query that asks
"what's the cryptographically-verifiable signed sum of children with keys
in range [a, b]?" against a ProvableSumTree, with proof size O(log n + |boundary|)
and a verify path that returns the root hash plus the aggregate i64 sum.
Mirrors AggregateCountOnRange line-for-line:
- QueryItem::AggregateSumOnRange(Box<QueryItem>) variant (wire tag 11)
- Query / SizedQuery / PathQuery::validate_aggregate_sum_on_range with the
same nested-rejection, no-subquery, no-pagination, allowed-inner-range rules
- merk/src/proofs/query/aggregate_sum.rs (~760 lines) implementing
create_aggregate_sum_on_range_proof + verify_aggregate_sum_on_range_proof
with the same Disjoint/Contained/Boundary classification, HashWithSum
self-verifying compression at fully-inside/outside subtrees, and
KVDigestSum at boundaries
- grovedb/src/operations/proof/aggregate_sum.rs (~330 lines) for the
GroveDB-level multi-layer envelope chain check
- prove_query / verify_query dispatch in generate.rs and verify.rs
- Tree-type rejection arms in BulkAppendTree, DenseTree, MMR for the new variant
Key correctness points handled differently from count:
- i128 accumulator throughout the verifier (sum can validly be 0 with
non-zero children, so no "if sum == 0" short-circuit; final narrow to
i64 with an explicit overflow error)
- No checked_sub equivalent for own_sum derivation — signed sums make
arithmetic-only corruption detection meaningless; the hash chain binds
the values regardless
- ProvableSumTree-only at the merk-level gate (Sum/BigSum use different
hash dispatches and can't host this proof shape)
Tests: 35 new tests total (14 merk-level in aggregate_sum.rs, 21 GroveDB-
level in aggregate_sum_query_tests.rs) covering empty trees, single-key
ranges, full/sub/boundary ranges, negative sums, mixed-sign extremes
including i64::MAX + i64::MIN = -1, tampering rejection, wrong-tree
rejection, validation rejection of nested/Key/RangeFull/orthogonal-aggregate
inners, multi-layer paths, NotSummed-wrapped subtree exclusion, V0 envelope
round-trip. Workspace test count: 2898 → 2938, zero failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: ProvableSumTree element + AggregateSumOnRange query
Final phase of the ProvableSumTree feature — documentation. Adds:
- `docs/book/src/aggregate-sum-on-range-queries.md`: new dedicated chapter
describing the AggregateSumOnRange query, the ProvableSumTree tree type
it operates on, why the existing sum trees can't be queried this way,
the proof node vocabulary (KVSum / KVHashSum / HashWithSum / KVDigestSum
/ KVRefValueHashSum at wire tags 0x30..=0x3D), and the signed-sum
correctness notes (no zero-sum short-circuit; i128 accumulator with
i64 narrowing at the entry points; overflow handling at i64::MAX
extremes).
- `docs/book/src/element-system.md`: ProvableSumTree row added to the
aggregate-tree table; ProvableSummedMerkNode added to the
TreeFeatureType enum block; NonCounted/NotSummed wrapper indices
surfaced; explanation of when to choose ProvableSumTree over plain
SumTree (sum is part of the protocol invariant vs metadata) and the
rationale for the explicit `NotSummedProvableSumTree = 177` slot.
- `docs/book/src/hashing.md`: parallel "Aggregate Hashing for
ProvableSumTree" section showing node_hash_with_sum's i64 BE input
layout and the wire-vs-hash encoding split.
- `docs/book/src/appendix-a.md`: rows for NonCounted (15), NotSummed
(16), and ProvableSumTree (17) added to the discriminant table.
- `docs/book/src/aggregate-sum-queries.md`: disambiguation banner at the
top distinguishing the existing sum-budget iterator from the new
AggregateSumOnRange query, with a cross-link.
- `docs/book/src/SUMMARY.md`: registers the new chapter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: targeted coverage for ProvableSumTree code paths
Adds focused unit tests for production code added by this PR that the
existing test suite happened to exercise indirectly. None of these tests
add new behavior; they only pin down branches that codecov flagged as
uncovered on the patch.
Covers:
- grovedb-query/src/proofs/encoding.rs: long-value (>= 65536 bytes)
round-trips for the four KV-style ProvableSumTree wire variants
(KVSum, KVRefValueHashSum in both Push and PushInverted directions
-- tag bytes 0x31, 0x34, 0x38, 0x3b).
- grovedb-query/src/proofs/mod.rs: Display tests for KVSum, KVHashSum,
KVRefValueHashSum, KVDigestSum, and HashWithSum proof nodes.
- grovedb-element/src/element_type.rs: proof_node_type dispatch on
ProvableSumTree parents (Items -> KvSum, References -> KvRefValueHashSum),
plus as_str / Display for ProvableSumTree, NonCountedProvableSumTree,
NotSummedProvableSumTree.
- grovedb-element/tests/element_constructors_helpers.rs: every
ProvableSumTree constructor + is_provable_sum_tree /
as_provable_sum_tree_value / into_provable_sum_tree_value helpers,
including the wrong-element error paths.
- grovedb-element/tests/element_display_and_serialization.rs: extends
the all-variants Display test to include ProvableSumTree.
- merk/src/proofs/tree.rs: forged-sum sensitivity for KVDigestSum and
KVRefValueHashSum (the latter exercises the full
combine(referenced_value_hash, node_value_hash) -> node_hash_with_sum
path), aggregate_data() returning ProvableSum for KVSum / HashWithSum,
and key() returning the right thing for all five sum node variants.
- merk/src/proofs/query/aggregate_sum.rs: two new regression tests for
Merk::prove (regular query) on a ProvableSumTree, asserting that the
emitted proof contains KVSum + KVHashSum (and KVDigestSum for an
absent-key boundary). These hit the to_kv_sum_node / to_kvhash_sum_node
/ to_kvdigest_sum_node helpers whose only callers are inside
create_proof_internal's ProvableSumTree branches.
All 2956 workspace tests pass (was 2938 before this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(verify): reject KVRefValueHashSum in trunk/branch chunk proofs
The trunk/branch proof extractor was rejecting Node::KVRefValueHash and
Node::KVRefValueHashCount as having an opaque value_hash, but the new
Phase 2 KVRefValueHashSum variant was missing from the rejection arm.
Without this guard, get_key_value_from_node still surfaces (key, value)
for KVRefValueHashSum nodes and the verifier would deserialize and
insert the value bytes into the elements map. The embedded
node_value_hash is opaque (combine_hash of the node_value_hash and the
referenced_value_hash) and cannot be recomputed from the value bytes
alone, so a forged value could ride along while the per-node merk
hash chain still appears valid.
Add KVRefValueHashSum to the rejection arm in
extract_elements_and_leaf_keys, alongside KVRefValueHash and
KVRefValueHashCount, with a regression test that mirrors the existing
KVRefValueHash trunk-proof rejection test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(query): scan conditional-branch selectors for AggregateSumOnRange
has_aggregate_sum_on_range_anywhere previously walked only
branch.subquery for each conditional branch, ignoring the branch
selector (the IndexMap key). Selectors are themselves QueryItems and
the type system permits an AggregateSumOnRange tag there even though
it is not a meaningful conditional matcher. The shape check is meant
to be exhaustive — if any ASOR is present "anywhere", the prover must
refuse to route through the regular-proof path — so the walker must
surface a selector-tagged ASOR too.
Iterate `(selector, branch)` instead of `branches.values()` and
short-circuit on `selector.is_aggregate_sum_on_range()` before
recursing into `branch.subquery`. Add a regression test that mirrors
the existing count walker test and explicitly covers the selector
case.
The same gap exists for has_aggregate_count_on_range_anywhere but is
left as-is here since it predates this PR and changing it would be
out of scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: error classification and entry preservation in aggregate-sum paths
Three small correctness improvements flagged by review:
* grovedb/src/lib.rs verify_merk_and_submerks_in_transaction: when both
the cryptographic combined_value_hash check and the
aggregate_consistency check fail for the same subtree path, the
aggregate-consistency branch's `issues.insert(...)` clobbered the
cryptographic mismatch entry. The real merk-hash chain mismatch is
the more diagnostic message, so switch to `.entry().or_insert(...)`
to preserve the first-inserted entry per path.
* merk/src/proofs/query/aggregate_sum.rs: the prover walks our *own*
in-memory merk. If `aggregate_data()` refuses to surface a
`ProvableSum` for a node in a tree we already gated as
`ProvableSumTree`, that is local storage/state corruption, not a
peer-supplied invalid proof. Reclassify three sites from
`Error::InvalidProofError` to `Error::CorruptedData` to match the
repo error-handling convention.
* grovedb/src/operations/proof/generate.rs: the two ASOR call sites
forwarded bare `Error::MerkError` for `prove_aggregate_sum_on_range`
failures, making them indistinguishable from surrounding
proof-generation merk errors. Wrap each with
`Error::CorruptedData(format!("prove_aggregate_sum_on_range failed: {}", e))`
per the repo convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: strengthen aggregate-sum regression coverage
Address review feedback on test gaps in the ProvableSumTree /
AggregateSumOnRange suite:
* provable_sum_tree_tests::populated_provable_sum_tree_round_trips:
the per-iteration loop fetched the parent psum but only asserted its
variant — the running aggregate after each insert was never checked.
Track an `expected_sum` accumulator and assert
`as_provable_sum_tree_value() == expected_sum` after every insert so
the running aggregate (7 → 20 → 40) is pinned down, not just the
final sum.
* direct_insert_provable_sum_tree_with_root_key_and_sum: the test was
named for a direct-insert path but never actually performed one — it
built a populated template tree and inspected its on-disk shape,
then stopped. Capture the template's root_key + sum, then run a
batch `insert_only_known_to_not_already_exist_op` over a
ProvableSumTree element carrying those values at a fresh top-level
key (the non-batch insert path forbids non-empty Tree elements with
"a tree should be empty at the moment of insertion when not using
batches", so the documented direct-insert semantics are only
reachable via the batch API). Assert the round-tripped element
preserves the captured root_key + sum.
* element/tree_type.rs get_feature_type_zeros_sum_for_not_summed_in_sum_parents:
extend the NotSummed wrapper regression to include
`TreeType::ProvableSumTree` so the new Phase 2 sum-bearing parent
variant's zero-sum semantics stay pinned alongside SumTree,
BigSumTree, CountSumTree, and ProvableCountSumTree.
* tree/link.rs round_trip_aggregate_data_provable_sum_negative: pin
down the new wire tag 7 introduced for
`AggregateData::ProvableSum`. Encode a negative-sum reference link
(also exercises the signed-i64 varint path), assert tag 7 is
present in the bytes, then decode and assert the link's fields
round-trip identically.
* aggregate_sum_query_tests::aggregate_sum_with_subquery_is_rejected_at_validation:
previously only fed a dummy proof to the verifier-side validator,
leaving the new prover-side gate
(`prove_query_non_serialized` short-circuit) unasserted. Build the
malformed PathQuery, set up the standard 15-key fixture, and assert
`db.prove_query(...)` returns `Err` so a regression in the prover
gate can't slip through with this test still passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: refresh ProvableSumTree doc comments
The TreeType::ProvableSumTree doc said "Phase 1: behaves identically
to SumTree everywhere except in inner_node_type /
empty_tree_feature_type. Phase 2 will diverge the hash computation."
Phase 2 has shipped — the hash dispatch now goes through
node_hash_with_sum and the new proof-node families (KVSum, KVHashSum,
KVDigestSum, KVRefValueHashSum, HashWithSum) plus the
AggregateSumOnRange query are all in place. Rewrite the comment to
describe the current (post-Phase-2) semantics: an i64 sum baked into
every node's hash, making sum tampering catchable via proof
verification, as the sum-side counterpart of ProvableCountTree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(verify): require provable tree type at aggregate query terminal layer
Security finding (Codex): the `verify_aggregate_sum_query` and
`verify_aggregate_count_query` chain walkers only checked
`element.is_any_tree()` for path elements. At the terminal (leaf) layer
this is insufficient — if the honest tree at the queried path happens
to be an EMPTY Merk-backed tree of any type (NormalTree, SumTree,
BigSumTree, CountTree, CountSumTree, ProvableCountTree,
ProvableCountSumTree, ProvableSumTree), its stored
`value_hash = combine_hash(H(element_bytes), NULL_HASH)`. The merk
verifier accepts empty proof bytes as `(NULL_HASH, 0)`, so an attacker
can construct a forged proof with:
- layer 0: honest single-key proof of the leaf path key in its parent
- layer 1: empty bytes (forged)
and the chain check passes uniformly. The verifier returns `sum = 0`
(or `count = 0`) against the trusted root hash, even though the leaf
isn't a Provable{Sum,Count}Tree.
The numeric answer is correct (an empty tree has sum 0 / count 0), so
this isn't a value forgery — but it IS a type-confusion soundness gap:
a caller that infers "leaf is a ProvableSumTree" from "the aggregate
verifier accepted" is deceived. The prover-side gate in
`Merk::prove_aggregate_{sum,count}_on_range` already rejects
non-provable inputs, but the verifier didn't mirror that invariant.
THE FIX
In `enforce_lower_chain`, add an `is_terminal: bool` parameter. At
intermediate depths nothing changes (`is_any_tree()` still suffices —
the GroveDB grove can route through any tree type on the way down). At
the terminal depth — passed `is_terminal = true` when
`depth + 1 == path_keys.len()` — the verifier now requires:
- aggregate-sum: `matches!(element, Element::ProvableSumTree(..))`
- aggregate-count: `matches!(element, Element::ProvableCountTree(..) |
Element::ProvableCountSumTree(..))`
Wrapper variants (NonCounted, NotSummed) are stripped via the existing
`into_underlying()` so they continue to work transparently.
TESTS
Three new regression tests that surgically construct the forgery from a
real honest single-key envelope and confirm the verifier now rejects:
- `empty_leaf_type_confusion_forgery_rejected` (sum side, empty
NormalTree at leaf)
- `empty_provable_count_tree_at_leaf_rejected_for_sum` (sum side,
empty ProvableCountTree at leaf — confirms type-specificity)
- `empty_leaf_type_confusion_forgery_rejected` (count side, empty
NormalTree at leaf)
The path == 0 case is unaffected: the merk-level hash divergence
between `node_hash` and `node_hash_with_sum` / `node_hash_with_count`
makes it computationally infeasible to forge a proof that matches the
trusted root, so the path-elements check is unnecessary at the root.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(verify): reject empty-path aggregate-sum/count queries at validation
Codex follow-up + CodeRabbit: the previous fix added a terminal-type
gate in `enforce_lower_chain`, but `verify_v0_layer` and
`verify_v1_layer` short-circuit to the leaf verifier when
`depth == path_keys.len()`. With an empty path (`path == []`) that's
true at depth 0, so the type gate is never invoked.
In practice the empty-path case is already protected by hash divergence:
the GroveDB root merk is always a `NormalTree` (built with
`Element::empty_tree()` by API), so its root_hash uses `node_hash`. An
attacker's forged proof of `HashWithSum` / `HashWithCount` ops would
reconstruct via `node_hash_with_sum` / `node_hash_with_count` — distinct
hash functions, no collision. So the caller's root-hash compare catches
the forgery cryptographically.
But the defense-in-depth principle says: don't rely on the cryptographic
divergence implicitly. Reject up-front, before any proof handling.
PathQuery::validate_aggregate_{sum,count}_on_range now check
`self.path.is_empty()` and return a clear InvalidQuery error naming why
(root is always NormalTree, no valid Provable* target at root).
The check fires at the entry of `verify_aggregate_{sum,count}_query`
(which call `validate_*` first thing) and at `prove_query` (the
generator also validates the path query before dispatch).
TESTS
- `empty_path_aggregate_sum_rejected_at_validation`
- `empty_path_aggregate_count_rejected_at_validation`
Both pin the rejection at both the PathQuery validator and the verify
entrypoint. 2964 workspace tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(merk,grovedb): add no-proof query_aggregate_sum entry point
Mirrors PR #662's `query_aggregate_count` for the signed-sum side.
Callers that need a sum value but not a proof (e.g. server handlers
answering `prove=false` sum requests) can now bypass proof
construction, serialization, and verification entirely.
The merk-level walk is `O(log n + |boundary|)` in the number of
distinct keys, identical complexity to the prover but without the
proof-op allocations or hash recomputations. The signed-sum
arithmetic carries the same `i128` accumulator the prover and
verifier use (so adversarial intermediate sums never wrap), and
narrows to `i64` at the public entry point. An out-of-i64 result is
classified as `Error::CorruptedData` since a real `ProvableSumTree`
maintains every aggregate as `i64` at every level.
NEW APIS
- `Merk::sum_aggregate_on_range(&inner_range, grove_version)
-> CostResult<i64, Error>` in `merk/src/merk/get.rs`. Checks
`tree_type == ProvableSumTree`; rejects any other tree type with
`Error::InvalidProofError`. Returns 0 for an empty merk.
- `RefWalker::sum_aggregate_on_range(&inner_range, grove_version)`
in `merk/src/proofs/query/aggregate_sum.rs`. Walks the same
Contained / Disjoint / Boundary classification path as
`create_aggregate_sum_on_range_proof`, but emits no proof ops.
- `GroveDb::query_aggregate_sum(path_query, transaction, grove_version)
-> CostResult<i64, Error>` in `grovedb/src/operations/get/query.rs`.
Validates the PathQuery up-front via
`validate_aggregate_sum_on_range` (same gate the prover and
verifier use — catches malformed ASOR queries plus the
empty-path rejection from the prior commit before any storage
reads), opens the leaf merk at `path_query.path`, and delegates
to the merk-level walk.
- New `query_aggregate_sum_on_range` field on
`GroveDBOperationsQueryVersions`, wired through v1/v2/v3 at
version `0`.
NotSummed-correctness is preserved via the same
`own_sum = node_sum - left_struct - right_struct` derivation the
prover uses. NotSummed-wrapped subtrees have stored aggregate 0, so
the subtraction yields 0 at the wrapper boundary - they do not
contribute to the in-range total.
The returned sum is **not** independently verifiable: callers are
trusting their own merk read path. For a verifiable sum, continue
using `prove_query` + `verify_aggregate_sum_query`. Documented
explicitly on both entry points.
TESTS
- 10 new merk-level cross-checks
(`merk/src/proofs/query/aggregate_sum.rs::tests`): each range
variant against `prove_aggregate_sum_on_range`'s computed sum,
plus empty-merk-returns-0, NormalTree rejection, ProvableCountTree
rejection (precise tree-type match, not "any provable aggregate
tree"), and a mixed-positive/negative scenario that exercises the
signed `own_sum` subtraction.
- 11 new GroveDB-level cross-checks
(`grovedb/src/tests/aggregate_sum_query_tests.rs::tests`): every
range shape on a populated `ProvableSumTree`, empty subtree
returns 0, negative-sum scenario, invalid-inner-range
(`Key`) rejected with `InvalidQuery`, empty-path rejected with
`InvalidQuery`, NormalTree leaf rejected with `MerkError` from
the merk-level gate.
Workspace `cargo test --all-features`: 2985 passing / 0 failing
(was 2964 / 0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(query_item): mirror AggregateCountOnRange tests for AggregateSumOnRange
Adds parallel coverage for the variant-11 dispatch paths in encode/
decode_with_depth/borrow_decode_with_depth/Display/serde, plus helper
accessors (lower_bound, upper_bound, is_aggregate_sum_on_range,
aggregate_sum_inner, ...). Mirrors the existing AggregateCountOnRange
test set so each match arm has direct exercise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(element/helpers): per-variant flag-accessor coverage
Adds direct round-trip tests of get_flags_owned / get_flags_mut /
set_flags on every aggregate-bearing variant (CountSumTree,
ProvableCountTree, ProvableCountSumTree, ProvableSumTree,
ItemWithSumItem, CommitmentTree, MmrTree, BulkAppendTree,
DenseAppendOnlyFixedSizeTree, BigSumTree, CountTree) plus
NotSummed / NonCounted delegation arms. Each test pins one match arm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(grovedb): unit coverage for aggregate_consistency_labels
Direct unit tests for the previously untestable internal helper.
Each aggregate-bearing tree variant now has both a matching (returns
None) and mismatching (returns Some) case, plus tests for the
empty-merk identity arms (zero-recorded with NoAggregateData),
non-Merk data tree arms (always None), and the catch-all
variant/shape mismatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(grovedb/proof): verifier error-path coverage for aggregate-sum
Adds 10 mutation-style tests for the GroveDB-side aggregate-sum
verifier (verify_v0_layer / verify_v1_layer / verify_sum_leaf /
verify_single_key_layer_proof_v0 / enforce_lower_chain). Each test
pins one previously-uncovered error arm:
- V1 unexpected non-merk leaf bytes
- V0 and V1 missing lower_layer for path key
- Malformed leaf sum proof (Phase 1 rejection)
- Corrupted non-leaf merk bytes (single-key proof failure)
- Non-leaf proof without target key
- KV replaced by KVDigest in non-leaf (no value bytes)
- Undeserializable value bytes on the path
- Intermediate-tree non-tree element rejection
- Unparsable envelope bincode decode error
Plus mirrors of the count-side AggregateSumOnRange rejection tests
in proof/generate.rs for dense/MMR/BulkAppend index helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(merk/aggregate_sum): unit coverage for helpers and edge cases
Direct unit tests for previously-uncovered internal helpers in
merk/src/proofs/query/aggregate_sum.rs:
- provable_sum_from_aggregate Err arm for every non-ProvableSum
AggregateData variant (CorruptedData classification check)
- provable_sum_from_aggregate happy path including i64::MIN/MAX
- is_provable_sum_bearing false for every non-ProvableSumTree
TreeType variant
- classify_subtree additional disjoint-above / contained-within /
boundary-overlapping-upper cases
- key_strictly_inside unbounded endpoint and equality cases
- empty ProvableSumTree prove+verify round-trip
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(non-merk trees): aggregate-sum/count rejection in index helpers
Adds parallel-variant rejection tests in the BulkAppendTree and
dense fixed-size Merkle tree proof modules. Both tree types have
no count or sum commitment in their node hash, so their
index-resolution helpers reject AggregateCountOnRange and
AggregateSumOnRange query items outright. This exercises the
previously-uncovered rejection arms in both proof modules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: per-variant tree_type extensions + sum-proof Display arms
Two small targeted coverage additions:
- merk/src/element/tree_type.rs: direct per-variant tests for the
ProvableSumTree / CommitmentTree / BulkAppendTree /
DenseAppendOnlyFixedSizeTree / MmrTree arms of
root_key_and_tree_type, tree_flags_and_type, tree_type,
maybe_tree_type, and tree_feature_type, plus a
ProvableSumTree-through-NotSummed delegation test.
- grovedb/src/tests/aggregate_sum_query_tests.rs: tests that
drive node_to_string's KVSum / KVHashSum / KVDigestSum /
HashWithSum / KVRefValueHashSum Display arms in
grovedb/src/operations/proof/mod.rs by formatting real
ProvableSumTree proofs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* nit(coderabbit): doc-comment + test + error-wrap polish
Four low-value but clean tweaks from CodeRabbit on PR #661:
- `grovedb-query/src/query_item/mod.rs`: refresh the stale
`NonAggregateInner::deserialize` inline comment to mention both
excluded aggregate variants (Count + Sum), matching the struct-level
doc and `NON_AGGREGATE_VARIANTS`.
- `grovedb/src/tests/aggregate_sum_query_tests.rs`: drop the redundant
disjunction `msg.contains("must be a ProvableSumTree") ||
msg.contains("ProvableSumTree")` — the first clause already implies
the second; pin the exact phrase.
- `grovedb/src/tests/aggregate_sum_query_tests.rs`: harden
`provable_sum_tree_overflow_at_i64_max_is_rejected` so it no longer
silently passes when insert AND prove AND verify all accept an
overflow. Replace the early-return-on-both-inserts-accepted with an
explicit "at least one stage must reject" assertion.
- `grovedb/src/operations/get/query.rs`: wrap the MerkError from
`Merk::sum_aggregate_on_range` (and the count sibling) with
contextual `CorruptedData(format!("query_aggregate_{sum,count} at
path {:?}: {}", path_slices, e))` per the repo
error-wrapping convention. Two test assertions updated from
`MerkError(_)` to `CorruptedData(_)` to match.
Workspace `cargo test --all-features`: 3102 pass / 0 fail (unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: clarify appendix-a Element/TreeType discriminant columns
CodeRabbit flagged the TreeType column for ProvableSumTree as "should be
17" — the technical claim was wrong (11 is the correct TreeType
discriminant per `merk/src/tree_type/mod.rs:77`), but the confusion was
fair: the column header "TreeType" was ambiguous and the table had
several pre-existing inaccuracies in adjacent rows. This commit fixes
the ambiguity AND the bugs.
Changes:
- Rename column header from "TreeType" to "TreeType disc" and add an
intro paragraph explaining that "Element disc" and "TreeType disc"
are discriminants of two SEPARATE enums.
- Add the TreeType-variant label to every tree row for consistency
(some had it, most didn't). The new format is `N (VariantName)` —
e.g. `5 (ProvableCountTree)` — which CodeRabbit-style auto-review
can't misread.
- Fix three pre-existing wrong TreeType disc values:
`BigSumTree`: 4 -> 2
`CountTree`: 2 -> 3
`CountSumTree`: 3 -> 4
(These were drift from the actual `TreeType::discriminant()`
implementation; the file had `4 (BigSumTree)` etc. but those labels
were wrong.)
- Swap the row order at Element discriminants 8 and 9 to match the
actual `Element` enum order:
8 = `ProvableCountTree` (was incorrectly listed as `ItemWithSumItem`)
9 = `ItemWithSumItem` (was incorrectly listed as `ProvableCountTree`)
- Tighten the `ProvableCountSumTree` Purpose blurb to note "only count
in hash" since the sum is tracked metadata, not bound — this is the
half-step variant a future `ProvableCountAndSumTree` would replace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(verify): accept Sum-family boundary nodes in range bound checks
Codex security finding: the regular query verifier's lower- and
upper-bound `last_push` match arms in
`merk/src/proofs/query/verify.rs::execute_proof` (lines 206 / 241)
accept the Count-family boundary node variants (`KVCount`,
`KVDigestCount`, `KVRefValueHashCount`) but omit the parallel Sum
family (`KVSum`, `KVDigestSum`, `KVRefValueHashSum`). Regular proofs
against a `ProvableSumTree` can legitimately emit `KVDigestSum` as the
absence-boundary node for a queried key, so a multi-item query like
`Key("aa")` followed by `Range("g".."j")` would reject the perfectly
valid proof with `Cannot verify lower bound of queried range` whenever
the preceding boundary happened to be sum-flavored.
The downstream absence check at line ~572 already handled all six
node types (Count + Sum), making the omission an asymmetry between
the two checks within the same function.
THE FIX
Add `KVSum`, `KVDigestSum`, `KVRefValueHashSum` to both the lower- and
upper-bound `last_push` match arms. While at it, also extend
`boundaries_in_proof` (line ~742) to surface `KVDigestSum` boundary
keys alongside `KVDigest` and `KVDigestCount` — same class of
omission, same trivial extension.
TESTS
New `provable_sum_tree_bound_regression_tests` module at the bottom of
`verify.rs` covering:
- `key_plus_range_on_provable_sum_tree_left_to_right_verifies` — the
exact `[Key("aa"), Range("g".."j")]` shape Codex flagged, in
forward iteration. Without the fix this returns
`InvalidProofError("Cannot verify lower bound of queried range")`.
- `key_plus_range_on_provable_sum_tree_right_to_left_verifies` —
same query with `left_to_right = false`. The bug is symmetric, so
the regression coverage is too.
- `kv_digest_sum_appears_in_boundaries_in_proof` — proves that
`boundaries_in_proof` now surfaces `KVDigestSum`-flavor boundary
keys produced by `ProvableSumTree` proofs.
Workspace `cargo test --all-features`: 3150 pass / 0 fail (was 3147 /
0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(query): split aggregate validator error label per count/sum variant
Develop landed a regression test
(`query_validation_error_to_static_str_projects_invalid_operation_and_catches_other_variants`,
commit 7a649386) that pins the catch-all fallback string returned by
`query_validation_error_to_static_str` to
`"AggregateCountOnRange query validation failed"`. This PR had
generalised the helper to serve both count and sum, returning
`"aggregate query validation failed"`, which broke the develop test
under GitHub's "merge into base" CI workflow.
Split the helper into two so each aggregate variant's error surface
stays self-describing:
- `query_validation_error_to_static_str` — count side, restored to
the `"AggregateCountOnRange query validation failed"` label so
develop's regression test stays green.
- `sum_query_validation_error_to_static_str` — new sum-side helper
returning `"AggregateSumOnRange query validation failed"`. Used by
`SizedQuery::validate_aggregate_sum_on_range`.
Both follow the same projection contract: `InvalidOperation(msg)`
passes the static string through unchanged; any other variant
(unreachable from real validators) gets the variant-specific
fallback.
No behavior change at the InvalidOperation happy path, which is all
real callers reach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(verify): key_exists_as_boundary_in_proof must accept KVDigestSum
CodeRabbit symmetry finding on top of commit 5b3a0460: that fix
extended `boundaries_in_proof` to recognize `KVDigestSum` boundary
nodes from `ProvableSumTree` proofs, but missed the parallel helper
`key_exists_as_boundary_in_proof`. The two public helpers are
documented to behave identically; without this both helpers
disagreed on valid `ProvableSumTree` absence proofs.
Add `Op::Push(Node::KVDigestSum(..))` and the PushInverted variant
to the match in `key_exists_as_boundary_in_proof`. Tighten the
doc-comment to spell out that the two helpers share node-type
coverage.
Extended the regression test
`kv_digest_sum_appears_in_boundaries_in_proof` (now renamed to
`kv_digest_sum_appears_in_both_boundary_helpers`) so every boundary
key surfaced by `boundaries_in_proof` is also reported by
`key_exists_as_boundary_in_proof`, pinning the symmetry.
Workspace `cargo test --all-features` for the affected module: 3 of
3 regression tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(comments): strip implementation-phase labels from ProvableSumTree code
Remove "Phase 1 / Phase 2 / etc." prefixes that referred to the PR's
implementation timeline. Retains "Phase 1 / Phase 2" labels that describe
runtime decode-vs-walk algorithm steps of the aggregate-count/sum verifiers
(documented in docs/book/src/aggregate-count-queries.md). In the
test-fixture stack-builder (merk/src/proofs/tree.rs) and the
provable_sum_tree direct-insert test, renamed enumeration-style "Phase N"
labels to "Step N" for clarity. Also renamed the phase2_* test fn prefixes
in encoding.rs and tree.rs to drop the timeline label.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(serde): QueryItem Serialize emits snake_case variant tags
CodeRabbit re-flagged a pre-existing asymmetry in `QueryItem`'s manual
serde implementations: the `Serialize` impl emitted PascalCase variant
tags via `serialize_newtype_variant` (e.g. `"AggregateSumOnRange"`,
`"Range"`), but the `Deserialize` impl uses `Field` enums marked
`#[serde(field_identifier, rename_all = "snake_case")]`, expecting
snake_case (`"aggregate_sum_on_range"`, `"range"`).
The asymmetry was invisible to bincode (the format GroveDB actually
uses in proofs and storage) because bincode identifies variants by
index, not by the textual tag. But it broke round-trip through every
text-based format that carries variant names verbatim (JSON, YAML,
TOML). An in-code comment at the existing token-stream test site
even documented the issue as "a pre-existing mismatch ... that
breaks JSON round-trip but is invisible to formats that don't carry
variant names textually."
THE FIX
Change every `serialize_newtype_variant` (and the one
`serialize_unit_variant` for `RangeFull`) call in
`grovedb-query/src/query_item/mod.rs` to emit snake_case variant
tags. The variant indices stay the same so the bincode wire format
is unchanged — only textual formats see the new tag names.
Affected variants: `Key`, `Range`, `RangeInclusive`, `RangeFull`,
`RangeFrom`, `RangeTo`, `RangeToInclusive`, `RangeAfter`,
`RangeAfterTo`, `RangeAfterToInclusive`, `AggregateCountOnRange`,
`AggregateSumOnRange` — i.e. every variant, not just the aggregate
ones. This is the symmetric fix; doing only the new
`AggregateSumOnRange` variant would have diverged it from the
existing `AggregateCountOnRange` (and from the other ten variants
that have always been broken the same way).
Also updated the in-code comment at the token-stream test site to
reflect the new contract.
TESTS
Two new `serde_test::assert_tokens` round-trip regression tests pin
the snake_case contract on both aggregate variants:
- serde_round_trip_aggregate_sum_on_range_uses_snake_case_tag
- serde_round_trip_aggregate_count_on_range_uses_snake_case_tag
assert_tokens exercises both Serialize AND Deserialize against the
same token stream, so any future regression on either side fails the
test immediately.
Workspace cargo test --all-features: 3152 pass / 0 fail (was 3150 / 0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(aggregate-sum): mirror PR #663 — split into subdir, reject V0 envelopes
V0 (`MerkOnlyLayerProof`) envelopes predate the aggregate-sum feature and
cannot legitimately carry an aggregate-sum proof, so the V0 layer walker
was unreachable in any honestly-produced proof. Mirror the count-side
changes from PR #663:
- Convert `aggregate_sum.rs` into a subdirectory mirroring
`aggregate_count/`: `mod.rs` (public API + V0 rejection),
`helpers.rs` (envelope decode, single-key layer verification,
chain enforcement, leaf sum verification), `leaf_chain.rs` (V1
leaf-chain walker). Removes the dead `verify_v0_layer` path.
- Add a prover-side V0 gate in `prove_query_non_serialized`: when
grove_version dispatches to V0 and the path query carries an
`AggregateSumOnRange` anywhere, return `NotSupported` instead of
emitting a V0 envelope the verifier would (correctly) reject.
- Update tests: replace the V0 round-trip test with a V0-rejection
test; broaden the empty-leaf type-confusion test to accept either
the V0-rejection or the terminal-type-gate error; remove the now-
unreachable V0 missing-lower-layer test (V1 counterpart already
pins the missing-layer behavior); refresh stale doc-comments that
pointed at `aggregate_sum.rs` line numbers.
No carrier-shape support yet — `AggregateSumOnRange` is still leaf-only
on the merk side, so there is no `classification.rs` / `per_key.rs`
mirror. The leaf-only shape can be extended later in parallel with
matching merk-level work, just like the count side did.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* harden(aggregate-sum): strict lower_layers shape + V1 terminal-type test
Addresses CodeRabbit findings on PR #661.
**Finding 1 (Major, fixed)** — `verify_v1_leaf_chain` accepted arbitrary
`lower_layers` entries at non-leaf depths and under the leaf merk. That
let two byte-distinct envelopes verify to the same `(root, sum)` because
the smuggled siblings/children were never inspected, and gave downstream
syntactic scanners an unverified surface to read from. Added a strict
shape gate that mirrors the honest prover (`prove_aggregate_sum_on_range`
short-circuit always emits `lower_layers: BTreeMap::new()` at the leaf,
and each path-prefix wrapper inserts exactly one entry):
- At `depth == path_keys.len()` (leaf merk): require
`lower_layers.is_empty()`.
- At non-leaf depths: require `lower_layers.len() == 1` and the sole
key to equal the expected descent key `path_keys[depth]`.
Two new regression tests (`sum_v1_envelope_with_extra_lower_layer_*`
and `sum_v1_envelope_with_lower_layers_under_leaf_*`) construct
byte-modified envelopes that the gate rejects.
**Finding 4 (nitpick, addressed)** — the existing empty-leaf
type-confusion tests build V0 envelopes and now hit the V0-rejection
gate before the terminal-type gate in `enforce_lower_chain` runs.
Added `empty_leaf_type_confusion_forgery_rejected_under_v1_envelope`
which builds the same forgery under a V1 `LayerProof` envelope so the
terminal-type gate fires directly. The test asserts the specific
"must be a ProvableSumTree" error from `enforce_lower_chain` so
future refactors that drop the gate are caught.
**Finding 2 (Major, skipped with reason)** — CodeRabbit asks to make
`hash_for_link` fail-closed (panic/Err) when a `ProvableSumTree` node's
`aggregate_data()` doesn't return `ProvableSum`. The current fallback
to `self.hash()` is identical across all three `Provable*` variants
(`ProvableCountTree`, `ProvableCountSumTree`, `ProvableSumTree`) and
also appears in commit-time dispatch (lines 1233-1304). Fixing only the
sum arm creates asymmetry with the count side; the broader refactor
(plus the matching dispatch-centralization in Finding 3) is out of
scope for this sum-feature PR and is documented in MEMORY M1 as
intentional.
**Finding 3 (nitpick, skipped with reason)** — the duplicated
`AggregateData → hash` dispatch in `merk/src/tree/mod.rs` predates this
PR and applies to all three `Provable*` variants. Centralizing it
would touch hot proof-emission paths; out of scope here.
Also updated `sum_v1_envelope_with_missing_lower_layer_is_rejected`
to accept the new strict-shape error message — removing an entry now
trips the shape gate first instead of the older missing-layer arm.
Both messages pin the same property.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(aggregate-sum): split strict-shape gate + cover trailing-bytes / wrong-key
After splitting the strict-shape gate in \`verify_v1_leaf_chain\` into
two reachable error arms (entry-count vs. entry-key), the
ok_or_else-arm on the subsequent \`lower_layers.get(&next_key)\` became
naturally reachable for the wrong-key case (single entry but under
an unexpected key). Previously the combined-OR gate made both error
sites mutually exclusive, leaving the get-arm dead.
Coverage impact (aggregate_sum/ subdir, local tarpaulin):
- leaf_chain.rs: 36/41 (88%) -> 44/44 (100%)
- Subdir total: ~80% -> 89.94%
New tests in aggregate_sum_query_tests:
- sum_v1_envelope_with_wrong_keyed_lower_layer_is_rejected — single
\`lower_layers\` entry under \"impostor\" instead of \"st\". Exercises
the key-shape gate distinctly from the count-shape gate.
- sum_proof_with_trailing_bytes_is_rejected — mirror of
\`aggregate_count_proof_with_trailing_bytes_is_rejected\`. Pins the
canonical-decode invariant in \`decode_grovedb_proof\`.
Tightened assertion in
\`sum_v1_envelope_with_extra_lower_layer_is_rejected\` to match the new
\"lower-layer entries at depth N\" error string, and broadened the
assertion in \`sum_v1_envelope_with_missing_lower_layer_is_rejected\`
to accept either the old missing-layer message or the new
entry-count message (removing the only entry trips the count gate
first).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(proof): hoist canonical proof decoder to operations/proof/mod.rs
The same `decode_grovedb_proof` function lived in both
`aggregate_count/helpers.rs` and `aggregate_sum/helpers.rs`, differing
only in whether the error string named "aggregate-count" or
"aggregate-sum" as the offending shape. Two copies of the same
canonical-decode contract is a maintenance hazard — drift between the
two would mean one aggregate path could (e.g.) accept trailing bytes
that the other rejects, breaking the equality-by-bytes assumption the
contract is meant to guarantee.
Hoist the function to `operations/proof/mod.rs` as
`decode_grovedb_proof_canonical` with `pub(super)` visibility. Both
helper modules now call `super::decode_grovedb_proof_canonical`. The
error message generalizes from "aggregate-{count,sum} proof has N
trailing bytes" to "proof has N trailing bytes" since the call site
provides the surrounding context; the existing
`*_proof_with_trailing_bytes_is_rejected` tests assert
`msg.contains("trailing bytes")` and remain green.
No behavior change beyond the wording adjustment in the trailing-
bytes error. Tests: workspace 3088 / 0 fail; aggregate tests
223 / 0 fail (113 sum + 75 count + 35 others).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* harden(proof): general verify paths now use canonical decoder
Migrate the four `bincode::decode_from_slice(...)?.0` sites in
`operations/proof/verify.rs` (`verify_query_with_options`,
`verify_query_get_parent_tree_info_with_options`, `verify_query_raw`,
`verify_trunk_chunk_proof`) to call
`super::decode_grovedb_proof_canonical(proof)?` — the same canonical
decoder the aggregate-count and aggregate-sum entry points already use.
Before this change, the general verifier silently accepted trailing
bytes after the encoded `GroveDBProof` envelope, because the `.0`
discarded the bincode `consumed` count. The aggregate-count and
aggregate-sum entry points rejected trailing bytes via their own
private decoder. That asymmetry meant the same logical proof could
have many distinct byte encodings through the general-verify surface
but only one through aggregate-verify — a malleability gap that
breaks any equality-by-bytes / caching / dedup assumption a consumer
might rely on. The cryptographic chain still bound the answer, so
this wasn't a soundness break.
The general-verify path now matches: trailing bytes are rejected with
`Error::CorruptedData("proof has N trailing bytes after the encoded
envelope")`.
Added `verify_query_rejects_proof_with_trailing_bytes` in
`proof_advanced_tests.rs` to lock the new behavior in place, mirror
of `sum_proof_with_trailing_bytes_is_rejected` and
`aggregate_count_proof_with_trailing_bytes_is_rejected`.
Tests: workspace 3089 / 0 fail (3088 + 1 new regression test).
No honest test/proof emitted trailing bytes, so no existing test
needed updating.
Co-Authored-…
QuantumExplorer
added a commit
that referenced
this pull request
May 18, 2026
Removed references that read as stale dev-process noise to a reader encountering the code months later, with no information loss about the code's current behavior or rationale: **PR # references (8 sites)** - generate.rs (combined-aggregate gate): "PR #670 / grove v3+ feature" → "grove v3+ feature". - count_offset_paginated_tests.rs (assert message + 2 docs): "PR #672 closes the P1 finding" / "the P1 finding's root cause" → describe the actual rule (NonCounted inserts into ProvableCountTree rejected). - provable_count_provable_sum_tree_tests.rs (4 docs): "PR #672 for ProvableCountTree…" → "Same rejection rule that applies to…". - aggregate_count_query_tests.rs (forgery test branches): drop "added in PR #663" / "added in this PR" qualifiers from documentation of two error-branches. - not_counted_or_summed_tests.rs: drop "PR #666's contract" qualifier. - reference_with_sum_item_tests.rs (2 sites): drop "added in PR #667" / "added in this PR" / "PR #667 already covers" — rephrase as factual statements. - aggregate_sum_query_tests.rs (test-section header): drop "PR #662's no-proof query_aggregate_count" → "Sum-side mirror of the no-proof query_aggregate_count tests." - query.rs (query_aggregate_sum doc): drop "Mirrors PR #662's". - non_counted_tests.rs (module header): drop "Codex review of PR #654". **CodeRabbit references (3 sites)** - count_offset_paginated_tests.rs (2 sites): drop "(CodeRabbit review on grovedb#669)" parenthetical. - merk/mod.rs (test doc): drop "Tightened (per CodeRabbit review)". - merk/proofs/query/aggregate_count/tests.rs (comment): drop "(per CodeRabbit review)". **Temporal markers — "Before this fix" / "in this PR" framing (4 sites)** - merk/mod.rs (2 docs): "Before this fix the supports_count match…" → "the support check delegates to is_count_bearing(), so any hand-rolled match here would be a drift-risk regression." / "previously omitted from this manual match" → drop the temporal qualifier. - aggregate_sum_carrier_query_tests.rs (test doc): "previously blanket — it blocked legitimate root-carrier queries" → describe current shape-aware behavior. - provable_count_provable_sum_tree_tests.rs (2 sites): "fixed in this PR — without KVRefValueHashCountSum" → "rely on the KVRefValueHashCountSum dispatch arm — without it". "both gained Element::ProvableCountProvableSumTree arms in this PR" → "both carry Element::ProvableCountProvableSumTree arms". **"before this feature" comments (2 sites)** - aggregate_count_query_tests.rs + aggregate_sum_carrier_query_tests.rs (per-key symmetry tests): "same proof bytes it did before this feature" → "same proof bytes whether the caller verifies via verify_aggregate_X_query or the per-key entry point." **"Before this module existed" / "forthcoming" framing (2 sites)** - aggregate_common.rs: "Before this module existed each axis carried its own private copy" → "These items would otherwise be byte-identical copies across each axis. Centralizing them here…" - aggregate_count/mod.rs: "The same leaf/carrier shape will apply to forthcoming aggregate variants (sum, average)" → "The same leaf/carrier shape applies to the sum and combined axes — see the sibling … modules." - grovedb-query/src/aggregate_count.rs: "Forthcoming aggregate variants (sum, average) will live in sibling modules" → "The sum and combined axes live in sibling modules." **Renamed:** `p1_noncounted_in_provable_count_tree_rejected_at_insert` → `noncounted_in_provable_count_tree_rejected_at_insert` to drop the "P1" audit-finding prefix from the test name itself. No behavior changes. All tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AggregateCountOnRange(ACOR) as a subquery item under an outerKeys/Range*carrier query — the natural multi-outer-key extension of the existing single-leg ACOR. Returns one(outer_key, u64)pair per matched outer key while leaving the leaf wire format byte-identical.Query::validate_aggregate_count_on_rangeinto a leaf validator (existing semantics, renamed) and a new carrier validator, with the top-level dispatcher routing by query shape and returning the leaf inner range either way.GroveDb::verify_aggregate_count_query_per_key(...) -> Result<(RootHash, Vec<(Vec<u8>, u64)>), Error>that accepts both leaf and carrier shapes (leaf collapses to a single-entry Vec with an empty key). The legacyverify_aggregate_count_querynow strictly validates the leaf shape.Why
Dash Platform's count-query module needs to answer
against a
byBrand / <brand> / color / <ProvableCountTree>layout. Today grovedb rejects the nested shape atQuery::validate_aggregate_count_on_range, forcing callers to fan out one proof per outer key (k× larger proofs) or fall back to the unproven path. With this change the same GroveDB proof contains all k counts and is verified in one pass.Asymptotic complexity for
|outer matches| = k, outer treeB, leaf treeC:O(k · (log B + log C)).Wire-format compat
aggregate_count_path_query+verify_aggregate_count_queryround-trip works without changes.GroveDBProofenvelope (multi-key outer merk proof + per-outer-keylower_layers) — it cannot be confused for a leaf proof.What's NOT in this PR
dashpay/platform).IN × IN) — the carrier accepts ONE outer fan-out dimension.Test plan
grovedb-queryexercising carrier rules: happy paths, RangeFull/ACOR outer rejection, missing/non-ACOR subquery rejection, conditional-branch rejection, empty-items rejection, Range-outer acceptance, dispatch behavior of the top-level validator.grovedb:acor_subquery_two_outer_keys_succeeds— 3-deep tree, two outer brands, 1 000 colors each, RangeAfter on color leaf; expect 2 entries in lex-asc order with the correct per-brand count.acor_subquery_with_unknown_outer_key_returns_present_keys_only— absent outer key contributes no entry (prover doesn't emit a lower layer for it).acor_subquery_rejects_acor_at_both_levels— validation and prove-entry gate both reject.acor_subquery_carrier_with_range_outer_succeeds— carrier with aRangeAfterouter item.acor_leaf_unchanged_under_per_key_verifier— leaf ACOR returns a one-entry Vec with an empty key and the same countverify_aggregate_count_queryreturns.acor_per_key_rejects_non_acor_path_query— non-ACOR path query rejected up front.acor_subquery_count_forgery_is_caught— bumping aHashWithCountin the leaf bytes through the carrier envelope is rejected by the chain check.cargo test --workspace --features 'minimal verify'— all 1516 lib tests pass.cargo clippy -p grovedb -p grovedb-query --features minimal -- -D warnings— clean.dashpay/platform).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests