feat: add Element::NotSummed wrapper to opt-out of sum propagation - #659
Merged
Conversation
Symmetric counterpart to Element::NonCounted (#654). Where NonCounted opts a child out of count propagation in count-bearing trees, NotSummed opts a sum-bearing subtree out of sum propagation in sum-bearing trees. The wrapper is strictly typed: it can ONLY contain one of the four sum-tree variants (SumTree, BigSumTree, CountSumTree, ProvableCountSumTree), and may only be inserted into sum-bearing parent trees. Items, sum items, references, non-sum trees, and any wrapper nesting are rejected at construction, serialization, and deserialization. When a NotSummed-wrapped sum-tree is inserted into a sum-bearing parent, it contributes 0 to the parent's running sum. Counts (if any) still propagate. The inner sum-tree retains its own internal sum aggregate unchanged — only the outward propagation is suppressed. Discriminant scheme: - bincode 16 (Element variant) - ElementType twins at 0x40 | base: 68, 69, 71, 74 (only the four legal sum-tree base discriminants). The 0x40 flag bit is disjoint from NonCounted's 0x80, so wrapper status is independently testable. Stack-overflow defense: deserialize pre-checks the leading two bytes for any wrapper combination ([15,15], [15,16], [16,15], [16,16]) and rejects before bincode recurses through the Box<Element> chain. Wiring: - grovedb-element: enum + constructor whitelist + serialize/deserialize guards + helpers (sum_value_or_default → 0, count still propagates) + ElementType twins + visualize. - grovedb-query: TreeFeatureType::zero_sum() symmetric helper. - merk: TreeType::is_sum_bearing(), insert/insert_subtree/ insert_reference parent-type guards, reconstruct preserves wrapper, costs/get account for the wrapper byte. - grovedb: batch propagation tracks not_summed alongside non_counted in InsertTreeWithRootHash, parent-type guard, estimated cost overhead, debugger renders inner element transparently. Tests (725+ existing pass, plus new): - grovedb-element: constructor whitelist, helper passthrough, bincode round-trip, deserialize rejects nested chains and non-sum-tree inner. - grovedb-merk: insert rejected in NormalTree/CountTree, accepted in SumTree contributes 0, NotSummed(SumTree(_,100,_)) in ProvableCountSumTree → count=1 sum=0, reconstruct preserves wrapper. - grovedb: 5 end-to-end tests covering batch parent-type guard, batch propagation preserving the wrapper, check_subtree_exists through the wrapper, and outer sum aggregate excluding the subtree. Breaking changes: none for existing callers. The on-disk serialization format gains bincode discriminant 16 — older code reading data written with this variant fails to deserialize, matching the existing "ONLY APPEND TO THIS LIST" comment on Element. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (29)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Member
Author
|
@coderabbitai review This is Claude. The PR is intentionally in draft while we wait for the first review pass — please review it now. |
Contributor
|
✅ Actions performedReview triggered.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #659 +/- ##
===========================================
+ Coverage 90.64% 90.74% +0.10%
===========================================
Files 184 184
Lines 54826 55532 +706
===========================================
+ Hits 49699 50395 +696
- Misses 5127 5137 +10
🚀 New features to boost your workflow:
|
QuantumExplorer
marked this pull request as ready for review
May 10, 2026 20:51
…tree extensions CodeRabbit's Codecov pass on PR #659 flagged 86.42% patch coverage. Most gaps were unreachable!() arms (genuinely uncoverable), but several testable additions had no direct unit tests yet. - grovedb-query/src/proofs/tree_feature_type.rs: new tests module covering both new and existing helpers — `zero_sum` (each variant + no-op cases), `zero_count` (mirror), `count`. The file had no test module previously. - grovedb-element/src/element/mod.rs: new tests for `Display` of `NotSummed` and `NonCounted`, plus `element_type()` resolving to the correct `NotSummedXxx` / `NonCountedXxx` synthetic twin for each legal inner. - merk/src/element/tree_type.rs: tests asserting all `ElementTreeTypeExtensions` methods (`tree_type`, `maybe_tree_type`, `root_key_and_tree_type{,_owned}`, `tree_flags_and_type`, `tree_feature_type`) delegate through the `NotSummed` wrapper, plus `get_feature_type` zeros sum (and propagates count where applicable) for all four sum-bearing parent tree types. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QuantumExplorer
commented
May 10, 2026
QuantumExplorer
left a comment
QuantumExplorer
left a comment
Member
Author
There was a problem hiding this comment.
Codex review findings:
- [P1] Reject cross-wrapper construction (
grovedb-element/src/element/constructor.rs:393-407)
new_non_counted only rejects NonCounted, so it can still construct NonCounted(NotSummed(...)) even though serialization rejects cross-wrapper nesting. That value can reach batch execution with is_non_counted() == true and underlying() == NotSummed, then hit the supposedly-unreachable wrapper arm instead of returning a typed error.
Suggested fix: make new_non_counted reject both wrapper variants, and make into_non_counted avoid wrapping NotSummed as well, either by making it fallible or by adding a separate checked helper for external callers. Add a batch regression for NonCounted(NotSummed(SumTree)) to confirm it returns an InvalidInput / InvalidBatchOperation instead of panicking.
- [P2] Add serde-side wrapper validation (
grovedb-element/src/element/mod.rs:45-47)
The bincode serialize/deserialize paths now reject nested wrappers and NotSummed with a non-sum-tree inner, but the serde feature still derives recursive Deserialize for Element. With serde enabled, a payload can construct invalid NotSummed(Item) / cross-wrapper values, and deeply nested wrapper payloads still recurse before any validation runs.
Suggested fix: replace the derived serde deserialize path with a manual visitor or serde helper wrappers that enforce the same invariants as Element::deserialize: wrapper inners must not be wrappers, and NotSummed may only wrap SumTree, BigSumTree, CountSumTree, or ProvableCountSumTree. Add serde-feature tests for nested wrapper rejection and NotSummed(non_sum_tree) rejection.
…hains `reference_path::follow_reference` matched on the resolved `element` directly, so a `NonCounted(Reference)` would have been returned as a target value rather than followed as a hop. The function is currently only used from tests (production reference resolution goes through `GroveDb::follow_reference` in `operations/get/mod.rs`, which already unwraps via `into_underlying()`), but the divergence was a latent trap for any future wiring. Symmetric to the existing get-path unwrap. The wrapper byte is part of `value_hash` (computed from the on-disk bytes earlier in the function), so dropping it from `target_element` does not affect cryptographic verification of subsequent hops. `NotSummed` cannot wrap a reference (constructor whitelist enforces sum-tree variants), so this fix is purely forward-safe and symmetric to the `NonCounted` handling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses Codex's review of PR #659. **P1 — Reject cross-wrapper construction** `new_non_counted` previously rejected only `NonCounted` inners, so a caller could build `NonCounted(NotSummed(SumTree))` even though serialization rejects cross-wrapper nesting. Such a value would reach batch execution with `is_non_counted() == true` and `underlying() == NotSummed`, then hit the supposedly-unreachable wrapper arm. - `Element::new_non_counted` now rejects both `NonCounted` and `NotSummed` inners with `InvalidInput`. - `Element::into_non_counted` is now fallible (`Result<Self, ElementError>`) and returns `Err` for `NotSummed` input. Two in-tree callers in `grovedb/src/batch/mod.rs` already pass freshly-constructed bare tree elements; updated to `.expect(..)` with a comment documenting the precondition. - New regression in `grovedb/src/tests/not_summed_tests.rs` builds a hand-rolled `NonCounted(NotSummed(SumTree))` (bypassing the constructor) and asserts `apply_batch` rejects it as a typed error rather than panicking. - Two new unit tests in `grovedb-element/src/element/helpers.rs`: `into_non_counted_rejects_not_summed` and `new_non_counted_rejects_not_summed`. **P2 — Manual serde Deserialize with wrapper validation** The previous `derive(serde::Deserialize)` did not enforce the wrapper invariants. With the `serde` feature enabled, payloads could construct invalid `NotSummed(Item)` or cross-wrapper values, and deeply-nested wrapper payloads recursed before any check fired. - Replaced the derive on `Element` with a manual `Deserialize` impl in a `serde_impl` module, gated behind `feature = "serde"`. - The impl deserializes through a private `ElementShadow` mirror enum (`#[serde(rename = "Element")]` so the wire format is unchanged), converts to `Element`, then calls `Element::check_recursive_wrapper_invariants` to validate every level of the tree. - New public `Element::validate_wrapper_invariants` codifies the rules in one place (used by both the serde path and available to external callers). - `Serialize` derive is preserved — serialization of a valid Element is always safe. New serde-feature tests in `grovedb-element/src/element/mod.rs`: - `serde_round_trip_valid_elements` — JSON round-trip for Item, SumTree, NonCounted(Item), NotSummed(SumTree). - `serde_rejects_nested_non_counted` / `serde_rejects_nested_not_summed` — nested same-wrapper payloads. - `serde_rejects_cross_wrapper_nesting` — both `NonCounted(NotSummed(_))` and `NotSummed(NonCounted(_))`. - `serde_rejects_not_summed_with_non_sum_tree_inner` — six illegal inner types covering items, plain trees, count trees, MMR. - `serde_rejects_deeply_nested_wrapper_chain` — depth-64 chain rejected via the per-level check fired during recursive conversion. Adds `serde_json` as a dev-dependency for the JSON round-trip tests. Test counts: grovedb-element 63 lib (was 57 + 6 serde = 63), merk 418, grovedb 1496 (was 1495 + 1 regression). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three follow-ups from review: **1. Replace `into_not_summed_unchecked` (panicking) with fallible `into_not_summed`.** The panic path was a latent foot-gun. The new helper mirrors `into_non_counted`'s API: returns `Result`, idempotent on `NotSummed`, returns `Err(InvalidInput)` for `NonCounted` (mutually exclusive) or any non-sum-tree variant. The single in-tree caller (batch propagation) now uses `.expect(..)` with a comment documenting that the input is always a freshly-constructed bare sum-tree element. **2. Move `NotSummed` twin discriminants from `0x40 | base` to `0xb0 | base`.** This places the four twins at 180, 181, 183, 186 — inside the high-bit-set range alongside `NonCounted` twins (128..142), rather than down at 64..78 mid-range. The two ranges are still distinguished by the upper nibble: `0x80` for `NonCounted`, `0xb0` for `NotSummed`. Detection is now an upper-nibble compare instead of a single-bit test: - `is_non_counted`: `disc & 0xf0 == 0x80` (was `disc & 0x80 != 0`). - `is_not_summed`: `disc & 0xf0 == 0xb0` (was `disc & 0x40 != 0`). The single-bit `is_non_counted` would have started returning true for `NotSummed` twins too once they shared bit 7, which would have been wrong. Upper-nibble compares keep the predicates disjoint. Renamed the now-misnamed `NOT_SUMMED_FLAG` constant to `NOT_SUMMED_TWIN_PREFIX` (`= 0xb0`) and updated `NOT_SUMMED_BASE_MASK` to `0x0F` (sufficient since base discriminants are 0..14). Updated the `try_from`, `from_serialized_value`, the discriminant-pinning test, and added a new `test_not_summed_helpers` mirroring `test_non_counted_helpers`. **3. Remove unused `TreeFeatureType::zero_sum` helper.** It was added preemptively for symmetry with `zero_count` but had no production callers — `Element::sum_value_or_default` already returns 0 for `NotSummed`, so `get_feature_type` produces the right zero-summed feature type without needing this helper. Removed the function and its unit test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `serde_impl` module already had a comment describing the *approach* (shadow + From conversion + recursive validation), but not the *why* — which serde patterns fail and which alternatives exist. Codifies the trade-off so future maintainers can see at a glance that: - `#[serde(try_from)]` needs a separate source type (no self-pointing). - `#[serde(remote)]` is for foreign types only. - `#[serde(deserialize_with)]` is field-level. - `#[serde(transparent)]` / `flatten` are for single-field structs. Leaving three real options for derive-and-validate: shadow enum, manual Visitor, or drop the derive. We keep the derive (external tooling consumers may rely on it) and use the shadow as the shortest correct form. Doc-only change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rors The three `.expect(..)` calls added in commit 4a93a88 around the batch propagation wrapper-rewrap path would panic if a future change ever passed a pre-existing wrapper element through these branches. Replace them with `CorruptedCodeExecution` errors so the failure surfaces as a typed `Result` and bubbles up via the cost-aware return machinery instead of aborting the process. Sites: - `GroveOp::InsertTreeWithRootHash` propagation: `into_non_counted` and `into_not_summed` calls now `map_err(..)` to `CorruptedCodeExecution`, drained via `cost_return_on_error_no_add!`. - `GroveOp::InsertNonMerkTree` propagation: same treatment for the `into_non_counted` call. Behavior in the happy path is unchanged — the input elements are freshly built from `aggregate_data` / `meta.to_element(..)`, never wrapped, and the wrappers always succeed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
|
Reviewed |
QuantumExplorer
added a commit
that referenced
this pull request
May 10, 2026
Two PRs landed on develop while this PR was open: - #659: Element::NotSummed wrapper variant - #658: aggregate_count proof verification under verify feature Conflicts resolved in 8 files. The substantive resolution work: DISCRIMINANT COLLISION. The shipped cidx PR used ElementType discriminant 16 for CountIndexedTree and 17 for ProvableCountIndexedTree. Develop's NotSummed PR allocated byte 16 as NOT_SUMMED_WRAPPER_DISCRIMINANT. Both are still pre-shipping, so renumbering is safe — and since the NotSummed wrapper byte semantics need to round-trip across the wire, the wrapper byte stays at 16 and the cidx discriminants shift: CountIndexedTree: 16 → 17 ProvableCountIndexedTree: 17 → 18 NonCountedCountIndexedTree: 144 → 145 (= 0x80 | 17) NonCountedProvableCountIndexedTree: 145 → 146 (= 0x80 | 18) ENUM VARIANT ORDER. Bincode encodes Element variants by ENUM ORDER (not by ElementType discriminant). The Element enum has been reordered so NotSummed appears at variant index 16 (matching NOT_SUMMED_WRAPPER_DISCRIMINANT), and CountIndexedTree / ProvableCountIndexedTree appear AFTER NotSummed at indices 17/18 — matching their new ElementType discriminants. Without this reordering, bincode would still write 16 for CountIndexedTree but from_serialized_value would interpret 16 as the NotSummed wrapper. WRAPPER NESTING CHECK in `from_serialized_value`: now explicitly rejects both nested NonCounted and cross-nesting with NotSummed (inner_byte == 15 || inner_byte == 16), in addition to the existing reject-high-bit-twin guard. Other resolutions are straightforward additions: both branches added arms to match expressions across helpers.rs, tree_type.rs, visualize.rs; merged additively. The two `mod tests` blocks in grovedb-element/src/element/mod.rs (one from each PR) renamed the cidx-side block to `cidx_tests` to avoid duplicate name. All 1944+ workspace tests pass post-merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
QuantumExplorer
added a commit
that referenced
this pull request
May 11, 2026
[P1] 247-byte ceiling on cidx primary item keys (direct + batch). Secondary keys are (count_be ‖ item_key); Merk requires keys < 256 bytes. Generic batch validation only enforces 255-byte cap, so cidx primaries need an 8-byte stricter ceiling. Added MAX_CIDX_ITEM_KEY_LEN = 247 and validate_cidx_item_key_len. Enforced on insert_into_count_indexed_tree_on_transaction and on execute_ops_on_path when in_tree_type is cidx primary. [P1] insert_into_count_indexed_tree overwrite cleanup. The dedicated API read existing element only for count delta — didn't clean up old tree storage when replacing a tree entry. Mirrors batch safe-subset semantics: - existing tree, new non-tree: ALLOW + cleanup - existing tree, new empty tree (root_key=None): ALLOW + cleanup - existing cidx, new non-empty cidx: REJECT (ambiguous) - existing tree, new non-empty non-cidx tree: REJECT (ambiguous) Cleanup mirrors db.delete/batch DeleteTree: find_subtrees + clear, plus secondary namespace clear for existing-cidx. [P2] Batch consistency check for safe-subset overwrite descendants. When scheduling cidx-overwrite cleanup, scan ops_by_qualified_paths for descendants of the cidx path and reject as InvalidBatchOperation if found. Defense in depth; the audit's worry about silent cleanup-drop is already caught by other checks in most flavors, but the new check provides a clearer error message. [P2] verify_grovedb duplicate-secondary detection. Changed HashMap<Vec<u8>, u64> to HashMap<Vec<u8>, Vec<u64>> so duplicate secondary rows for the same primary key (real drift class) are flagged via __cidx_secondary_duplicate__ sentinel path. [P2] appendix-a.md: updated discriminants 16/17/144/145 → 17/18/ 145/146, added NotSummed wrapper byte row, added 247-byte ceiling note. Now matches code (post-merge with #659). All 1611 grovedb tests pass; 5 new audit-targeted tests added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QuantumExplorer
added a commit
that referenced
this pull request
Aug 2, 2026
…ti-axis secondary indexing (#657) * feat: add CountIndexedTree element with auto-cascading secondary index Adds two new GroveDB element types — CountIndexedTree and ProvableCountIndexedTree — that pair a CountTree-shaped primary Merk with a count-ordered secondary Merk for sub-linear top-k and count-range queries. Each element points at two child Merks. The parent Merk binds both via H1-A composition: combined_value_hash = Blake3(actual_value_hash || primary_root_hash || secondary_root_hash). The secondary is itself a ProvableCountTree (each entry contributes count = 1) so existing AggregateCountOnRange machinery applies natively. Storage prefix derivation (S2-B): primary keeps the existing build_prefix(path); secondary is Blake3(primary_prefix || 0x01). Public API: - insert_into_count_indexed_tree / delete_from_count_indexed_tree — dedicated direct APIs that mirror to the secondary inline and chain the H1-A combine into the parent. - count_indexed_top_k / count_indexed_count_range — read APIs walking the secondary in count order. - reconcile_count_indexed_tree_secondary — rebuild the secondary from the primary on demand; used after batch operations that bypass the dedicated write path. - prove_count_indexed_top_k / verify_count_indexed_top_k — proof generation and verification for top-k queries, binding the secondary range proof to the GroveDB root hash via the H1-A composition. - Empty CountIndexedTree elements can be created via apply_batch. Auto-cascading: propagate_changes_with_transaction is now CountIndexed- aware. When the propagation pass crosses a CountIndexedTree primary level, it mirrors the count delta to that level's secondary; when a CountIndexedTree element needs reconstruction, it uses the H1-A three-input combine. Nested CountIndexedTrees and deep db.insert paths through sub-trees of a cidx primary cascade correctly. Design doc at docs/book/src/count-indexed-tree.md captures the ratified decisions (H1-A, S2-B, V1-A, Q1-A, S1-A, Q2 with conditional subqueries deferred). Spike note at docs/spikes/cascading-aggregation-spike.md records the architectural analysis for the propagation refactor. Tests: 27 dedicated tests covering empty creation, insert/update/delete with count deltas, NonCounted handling, deep cascading through sub-trees, nested CountIndexedTrees, top-k and count-range queries, reconciliation, batch creation, proof round-trips, and forge tests (tampered bytes, wrong path). Workspace: 2615 lib tests pass, no regressions. Deferred for follow-up: - Item-level batch inserts INTO a cidx primary (use the dedicated API) - Replication / chunk restoration support for two-Merk subtrees - Conditional-by-count subqueries within CountIndexedQuery (Q2.3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address CodeRabbit review on cidx PR #657 Fixes CI lint failure (debugger.rs match arms) and ten CodeRabbit review items on the CountIndexedTree implementation: - Doc status banner: "awaiting implementation" → "implemented" - Doc wording: "collision-free" → "domain-separated" for hash-derived prefixes - verify_grovedb: fail closed (NotSupported) for cidx instead of silently skipping; integrity verification needs the H1-A three-input combine and dual-Merk traversal which is not yet wired - V1 prove_subqueries_v1: explicitly reject subqueries into cidx with NotSupported instead of silently emitting an unverifiable proof; callers must use prove_count_indexed_top_k - Batch DeleteTree on cidx: reject because the standard delete path only cleans up one child Merk and would orphan the secondary storage namespace - Generic batch path: document the cidx overwrite footgun (same shape as other tree types when the override-protection flag is off) - count_indexed_count_range: replace full secondary scan with a bounded Query::insert_range using big-endian count bytes, falling back to insert_range_from when hi_count == u64::MAX - query_item_value_or_sum reference branch: include cidx variants alongside the direct-element branch - prove_count_indexed_top_k: reject nested cidx on the proven path with NotSupported (envelope only carries H1-A attestation data for the terminal cidx); verifier naturally fails the chain check if a forged envelope smuggles a nested cidx - combine_hash_three: correct the doc comment to match the cost constant; 96 bytes spans two 64-byte Blake3 blocks (the previous comment incorrectly conflated blocks with chunks) - reconcile test: rename to reconcile_after_query_returns_correct_top_k to reflect what the test actually verifies (true desync test requires unavailable internal APIs) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: support direct insertion of non-empty CountIndexedTree elements The direct (non-batch) insert path previously rejected any CountIndexedTree element whose primary_root_key, secondary_root_key, or count_value was non-zero, with an error claiming non-empty insertion required the batch path (which itself does not yet support non-empty cidx). This is the migration / restore-from-backup direct-insertion path. For non-empty cidx elements, open the existing primary and secondary Merks at the new path, validate that the caller's declared root keys match the on-disk state, and read the actual root hashes for the H1-A combined value hash so the parent's value_hash is consistent with disk. Mismatched root keys fail loudly. Also delete docs/spikes/cascading-aggregation-spike.md — internal and external dev-relevant content for cidx lives in the book chapter (docs/book/src/count-indexed-tree.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover new cidx reject/error paths and edge cases Lifts patch coverage on the cidx PR by adding focused tests for the error paths and rejections introduced over the last few commits, plus two extra cidx behaviors that were not yet exercised: - direct_insert_rejects_mismatched_secondary_root_key (mismatch on secondary key, mirroring the existing primary-key test) - batch_delete_tree_on_cidx_is_rejected (DeleteTree on cidx via batch must error to avoid orphaning secondary storage) - verify_grovedb_fails_closed_for_cidx (NotSupported instead of silent skip) - prove_count_indexed_top_k_at_root_path_errors - prove_count_indexed_top_k_on_non_cidx_target_errors - count_indexed_top_k_on_non_cidx_target_errors - count_indexed_count_range_on_non_cidx_target_errors - reconcile_on_non_cidx_target_errors - delete_from_count_indexed_tree_on_non_cidx_target_errors - delete_from_count_indexed_tree_returns_false_for_unknown_key - count_indexed_count_range_descending_returns_descending_order (covers the descending bounded-range branch) - test_v1_proof_rejects_count_indexed_tree_subquery (V1 generic prove path rejects cidx subqueries) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover cidx Display, Visualize, and proof/range edge paths Lifts patch coverage above the codecov 80% threshold by hitting the 0%-covered Display impls, the gated Visualize impls, helper queries on Element, the count_range / top_k edge cases, and the verifier's error paths: - count_indexed_tree_display_renders_fields - provable_count_indexed_tree_display_renders_fields - count_indexed_tree_helpers_report_count_and_type (is_count_indexed_tree, is_any_tree, element_type, NonCounted look-through) - test_visualize_count_indexed_tree_empty (visualize feature) - test_visualize_count_indexed_tree_with_keys (visualize feature) - test_visualize_provable_count_indexed_tree (visualize feature) - count_indexed_count_range_with_lo_greater_than_hi_returns_empty - count_indexed_count_range_with_hi_count_u64_max_uses_range_from - count_indexed_count_range_respects_limit - count_indexed_top_k_with_zero_returns_empty - count_indexed_top_k_at_root_path_errors - count_indexed_count_range_at_root_path_errors - verify_count_indexed_top_k_rejects_corrupt_proof_bytes - verify_count_indexed_top_k_rejects_path_length_mismatch Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: clarify V0 proof path for cidx is a permanent NotSupported V0 is a frozen on-the-wire proof format. Adding cidx descent to it would be a wire-format change, so V0 will never learn cidx subqueries. Reword the V0 prover and verifier comments / error messages to make that explicit instead of implying the work is pending in a follow-up PR. The dedicated `prove_count_indexed_top_k` / `verify_count_indexed_top_k` entry points and the (still TODO) V1 generic path remain the supported routes for cidx queries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: H1-A walk in verify_grovedb + nested cidx in prove envelope verify_grovedb: replace fail-closed NotSupported with the actual H1-A integrity walk for cidx nodes. Open both child Merks, read their root hashes, verify the parent's recorded value_hash equals combine_hash_three(value_hash(cidx_bytes), primary_root, secondary_root), then recurse into the primary normally. While doing this, fix a pre-existing bug in insert_into_count_indexed_tree: it called Element::insert (Op::Put, no combine) regardless of element kind. For tree subtree elements that meant the cidx primary's merk node stored value_hash = value_hash(serialized) instead of combine_hash(value_hash, NULL_HASH), breaking the merkle invariant of the cidx primary until a deep insert later updated it via propagation. Dispatch on element kind so trees take Element::insert_subtree, nested cidx takes Element::insert_count_indexed_subtree, references and items keep the prior path. Now the cidx primary's root hash is correct immediately after creation, and verify_grovedb can recurse cleanly. prove_count_indexed_top_k: extend CountIndexedRangeProof with ancestor_cidx_secondary_root_hashes (Vec<Option<[u8;32]>> aligned with intermediate layers). When building, capture each cidx ancestor's secondary root hash. When verifying, chain via combine_hash_three at cidx ancestor layers, combine_hash elsewhere. Removes the prior nested-cidx prover-side rejection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: V1 generic prove/verify support cidx subqueries Subqueries into CountIndexedTree via the generic V1 PathQuery pipeline now produce a verifiable proof. The cidx primary is the descent target; queries against the secondary still go through the dedicated prove_count_indexed_top_k path. Wire format: - New ProofBytes::CountIndexedTree(secondary_root || primary_proof) variant. The 32-byte secondary attestation is captured from the cidx's secondary Merk root hash at proof-build time; the primary proof bytes are a standard Merk proof of the subquery results generated by prove_subqueries_v1 against the cidx primary. - LayerProof and ProofBytes derive Clone so the verifier can synthesize a sibling Merk-shaped LayerProof from the cidx-prefixed bytes and recurse into the existing verify_layer_proof_v1. Generate (V1): replace the previous NotSupported with descent that calls prove_subqueries_v1 on the cidx primary, opens the secondary to capture its root hash, and re-wraps the resulting Merk proof bytes with the secondary attestation prefix. Verify (V1): when a lower_layer's parent element is a cidx, require ProofBytes::CountIndexedTree, split off the 32-byte secondary attestation, synthesize a Merk LayerProof for the primary, recurse to obtain primary_root_hash, then chain via combine_hash_three(value_hash, primary_root, secondary_root) instead of combine_hash. Reject any other ProofBytes variant under a cidx parent and any ProofBytes::CountIndexedTree under a non-cidx parent. V0 still rejects (V0 wire format is frozen). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: batch path fail-closed for cidx primary mutations The level-by-level batch propagation has no two-Merk hook for CountIndexedTree primaries: applying mutation ops directly to the primary updates the primary's root hash but leaves the secondary index stale, breaking both the H1-A composition stored in the parent's cidx element bytes and the count-ordered query semantics. Reject mutation ops (Insert/Replace/Patch/Delete/RefreshReference) in execute_ops_on_path when the merk's tree_type is a cidx primary, with a clear NotSupported message pointing callers to the dedicated APIs (insert_into_count_indexed_tree / delete_from_count_indexed_tree). Up-bubbled internal ops (ReplaceTreeRootKey, InsertTreeWithRootHash, etc.) remain allowed — those represent a child subtree's response to its own change and are handled correctly by the existing propagate_changes_with _transaction_with_initial_deferred path that already mirrors to the secondary at the cidx element boundary. Full batch integration of cidx primary mutations would require a new GroveOp variant carrying both primary and secondary state plus a refactor of the per-level propagation pass; that is a substantial piece of work and belongs in its own follow-up. Until then, fail-closed is preferable to silently corrupting the index. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: refresh stale cidx follow-up comments Several module/function comments still claimed cidx features were "a follow-up" or "not yet wired" after this PR's earlier commits implemented them. Update wording to reflect current state: - count_indexed_tree.rs module doc: clarify the dedicated APIs are required for direct cidx primary mutations and that the batch path fails closed until full batch integration lands; deep ops under sub-trees of cidx primaries propagate correctly today. - count_indexed_top_k doc: drop the "no proofs yet" note and point at prove_count_indexed_top_k / verify_count_indexed_top_k. - count_indexed_tree_tests.rs module doc: drop the PR-2-staging banner that claimed item insertion / cascading aggregation were unexercised. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: lift codecov patch coverage above 80% CI codecov/patch is failing at 79.45% (target 80%). Add focused tests targeting recently-added paths that were not yet exercised: - insert_into_count_indexed_tree_with_reference_to_missing_target_errors: covers the new reference-resolution path for cidx primary inserts when the target does not exist. - deep_insert_under_nested_cidx_propagates_through_both_levels: covers the nested-cidx propagation path end-to-end (deep insert three levels under outer cidx -> inner cidx -> sub count tree) including the new H1-A walk in verify_grovedb at both cidx levels. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: more cidx coverage to clear codecov 80% threshold Codecov patch is at 79.91% (target 80%) — 13 hits short. Add four focused tests covering paths recently added but not yet exercised: - delete_from_count_indexed_tree_round_trip_with_proof: end-to-end delete + prove + verify. - verify_count_indexed_top_k_rejects_truncated_proof: covers the bincode decode error branch. - verify_grovedb_walks_provable_count_indexed_tree: same H1-A walk on the ProvableCountIndexedTree variant. - test_v0_proof_rejects_count_indexed_tree_subquery: covers the V0 prover's cidx subquery rejection arm. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: cidx batch foundation - new op + parent-level handler Lands the structural pieces for cidx primary batch-path support. No user-visible behavior change: the new op variant is never produced yet (the rejection at execute_ops_on_path:1862 still fires for cidx primary mutations), but the parent-level handler is in place so that emitting the op from a future bubble-up hook is mechanical. - GroveOp::ReplaceCountIndexedTreeRootKeys: new internal op variant. Carries both primary and secondary new-state (root_hash + root_key + count_aggregate). Marked #[non_exhaustive] like the other internal variants. Sort weight 17, debug formatter, all match arms in references / preprocessing / format / cost / sort logic exhaustively cover it (rejected as 'internal only' from user- facing entry points). - update_count_indexed_tree_item_preserve_flag_into_batch_operations: parallels update_tree_item_preserve_flag_into_batch_operations but reconstructs via reconstruct_with_two_root_keys (cidx) and emits Op::ReplaceLayeredCountIndexedReference (combine_hash_three / H1-A) instead of Op::ReplaceLayeredReference. Preserves flags. - Parent-level handler: when execute_ops_on_path sees the new op at a parent merk, it calls the helper above to recompute the cidx element's value_hash via H1-A. Subsequent commits will: (a) wire a get_secondary_merk_fn closure through TreeCacheMerkByPath, (b) detect cidx primaries in execute_ops_on_path and mirror item-level mutations to the secondary, (c) modify the bubble-up to emit the new op variant when the just-finished level was a cidx primary. Tests for the end-to-end behavior land alongside (c). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: batch path supports cidx primary item-level mutations Lifts the level-by-level batch path from rejecting cidx primary mutations to supporting them end-to-end. A batch op that inserts / replaces / patches / deletes / refreshes items inside a cidx primary now correctly mirrors to the secondary index and updates the cidx element on the parent merk via H1-A composition. Implementation: 1. TreeCacheMerkByPath gained a get_secondary_merk_fn closure (opens the cidx secondary by primary path, looks up secondary_root_key from the parent merk's cidx element internally) and a side-channel cidx_secondary_after_apply: HashMap<Vec<Vec<u8>>, ...> populated by execute_ops_on_path when the level was a cidx primary. 2. execute_ops_on_path: when in_tree_type is cidx primary, captures pre-state (per-key old count_value via merk.get) before the apply pass. After apply_with_specialized_costs returns it re-reads each key's post-apply element, opens the secondary, runs mirror_to_secondary_for_batch (new helper handling all four insert/update/delete/no-op cases), and stores secondary's state in the side-channel. 3. Bubble-up: pulls the cidx state via the new take_cidx_secondary_after_apply trait method. When present, emits GroveOp::ReplaceCountIndexedTreeRootKeys instead of ReplaceTreeRootKey at the parent level (covers all four bubble-up paths: Vacant, Occupied, missing parent map, missing level-above). 4. Parent execute_ops_on_path: handles the new op via update_count_indexed_tree_item_preserve_flag_into_batch_operations which reconstructs with new root keys + count and emits Op::ReplaceLayeredCountIndexedReference for combine_hash_three. 5. open_count_indexed_secondary_for_batch helper on GroveDb: convenience wrapper used by the closure that does the parent merk lookup + secondary open in one call. batch_insert_into_cidx_primary_works test verifies end-to-end. verify_grovedb walks the H1-A chain and finds no issues afterward. Still TODO (separate follow-up): DeleteTree on cidx primary, cidx overwrite via Replace, comprehensive atomicity tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: prove_count_indexed_query accepts an arbitrary secondary query prove_count_indexed_top_k was a special case (full-range, ascending or descending). Lift it to a thin wrapper around a general prove_count_indexed_query that takes any MerkQuery over the cidx secondary's keyspace (keys are count_value_be ‖ original_key, so callers can express count == X, count in [lo, hi], count >= X, count == X AND original_key starts with Y, etc. by building the query in those bytes). Refactored the inner build_count_indexed_proof to take (secondary_query, limit) instead of (k, descending); the user- supplied query.left_to_right is echoed in the envelope's `descending` field for the existing top-k convenience field, and limit's None gets stored as 0 (verifier treats 0 as no-limit). Symmetric verifier change: split verify_count_indexed_top_k into a thin wrapper + verify_count_indexed_inner generic core, and added verify_count_indexed_query taking the same MerkQuery the prover used (positional binding requires identical query at both ends). Test prove_count_indexed_query_with_count_range covers a non-trivial case: a cidx with five items at counts {1,2,3,5,8}, query [3, 6) inclusive of 3 and 5, exclusive of 8. Verifier returns exactly (3, c), (5, d). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: reject silent cidx overwrites in the batch path Closes a real corruption gap: when validate_insertion_does_not_override_tree was off, a batch InsertOrReplace / Replace / Patch could silently overwrite an existing cidx element. The merk node value would change, but the cidx primary's storage namespace + the secondary's storage namespace (Blake3(primary_prefix || 0x01)) would be left behind. Future inserts under the new cidx's primary_root_key could then collide with the orphaned data, and the secondary index on the old data would be unreachable. When the override-protection flag is on (typical case), the existing rejection of "attempting to overwrite a tree" already catches cidx since is_any_tree() returns true. When the flag is off, however, the path silently corrupts. Add an unconditional cidx-specific check that fires for InsertOrReplace / Replace / Patch ops on non-reference elements when the override flag is off: read the existing element at the key once, and if it decodes to CountIndexedTree / ProvableCountIndexedTree, reject with NotSupported pointing at the delete_from_count_indexed_tree / delete_up_tree workflow. Other tree-type overwrites remain permitted under the existing opt-out semantics for backwards compatibility — this stricter treatment is specific to cidx because cidx owns two storage namespaces and the corruption is qualitatively worse. Updates one cost test (+1 seek, +129 storage_loaded_bytes) where the new check fires. The new test batch_overwrite_existing_cidx_with_item_is_rejected verifies the guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover cidx batch delete + multi-op paths to clear codecov 80% Codecov patch is at 79.75% (target 80%) — 7 hits short. Add two focused tests covering the new batch cidx code paths: - batch_delete_item_from_cidx_primary_works: covers the Delete arm of mirror_to_secondary_for_batch (new_count = None) and the pre-state capture for Delete ops. - batch_multiple_inserts_into_cidx_primary_in_one_call: covers the multi-key pre-state capture loop and the per-key mirror loop in execute_ops_on_path on a cidx primary path. Both run verify_grovedb afterward to walk the H1-A chain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: db.delete() on a cidx element cleans up secondary storage Closes a real corruption gap: db.delete() of a CountIndexedTree element walked the primary's storage namespace via find_subtrees + storage.clear() but left the secondary's storage namespace (Blake3(primary_prefix || 0x01)) untouched. After the cidx element was removed from the parent merk, the secondary's data became unreachable but stayed on disk; if the user later re-created a cidx at the same path, queries against the secondary could observe stale entries from the previous incarnation. Add a cidx-specific cleanup branch in delete_internal_on_transaction (the standard tree-delete code path). When the deleted element's tree_type is a cidx primary, derive the secondary prefix via the existing RocksDbStorage::secondary_prefix_for helper, open storage at that prefix, and call .clear(). Runs unconditionally (not gated on is_empty) so empty-cidx deletes also clear the secondary's root metadata for consistency. Two new tests verify the cleanup end-to-end via the re-create- and-query pattern: if the secondary wasn't cleaned, the new cidx's top-k query would return stale entries. - direct_delete_empty_cidx_cleans_up_secondary_storage - direct_delete_non_empty_cidx_cleans_up_both_namespaces Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: batch DeleteTree on cidx primary works end-to-end Lifts the rejection of DeleteTree on CountIndexedTree / ProvableCountIndexedTree in the batch path. Previously batch users were forced to fall back to db.delete() outside the batch — fine for single-cidx workflows but breaks atomicity when DeleteTree is mixed with other batch ops. Implementation parallels the H1-A delete fix in commit 6b7ec21d (direct path): the existing tree-delete cleanup pipeline collects deleted Merk paths into `merk_delete_paths` and runs find_subtrees + storage.clear() on each post-apply. Since find_subtrees only walks primary keys, the cidx secondary storage namespace at Blake3(primary_prefix ‖ 0x01) was orphaned. Add a parallel cidx_primary_delete_paths collector that captures cidx primary DeleteTree ops at validation time, then runs an explicit secondary-prefix .clear() in the post-apply pass alongside the primary cleanup. Done in both apply_batch_with_element_flags_update and apply_partial_batch (the partial-batch variant). Two new tests use the re-create-and-query pattern to verify the cleanup: - batch_delete_tree_on_empty_cidx_works - batch_delete_tree_on_non_empty_cidx_works Both query the new cidx's secondary index after re-creation; if the old secondary weren't cleaned the queries would return stale entries. Cidx overwrite via batch (Replace cidx → cidx / non-cidx) remains rejected. The semantics of replacing an existing cidx element where the new element references on-disk data are ambiguous and the safe subset will land separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: refresh cidx-overwrite rejection error message Now that batch DeleteTree on cidx works (commit 0688731a), the recommended workaround for overwriting an existing cidx is: 1. delete_from_count_indexed_tree to empty it 2. DeleteTree via batch (now supported) 3. Re-create in a follow-up batch Update the rejection error message to point at this clean workaround instead of the older "delete_up_tree outside of a batch" guidance. The full safe subset of cidx overwrites (cidx → non-cidx, cidx → empty cidx) requires moving cidx-overwrite detection into the pre-apply scan alongside the DeleteTree discovery loop, plus careful sequencing of post-apply cleanup vs. new-element write. That is left for a follow-up; the workaround above is clean today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover batch cidx override-protection + recreate workflow Codecov patch is at 79.52% (target 80%) — 14 hits short. Add two focused tests covering newly-added batch paths not yet exercised: - batch_overwrite_cidx_rejected_with_override_protection_on: covers the validate_insertion_does_not_override_tree=true branch hitting cidx (existing-element-is-tree path). - batch_delete_tree_on_cidx_then_recreate_in_separate_batch_works: covers the recommended cidx-overwrite workaround end-to-end — DeleteTree the cidx in batch 1, re-create empty in batch 2, populate in batch 3 — and verifies via verify_grovedb that the H1-A chain is consistent throughout. The recreate test highlights an important sequencing detail: a cidx and ops INSIDE the cidx primary cannot share a single batch because deeper-path ops execute before the cidx itself exists. This is documented in the test's structure (3 separate batches). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover apply_partial_batch + DontCheckWithNoCleanup cidx paths Codecov patch is at 79.52% (target 80%). Earlier tests exercised the apply_batch cidx-cleanup path but the parallel cleanup pass in apply_partial_batch and the DontCheckWithNoCleanup branch were untested. Add two focused tests: - apply_partial_batch_with_delete_tree_on_cidx_cleans_up_secondary: routes through apply_partial_batch and verifies the secondary cleanup ran via the re-create-and-query pattern. - batch_delete_tree_on_cidx_dont_check_with_no_cleanup_still_clears _secondary: covers the DontCheckWithNoCleanup branch which skips primary find_subtrees but must still clear the cidx secondary prefix (a different namespace not covered by find_subtrees). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs+test: book chapter refresh + batch cidx atomicity stress tests Two parallel polish items: DOCS — refresh the book chapter to reflect what shipped. The chapter was design-spec style (Status: implemented, but conceptual APIs that don't match the actual code). Update the API code blocks to the shipped function signatures (count_indexed_top_k, count_indexed_count_range, prove_count_indexed_top_k, the new prove_count_indexed_query taking arbitrary MerkQuery), replace the hypothetical CountIndexedQuery struct with the two-route subquery description (V1 generic PathQuery + dedicated cidx proof), add a new "Batch path semantics" section documenting supported / rejected ops plus the cidx-overwrite workaround, and update the Implementation-detail items table from "Recommended default" to "Resolution" reflecting what landed (W1: specialized propagation through propagate_changes_with_transaction_with_initial_deferred + GroveOp::ReplaceCountIndexedTreeRootKeys at the bubble-up). ATOMICITY — five new stress tests for batches mixing cidx + non-cidx. GroveDB batches are atomic by design (validation runs over the full op list before any writes hit storage). These tests verify the cidx- aware paths preserve that invariant under mixed workloads: - batch_mixed_cidx_and_non_cidx_ops_apply_atomically - batch_failure_in_non_cidx_op_rolls_back_cidx_mutations - batch_with_multiple_cidx_primaries_each_get_updated - batch_cidx_delete_with_concurrent_cidx_inserts_atomic - batch_failure_after_cidx_delete_tree_rolls_back Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: nested cidx bubble-up mirrors count change to outer's secondary Real audit finding: the cidx primary pre-state capture in execute_ops_on_path lists every mutating op variant EXCEPT the new GroveOp::ReplaceCountIndexedTreeRootKeys variant introduced for the cidx-aware bubble-up. When a NESTED cidx primary bubbles up to its OUTER cidx primary via the batch path: Level N (inner cidx primary): mutates fire, secondary mirrored, bubble emits ReplaceCountIndexedTreeRootKeys to level N-1. Level N-1 (outer cidx primary): receives the op at key=inner_key; handler `update_count_indexed_tree_item_preserve_flag_into_ batch_operations` correctly updates the inner_key element's bytes (new primary_root_key, secondary_root_key, count_value). But pre-state capture skipped this op type, so post-apply mirror walked an empty deltas list. Outer's secondary was not updated. The corruption was silent: H1-A integrity (verify_grovedb) still passed because the outer's stored value_hash is recomputed from the actual on-disk secondary root hash — the secondary just has stale content. Top-k / count-range queries on the outer returned stale counts. Fix: add the variant to the mutates match. With the fix, the outer's secondary entry for inner_key correctly moves from (old_count_be ‖ inner_key) to (new_count_be ‖ inner_key) when the inner's count changes. Test batch_insert_into_nested_cidx_primary_bubbles_count_up_outer_ secondary fails BEFORE the fix (asserts top[0] == (1, b"inner_cidx") but gets (0, b"inner_cidx")) and passes AFTER. Found via audit of the new code paths — there was no batch-path nested-cidx test before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: nested cidx coverage — direct path, triple-nested, mixed nesting Lock down the ReplaceCountIndexedTreeRootKeys-mutates fix from the prior commit with three additional nesting tests: - direct_insert_into_nested_cidx_primary_bubbles_count_up_outer_ secondary - batch_insert_into_triple_nested_cidx_propagates_through_all_levels - batch_insert_through_cidx_then_regular_tree_then_cidx (cidx → regular CountTree → cidx mixed nesting) All 1566 grovedb tests pass; release-mode build also passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix+test: close verify_grovedb cidx content-consistency gap Was the highest-priority audit item from my earlier self-grade: the verify_grovedb H1-A walk verifies *chain* integrity but not *content* consistency between the primary's count_value field and what the secondary actually contains. The nested-cidx bug found in a8bb34fb was exactly this class — stale secondary that internally hashes correctly but reports wrong counts. H1-A passed; queries lied. Three changes: 1. CONTENT-CONSISTENCY CHECK in verify_grovedb. After the H1-A check on each cidx primary, walk both Merks' raw storage and assert per-entry consistency. Mismatches are recorded in the existing VerificationIssues HashMap with sentinel path suffixes (__cidx_primary_orphan__, __cidx_secondary_orphan__, __cidx_count_mismatch__, __cidx_secondary_malformed_key__) so the public API stays unchanged. 2. db.insert() REJECTS cidx primary targets. Adding the check revealed a real direct-path bug: db.insert(cidx_primary, ...) wrote to the primary without mirroring to the secondary, leaving the same kind of drift the new check catches. Route users to insert_into_count_indexed_tree with a NotSupported error. 3. DELIBERATE-CORRUPTION TESTS. Three tests directly manipulate the secondary's storage via Element::insert/delete to introduce each drift class: - verify_grovedb_catches_secondary_missing_entry_for_primary - verify_grovedb_catches_orphan_in_secondary - verify_grovedb_catches_count_mismatch_between_primary_and_ secondary Plus direct_db_insert_into_cidx_primary_is_rejected covering the rejection from item 2. Without item 1, all three corruption tests would silently pass an integrity check. With it, the class of bug that took an audit to find is now CI-caught for any future regression. All 1576 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract cidx mutates check into exhaustive GroveOp method Structural guard against the nested-cidx bug class (commit a8bb34fb). That bug existed because the pre-state capture in execute_ops_on_path used an inline `matches!()` against a hand-maintained list of mutating GroveOp variants. When `ReplaceCountIndexedTreeRootKeys` was added as part of cidx primary bubble-up support, the inline match wasn't updated — so the new variant silently fell through to "doesn't mutate" and the outer's secondary stayed stale. The bug passed H1-A integrity checks; only a manual audit caught it. The fix in a8bb34fb added the variant to the list, but the list is still hand-maintained and the next new variant has the same trap. This commit converts the inline `matches!()` into a method `GroveOp::can_mutate_child_count(&self) -> bool` with an exhaustive `match` and no wildcard arm. Adding any new `GroveOp` variant now forces the author to classify it explicitly — the compiler will refuse to compile until they do. Same protection the existing `to_u8()` method already provides for ordering. Each variant has a comment explaining why it's `true` or `false`: - Leaf-level mutations (Insert/Replace/Patch/Delete/RefreshReference) → true: directly change a key's count_value. - Bubble-up ops (ReplaceTreeRootKey, InsertTreeWithRootHash, ReplaceNonMerkTreeRoot, InsertNonMerkTree, and the new ReplaceCountIndexedTreeRootKeys) → true: each updates the child element bytes which, for count-bearing trees, changes the aggregated count_value (the secondary's sort key). - Non-Merk-tree leaf inserts (Commitment/MMR/BulkAppend/DenseTree) → false: these trees use non-Merk storage and don't contribute counts the same way; their propagation is tree-specific. No behavior change. All 1576 grovedb tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix+test: property tests + delete_from_count_indexed_tree storage cleanup Property test (A-grade item 3) found a real bug: delete_from_count_indexed_tree didn't clean up the deleted entry's child storage when the entry was a tree (CountTree, etc.). Same class as the storage-orphan bugs in db.delete (6b7ec21d) and batch DeleteTree (0688731a): - Remove cidx entry from primary's merk tree ✓ - Remove secondary mirror entry ✓ - Clean up the entry's child subtree storage ✗ ← THE BUG Caught in iteration <100 of the property test. FIX: after Element::delete on the cidx primary entry, when the entry was a tree, run find_subtrees on its path and clear each subtree's storage. For nested cidx entries also clear its secondary storage. primary_merk stays live (no drop+reopen) because dropping would lose the staged Element::delete write. PROPERTY TESTS: 300 iterations against single-level cidx + 200 iterations against nested two-level cidx. Each op type (insert / delete / re-create / batch insert) followed by verify_grovedb + top-k diff against an in-memory model. Hand-rolled SplitMix64 PRNG with fixed seeds for reproducibility, no new deps. All 1578 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: tighten weak cidx rejection tests to check error variants A-grade item 5. Replaces the `assert!(result.is_err())` cluster of weak rejection checks with variant-specific assertions. Tests pass even if the underlying code's failure mode shifts — that's a real test smell that masks regressions. Tightened ~13 tests across the file with `assert!(matches!(result, Err(VARIANT(...))))` plus key-phrase substring checks on the error messages. For two tests (direct_insert_rejects_mismatched_*_root_key) discovered the actual error variant is `InvalidParentLayerPath` (the primary merk can't be opened because the cidx hasn't been created yet); documented and accept both variants. A few `is_err()` cases are left intentionally loose — atomicity tests where the assertion is about "post-state matches pre-state" and the specific failure mode is secondary. All 1578 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: differential test for direct API vs batch path cidx state A-grade item 4. The dedicated `insert_into_count_indexed_tree` / `delete_from_count_indexed_tree` API and the batch path (`apply_batch` with InsertOrReplace/Delete ops) are two independent implementations of the same logical operation. They must produce byte-identical state. Test parameterizes over 4 representative operation sequences: - 3 distinct inserts - insert + overwrite same key - inserts in varying key order - inserts then a middle-key delete For each sequence, applies it twice — once via direct API, once via batch — into two fresh DBs and asserts on the result: 1. Identical GroveDB root hashes 2. Identical cidx element bytes at the parent merk 3. Identical top-k results 4. Both pass verify_grovedb (chain + content) If either path drifts (different mirror order, different value_hash recomputation, different cost surface), the test fails. Acts as a permanent regression guard for the dual-implementation surface. All 1579 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: safe-subset cidx-overwrite via batch (cidx → non-cidx, cidx → empty cidx) A-grade item 1 — closes the "incomplete feature" knock. The batch path now supports cidx-overwrite in the unambiguous-semantics cases: cidx → non-cidx element → ALLOW + cleanup cidx → empty cidx → ALLOW + cleanup cidx → non-empty cidx → REJECT (ambiguous, unchanged) IMPLEMENTATION: TreeCacheMerkByPath gains cidx_overwrite_cleanup_ paths populated by execute_ops_on_path. Trait method take_cidx_overwrite_cleanup_paths extracts them. apply_batch_ structure's return type changes to a tuple so paths flow up to apply_batch_with_element_flags_update and apply_partial_batch, which run post-apply cleanup mirroring the cidx DeleteTree path (find_subtrees + storage.clear() + secondary prefix clear). apply_partial_batch unions paths from both apply_body and continue_partial_apply_body. TESTS: 3 new tests covering the safe-subset cases + the still- rejected non-empty case. Previous rejection test repurposed as the positive `..._allowed_and_cleans_up` variant. All 1581 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix+test: fuzz-style cidx tests + DoS bug fix (capacity-overflow panic) A-grade item 2. Added 4 fuzz-style tests; the verifier-panic- resistance test FOUND A REAL DoS BUG on the first run. THE BUG: GroveDb::verify_count_indexed_top_k / verify_count_indexed_ query bincode-decoded the proof envelope with no length limit. An attacker supplying a crafted byte buffer with a huge Vec length triggered `Vec::with_capacity(huge_n)` which panics with "capacity overflow" — a DoS vector against any process running proof verification on untrusted input. THE FIX: bound the decode with bincode's `with_limit::<16 MiB>()`. THE TESTS: - fuzz_verify_count_indexed_top_k_never_panics_on_arbitrary_bytes - fuzz_verify_count_indexed_query_never_panics_on_arbitrary_bytes - fuzz_prove_verify_round_trip_with_arbitrary_count_ranges - fuzz_large_random_op_sequence_against_cidx Seeded SplitMix64 PRNG with CIDX_FUZZ_SEED env var override. All 1585 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cost regression tests for cidx ops A-grade item 3. Pins down OperationCost shape for each cidx op. Four tests: insert, delete, top_k (read-only), count_range (read- only). Read ops are asserted to NEVER write (storage_cost.added_ bytes == 0); write ops are asserted to have non-trivial seek + hash work. Captures the actual cost via eprintln so a developer reviewing a flagged regression sees the exact shape change. All 1589 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: GroveDb::open_with_cidx_integrity_check A-grade item 4. Opt-in integrity check at database open time. Walks verify_grovedb (cidx H1-A chain check + content-consistency drift check) and returns Err if any cidx primary is inconsistent. Default open() is unchanged (zero overhead). Two tests: clean DB passes, corrupted secondary fails with the right CorruptedData message. All 1591 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test+bench: cidx benchmarks + concurrent stress tests A-grade items 6 + 9. ITEM 6 (benchmarks): grovedb/benches/cidx_benchmark.rs with 4 benchmark groups using criterion. Run with: cargo bench --features minimal --bench cidx_benchmark Benchmarks the top-k by count claim (O(log n + k)) and insert amplification claim from the book chapter against plain CountTree baselines. ITEM 9 (concurrent stress tests): - concurrent_readers_against_populated_cidx_see_consistent_state: passes — 8 readers × 100 top-k queries, all see same state. - concurrent_writers_against_disjoint_cidx_primaries: #[ignore]'d — REVEALS TWO PRE-EXISTING ISSUES in merk/storage: 1. Concurrent writes to SAME cidx primary trigger merk-level panic ("Tried to attach tree with same key") rather than a clean tx-conflict error. 2. Concurrent writes to DISJOINT cidx primaries occasionally produce __cidx_secondary_orphan__ drift despite retry-on- Resource-busy — suggests failed txs leak storage writes OR retry misses conflict variants. Both are NOT cidx-specific (merk + storage atomicity issues). Documented in #[ignore] comment for the merk/storage team follow-up. The test artifact stays so the finding isn't lost. The user's prompt predicted "May reveal real production-load bugs" — the test did exactly that. All 1592 grovedb tests pass + 2 ignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: remove misleading concurrent-writer test (out of scope for grovedb) GroveDB currently supports a single writer as a contract; multi- writer semantics are not claimed. The previous commit (concurrent_writers_against_disjoint_cidx_primaries, #[ignore]'d) framed its panic/drift findings as "pre-existing concurrency issues in merk/storage" — but those aren't bugs against a claimed contract; they're behavior of unsupported scenarios. Keeping the test (even ignored) misleads future readers about what GroveDB is supposed to support. Remove it cleanly. Keep `concurrent_readers_against_populated_ cidx_see_consistent_state` since concurrent reads ARE a supported property. Update the section header to reflect the narrower scope. No code change, just test removal + comment refresh. All 1592 grovedb tests pass + 1 unrelated ignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cidx coverage push — 13 targeted tests for highest-miss areas Codecov patch sat at 80.9% (575 missing lines). 13 new tests targeting the top-miss regions: partial-batch cidx overwrite, root-path rejection on each dedicated API, update-same-count short circuit, count-delta replacement (Item↔CountTree), safe-subset cidx→CountTree overwrite, descending+ascending proof round-trip, bounded-range count_range with sub-cases, inverted-range edge case, descending top-k explicit, atomicity rollback after safe-subset overwrite, non-cidx fallthrough in batch. All 1605 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: 5 audit findings on commit cc4db742 (P1 + P2) [P1] 247-byte ceiling on cidx primary item keys (direct + batch). Secondary keys are (count_be ‖ item_key); Merk requires keys < 256 bytes. Generic batch validation only enforces 255-byte cap, so cidx primaries need an 8-byte stricter ceiling. Added MAX_CIDX_ITEM_KEY_LEN = 247 and validate_cidx_item_key_len. Enforced on insert_into_count_indexed_tree_on_transaction and on execute_ops_on_path when in_tree_type is cidx primary. [P1] insert_into_count_indexed_tree overwrite cleanup. The dedicated API read existing element only for count delta — didn't clean up old tree storage when replacing a tree entry. Mirrors batch safe-subset semantics: - existing tree, new non-tree: ALLOW + cleanup - existing tree, new empty tree (root_key=None): ALLOW + cleanup - existing cidx, new non-empty cidx: REJECT (ambiguous) - existing tree, new non-empty non-cidx tree: REJECT (ambiguous) Cleanup mirrors db.delete/batch DeleteTree: find_subtrees + clear, plus secondary namespace clear for existing-cidx. [P2] Batch consistency check for safe-subset overwrite descendants. When scheduling cidx-overwrite cleanup, scan ops_by_qualified_paths for descendants of the cidx path and reject as InvalidBatchOperation if found. Defense in depth; the audit's worry about silent cleanup-drop is already caught by other checks in most flavors, but the new check provides a clearer error message. [P2] verify_grovedb duplicate-secondary detection. Changed HashMap<Vec<u8>, u64> to HashMap<Vec<u8>, Vec<u64>> so duplicate secondary rows for the same primary key (real drift class) are flagged via __cidx_secondary_duplicate__ sentinel path. [P2] appendix-a.md: updated discriminants 16/17/144/145 → 17/18/ 145/146, added NotSummed wrapper byte row, added 247-byte ceiling note. Now matches code (post-merge with #659). All 1611 grovedb tests pass; 5 new audit-targeted tests added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cidx coverage push — 8 more tests for new overwrite-cleanup branches Codecov patch coverage at 82.3%. Adding 8 targeted tests around the new overwrite-cleanup code paths landed in commit 978dc2d9: - count tree → empty Tree (different non-cidx tree type) - empty count tree → Item - reject non-empty non-cidx tree as new element - batch safe-subset overwrite cidx → empty SumTree - atomicity rollback variant with safe-subset overwrite - explicit 247-byte boundary acceptance in batch path - delete after tree-→-Item overwrite All 1618 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address 8 CodeRabbit review findings on commit 978dc2d9 [1] count-indexed-tree.md: replace nonexistent verify_aggregate_count_query_on_secondary example with the actual shipped verify_count_indexed_query API using a MerkQuery range. [2] lib.rs open_with_cidx_integrity_check: doc no longer overstates narrowness; matches the underlying verify_grovedb full-walk behavior. Error message updated. [3] operations/insert/mod.rs: reject partially-initialized cidx claims on direct insert. Non-empty cidx (count > 0 OR either root is Some) now requires BOTH root keys to be Some(_). [4] operations/proof/count_indexed.rs: guard zero-layer envelope — previously underflowed at `last_idx = len - 1`. Same DoS class as the capacity-overflow fuzz-found fix. [5] operations/proof/generate.rs + verify.rs: empty cidx primary handling. Generate gates descent on Some(_) primary; verify adds empty-cidx terminal check using combine_hash_three(H, NULL, NULL) instead of the regular-tree combine_hash. [6] operations/delete/mod.rs: nested cidx secondary cleanup in the find_subtrees loop (idempotent on non-cidx subtrees). [7] grovedb-element/src/element/mod.rs: ElementShadow gains CountIndexedTree / ProvableCountIndexedTree variants for serde Deserialize round-trip. [8] batch/mod.rs: apply_unchecked flags-update arm now includes CountIndexedTree / ProvableCountIndexedTree so flags changes recompute the stored layered value cost. Plus 3 nitpicks: - v1_proof_tests: results.len() == 3 instead of non-empty - element_type tests: is_tree() + is_count_indexed_tree() coverage - benches: remove no-op bench_plain_count_tree_top_k Made operations::proof::count_indexed module public for tests. Added docs on previously-undocumented requested_limit/descending fields. 5 new audit-fix tests. All 1622 grovedb tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: is_non_counted() catches cidx NonCounted twins (145, 146) The is_tree() test I added for NonCountedCountIndexedTree exposed a real bug in is_non_counted() that was hidden until cidx variants introduced discriminants in the 0x90 upper-nibble range: - NonCountedItem (128) = 0x80 — upper nibble 0x80 ✓ - NonCountedDenseTree (142) = 0x8E — upper nibble 0x80 ✓ - NonCountedCountIndexedTree (145) = 0x91 — upper nibble 0x90 ✗ - NonCountedProvableCountIndexedTree (146) = 0x92 — upper nibble 0x90 ✗ The previous `(disc & 0xf0) == 0x80` check missed the two cidx twins, so: - `is_non_counted()` returned false on them - `base()` fell through to `self` instead of stripping 0x80 - `is_tree()` (which uses `self.base()`) returned false on the NonCounted cidx twins The fix gates on bit 7 set AND below the NotSummed prefix (0xb0), which catches everything in [128, 175] — covering both the existing twins and the new cidx ones, while excluding the NotSummed range. Caught by the workspace test that was failing in CI (Ubuntu 1/3) — the unit test for grovedb-element/element_type that I added in the previous round (`is_tree()` cidx coverage) was the exact thing that surfaced the bug. Workspace now passes 3004 tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: insert_into_count_indexed_tree rejects non-empty tree/cidx claims unconditionally Codex re-review finding: the non-empty-tree and non-empty-cidx rejection added in commit 978dc2d9 was gated inside the `existing_is_tree` branch. So a brand-new item_key, or replacing an existing non-tree (e.g. Item) with a tree/cidx claim carrying root keys, slipped through. The merk insert then wrote NULL_HASH child roots while the serialized element preserved the supplied root keys — persisting an inconsistent chain. Fix: lift the new-element validation OUT of the existing_is_tree branch so it runs unconditionally before any merk write: - Element::CountIndexedTree / ProvableCountIndexedTree with any of {primary_root_key.is_some(), secondary_root_key.is_some(), count_value != 0}: REJECT. - Element::Tree(Some(_)) / SumTree(Some(_)) / BigSumTree(Some(_)) / CountTree(Some(_)) / CountSumTree(Some(_)) / ProvableCountTree(Some(_)) / ProvableCountSumTree(Some(_)): REJECT. - Anything else: allow (the existing cleanup path runs if overwriting a tree). Error message points callers at generic db.insert (which validates root keys against on-disk state by opening the claimed Merks) for the cases where they really do mean to point a cidx at existing on-disk data. 3 new tests cover brand-new key + cidx replacing item: - insert_into_count_indexed_tree_rejects_non_empty_cidx_on_brand_new_key - insert_into_count_indexed_tree_rejects_non_empty_tree_on_brand_new_key - insert_into_count_indexed_tree_rejects_non_empty_cidx_replacing_item Existing overwrite-rejection tests updated to accept the new error message wording. All 3007 workspace tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(clippy): collapse non-empty-cidx match into guard pattern clippy::collapsible-match flagged the nested `if p.is_some() || s.is_some() || *c != 0` inside the cidx variant arm. Convert to a match guard pattern with the same semantics — purely stylistic, no behavior change. All 129 cidx tests still pass; clippy --workspace --all-features clean under -D warnings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cidx): bind caller intent in verifiers; reject oversize primary keys Addresses two audit findings from PR #657 comment 4422941255. P1 — cidx proof verification was trusting envelope-supplied query semantics. A malicious prover could answer a (k=10, descending=true) request with a valid (k=5, descending=false) proof reconstructing the same root via the same path but exposing different content. - CountIndexedRangeProof.requested_limit: u16 -> Option<u16>; encoding no longer conflates None with Some(0). - verify_count_indexed_top_k(proof, path, expected_k, expected_descending): authenticates envelope against caller intent, rejects mismatches with CorruptedData("...limit/direction mismatch..."). - verify_count_indexed_query(proof, query, expected_limit, path): same treatment; direction also re-derived from secondary_query.left_to_right for symmetry. P2 — reconcile_count_indexed_tree_secondary_on_transaction iterated primary entries with no length check, so a legacy/corrupt/externally- injected primary key > 247 bytes would drive make_secondary_key to synthesize a >= 256-byte secondary key, violating Merk's < 256 invariant. - Reconcile validates each primary key length before secondary synthesis and fails closed with CorruptedData. - verify_grovedb's cidx walk records a `__cidx_primary_key_oversize__` sentinel for oversize primary keys with the actual length encoded in the diagnostic hash slot. Tests cover both code paths: - verify_count_indexed_top_k rejects wrong expected_descending and wrong expected_k. - verify_count_indexed_query rejects wrong expected_limit. - verify_count_indexed_query distinguishes None from Some(0). - reconcile_rejects_oversized_primary_key. - verify_grovedb_flags_oversized_primary_key. - corrupt_primary_insert_helper sanity test for the injection helper. All ~15 existing call sites of the verifier functions updated to pass matching expected_k/expected_descending/expected_limit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cidx): clear secondary on empty cidx delete; bench unwrap context Addresses CodeRabbit review comments on PR #657. 1) delete/mod.rs (outside-diff Major): the cidx secondary cleanup at the end of `delete_internal`'s tree branch lived inside the `if !is_empty { ... }` block. When deleting an EMPTY cidx primary that had a drifted secondary (e.g. an orphan injected by a bug that mirrored deletes into the primary but failed to mirror into the secondary), the secondary namespace was left untouched on delete, leaving stale entries that could collide with a later cidx recreation at the same path. Hoist the explicit `if tree_type.is_count_indexed_primary()` clear out of the `if !is_empty` gate so it runs unconditionally on every cidx primary delete. Remove the now-redundant copy that lived inside the non-empty branch; the per-prefix cleanup inside the `find_subtrees` loop still handles nested cidx primaries and the hoisted block re-handles the target (both clears are idempotent on empty namespaces — intentional defense-in-depth, documented in the comment). Regression test `direct_delete_empty_cidx_with_drifted_secondary_clears_namespace` injects an orphan into the secondary of an empty cidx, deletes the cidx, and verifies the secondary namespace is empty via a raw RocksDB scan over the S2-B prefix. Verified to fail on the pre-fix code path. 2) cidx_benchmark.rs (Major nit + several nitpicks): replace bare `.unwrap()` on TempDir/GroveDb/insert and inner measurement calls with `.expect("...")` carrying a descriptive context message. A panic in the bench harness now surfaces which operation failed rather than just a backtrace-only "called Option::unwrap on a None". Applied consistently across populate_cidx, populate_plain_count_tree, bench_cidx_top_k, bench_insert_into_cidx, and bench_insert_into_plain_count_tree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(cidx): add 12 coverage tests for under-tested cidx code paths Raises count_indexed_tree.rs coverage 86.1% → 92.2% and proof/count_indexed.rs coverage 84.1% → 87.9% (locally measured via cargo-llvm-cov on the grovedb crate's test suite). Targets ~80 newly covered lines on PR-introduced code, which directly improves the codecov patch-coverage metric for #657. New tests (all 12 passing; full cidx suite 147 tests green): direct_insert path: - direct_insert_into_cidx_overwrites_nested_cidx_entry_and_cleans_secondary Covers count_indexed_tree.rs:407-428 (existing_is_cidx overwrite-cleanup branch). direct_delete path: - direct_delete_from_cidx_removes_nested_cidx_entry_and_cleans_secondary Covers count_indexed_tree.rs:1412-1438 (deleted_was_cidx_primary branch in delete_from_count_indexed_tree). reconcile loops: - reconcile_repairs_missing_secondary_entry_via_insert_loop Covers the insert loop at count_indexed_tree.rs:927-937 by deleting a real mirror first. - reconcile_removes_orphan_secondary_entry_via_delete_loop Covers the delete loop at count_indexed_tree.rs:909-919 by injecting an orphan. - reconcile_errors_on_undecodable_element_bytes_in_primary Covers the raw_decode error path (842-845) by writing garbage bytes directly to the primary's storage namespace via StorageContext::put. secondary key drift / query error paths: - count_indexed_top_k_errors_on_short_secondary_key_drift Covers count_indexed_tree.rs:1061-1065, 1750. - count_indexed_count_range_errors_on_short_secondary_key_drift Covers count_indexed_tree.rs:1160-1164 plus the unbounded-upper branch (lo=0, hi=u64::MAX) of the range builder. - count_indexed_count_range_returns_empty_when_lo_greater_than_hi Covers the lo>hi early-return at 1096-1098. proof verifier error paths: - verify_count_indexed_top_k_rejects_proof_with_short_secondary_key_drift Covers proof/count_indexed.rs:540-545 (verifier rejects < 8-byte proved key). - verify_count_indexed_top_k_rejects_tampered_primary_root_hash Covers H1-A cidx-layer chain mismatch at 566-572 (envelope tampering). - verify_count_indexed_top_k_rejects_tampered_intermediate_layer_proof Covers shallower-layer chain mismatch at 600-613. - verify_count_indexed_top_k_rejects_tampered_secondary_root_hash_via_query Covers ancestor_cidx_secondary_root_hashes length-mismatch at 580-585 (via a nested-cidx layout so last_idx > 0). Remaining uncovered lines are dominated by defensive error closures (.map_err(|e| Error::CorruptedData(format!(...))) bodies that only fire on storage-level failures) and unreachable!() / CorruptedCodeExecution guards that require contradictory state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(cidx): 7 more coverage tests targeting patch-coverage gaps Codecov patch coverage was 83.75% on the previous head (`15260967`). develop's commit #660 raised the patch-coverage floor from 80% to 90%, so the codecov/patch check fails. This commit adds 7 tests targeting the two biggest under-covered patch areas: 1. Direct cidx insertion non-empty path (insert/mod.rs:314-410). The migration / restore-from-backup path for inserting a CountIndexedTree element directly via db.insert(...) with concrete primary/secondary root_keys was at 43% patch coverage. Five new tests: - direct_insert_non_empty_cidx_with_matching_roots_succeeds - direct_insert_partial_cidx_with_one_root_none_rejected (covers (Some, None), (None, Some), (None, None, count>0)) - direct_insert_cidx_with_mismatched_primary_root_key_rejected - direct_insert_cidx_with_mismatched_secondary_root_key_rejected - direct_insert_provable_count_indexed_tree_with_matching_roots_succeeds (covers the ProvableCountIndexedTree arm of the same pattern) Coverage of insert/mod.rs jumped 93.0% -> 96.2% locally. 2. V1 proof verifier cidx-error branches (proof/verify.rs:540-602). Two new tampering tests build a valid V1 cidx-subquery proof, decode the GroveDBProof envelope, mutate the cidx sublayer, and re-encode: - v1_verify_rejects_cidx_subquery_proof_with_non_cidx_lower_layer_bytes Covers 547-553 (lower_layer.merk_proof must be ProofBytes::CountIndexedTree). - v1_verify_rejects_cidx_subquery_proof_with_short_cidx_bytes Covers 555-561 (cidx_bytes must be >= 32 bytes for the secondary_root attestation prefix). Full cidx suite: 154/154 passing. Full grovedb library: 1665/1665. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(cidx): 4 more coverage tests for verify and propagate paths Codecov patch coverage was 85.41% on a0185f45. Develop's #660 requires 90%. This commit adds 4 more tests targeting: 1. proof/count_indexed.rs verifier branches (472-494, 534-537): - verify_count_indexed_query_rejects_wrong_expected_descending (the _query variant of the direction-mismatch reject; the _top_k variant is already covered). - verify_count_indexed_top_k_rejects_proof_with_layer_count_mismatch (env.layer_proofs.len() != path.len()). - verify_count_indexed_top_k_rejects_proof_with_corrupted_secondary_proof (envelope's secondary_proof bytes replaced with garbage so execute_proof errors). proof/count_indexed.rs locally moved 89.6% -> 90.2%. 2. lib.rs cidx cascading aggregation propagation path (lib.rs:840-998): - deep_insert_under_triple_nested_cidx_propagates_all_levels A 3-level cidx layout (outer/middle/inner cidx) with a leaf CountTree at the bottom; a single item insert bubbles count updates through three cidx secondaries. Full cidx suite: 158/158 passing. Full grovedb lib: 1669/1669. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(cidx): 8 more coverage tests for query + V1 proof terminal paths Codecov patch coverage was 85.62% on 3b9ffbf7. Targeting the remaining ~143 lines needed for 90% threshold: V1 proof terminal cidx path (proof/verify.rs is_empty_cidx block): - v1_proof_round_trips_for_empty_cidx_terminal_query Queries an empty CountIndexedTree as a terminal (no subquery). Forces the combine_hash_three(H(value), NULL_HASH, NULL_HASH) check used by V1 for empty-cidx parent elements. - v1_proof_round_trips_for_provable_empty_cidx_terminal_query Same shape for ProvableCountIndexedTree. - v1_proof_query_with_limit_terminates_early_at_cidx_subquery limit_left == Some(0) break inside the cidx subquery handler (proof/verify.rs around 521 and 604). count_indexed_tree.rs query paths: - count_indexed_top_k_descending_returns_largest_counts_first Exercises the descending top-k scan path; populates a cidx with varied count_values and checks ordering. - count_indexed_count_range_filters_to_inclusive_band Concrete (lo, hi) bounds (not the lo=0, hi=u64::MAX case), exercises the Some(upper_bytes) branch of the range builder. - count_indexed_count_range_with_limit_cuts_short Limit-respecting path in the range scan. - cidx_top_k_with_k_larger_than_entries_returns_all Iterator-exhausted termination branch. Batch wrapper path: - batch_insert_non_counted_wrapped_into_count_indexed_tree Inserts a NonCounted-wrapped CountTree into a cidx primary via apply_batch. Exercises the wrapper-element handling in the cidx propaga…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
Symmetric counterpart to Element::NonCounted (#654). Where NonCounted opts a child out of count propagation in count-bearing trees, NotSummed opts a sum-bearing subtree out of sum propagation in sum-bearing trees.
Use case: a SumTree of category subtotals, where one inner sum-tree should be informational only — it tracks its own sum but does not bubble up to the parent total.
What was done?
Added a new
Element::NotSummed(Box<Element>)variant that:SumTree,BigSumTree,CountSumTree,ProvableCountSumTree),SumTree,BigSumTree,CountSumTree,ProvableCountSumTree).Design choice: stricter inner-type whitelist than NonCounted
NonCounted can wrap any element (including items, references, all tree types). NotSummed restricts the inner to the four sum-tree variants because that is where the wrapper is semantically meaningful — a sum-bearing subtree that retains its own internal sum but contributes nothing to the parent. Wrapping items or non-sum trees would have no effect distinct from using the bare type.
The constructor
Element::new_not_summedreturnsResultand rejects everything else withInvalidInput. Serialization and deserialization enforce the same whitelist, plus an O(1) two-byte pre-check that rejects all four wrapper-on-wrapper combinations ([15,15],[15,16],[16,15],[16,16]) before bincode can recurse — closing a stack-exhaustion vector.Discriminant scheme
Element)ElementTypeflag bit0x800x40The disjoint flag bits keep
is_non_counted()andis_not_summed()independently testable onElementType.base()strips whichever flag is set. The two wrappers are mutually exclusive in practice — constructors and (de)serializers reject any nesting, including cross-nestingNotSummed(NonCounted(_))andNonCounted(NotSummed(_)).Cryptographic notes
The wrapper byte is part of the serialized value, so the value hash distinguishes
NotSummed(X)from a bareX. The parent'sTreeFeatureTypefor a not-summed child is the inner element's feature type with its sum component zeroed (via a newTreeFeatureType::zero_sum()helper, symmetric tozero_count()); no newTreeFeatureTypevariants are introduced.Scope
Touched 27 files across
grovedb-element,grovedb-query,grovedb-merk, andgrovedb. Mirrors PR #654's wiring throughout the stack:TreeType::is_sum_bearing(), insert validation, feature-type dispatch, get/cost paths, reconstruct preserves wrappernot_summedflag alongsidenon_countedinInsertTreeWithRootHash, parent-type guard, estimated cost overhead, debugger transparencytests/{batch_unit,batch_coverage,batch_rejection}_tests.rsupdated for the new fieldHow Has This Been Tested?
New tests:
is_*,sum_value_or_default → 0,count_value_or_defaultstill propagates,count_sum_value_or_default → (count, 0), flag accessors); bincode round-trip; reject nested wrappers at deserialize and at serialize; long-chain pre-check stack-overflow guard.NOT_SUMMED_WRAPPER_DISCRIMINANT = 16and the four0x40 | basetwin discriminants; rejects all illegal inner bytes including the wrapper bytes themselves and the synthetic twin range.NormalTreeandCountTreeis rejected; insert intoSumTreesucceeds and contributes 0;NotSummed(SumTree(_,100,_))in aProvableCountSumTreeyields count=1 sum=0;reconstruct_with_root_keypreserves the wrapper through propagation; constructor rejects non-sum-tree inner.tests/not_summed_tests.rs, 5 tests): batch insert rejected into NormalTree and CountTree; direct insert in SumTree excludes subtree sum from parent aggregate; batch propagation preserves wrapper throughInsertTreeWithRootHash(with both wrapper insert AND child insert in the same batch);check_subtree_existsworks through the wrapper.Full workspace test suite passes:
cargo test -p grovedb-element— 74 passed (57 unit + 17 integration)cargo test -p grovedb-query— 235 passedcargo test -p grovedb-merk --features minimal,test_utils,full— 416 passedcargo test -p grovedb— 1495 passedBreaking Changes
None for existing callers. The change adds a new
Elementvariant and is additive at the API surface. The on-disk serialization format gains bincode discriminant 16 — older code reading data written with this variant would fail to deserialize, so producer/consumer pairs need to be upgraded together. This matches the existing "ONLY APPEND TO THIS LIST" comment on theElementenum.The internal
GroveOp::InsertTreeWithRootHashvariant gains anot_summed: boolfield; the variant is#[non_exhaustive]and only constructed inside this crate, but the four in-tree test sites that build it directly are updated.Checklist:
🤖 Generated with Claude Code