feat: replace indexed secondary rows with canonical references

docs/book/src/count-indexed-tree.md

@@ -152,19 +152,51 @@ The secondary Merk holds one entry per element in the primary, keyed by:
 
 ```text
 secondary_key = count_be_bytes(8) ‖ original_key
-secondary_val = ()        // empty; the original_key is encoded in the key
+secondary_val = ReferenceWithSumItem(
+                    SiblingReference(original_key),
+                    max_reference_hop = Some(1),
+                    sum = count_value,
+                )
 ```
 
 - **`count_be_bytes`** is the element's `count_value` encoded big-endian, 8
   bytes. Big-endian gives natural numeric order under lexicographic
   comparison, so right-to-left iteration yields highest-count-first.
 - **`original_key`** is appended to break ties among elements with equal
-  counts and to make each secondary key unique and reversible.
+  counts and to make each secondary key unique and reversible. It stays in the
+  key because ordering, uniqueness, and boundary decoding must not depend on
+  resolving the row value.
 
-The secondary Merk uses node feature type `ProvableCountedMerkNode(1)` —
-every entry contributes a count of `1`, so the aggregated count at the
-secondary's root equals the total number of indexed entries (which also
-equals the number of entries in the primary).
+#### Canonical reference rows
+
+Every secondary row is a canonical one-hop combined reference back to its
+primary entry. Its committed value hash is:
+
+```text
+combine_hash(H(reference bytes), primary_node_committed_value_hash)
+```
+
+The binding is deliberately to the immediate primary node, not directly to a
+terminal value reached through another reference. This keeps refresh local:
+every operation that rewrites the primary entry also refreshes its rows. Reads
+then apply ordinary GroveDB reference semantics and return the terminal value.
+This immediate binding is dedicated indexed-tree behavior; ordinary user
+references retain their existing terminal-reference rules.
+
+Because the row binds the primary commitment, value-only updates and deep
+subtree-root changes rewrite the row even when its ordering aggregates stay
+unchanged. This write amplification is intentional and included in cost
+tracking.
+
+`SiblingReference` keeps row size independent of grove depth. The secondary's
+physical prefix (`blake3(primary_prefix ‖ axis_tag)`) is not a GroveDB path, so
+the row is interpreted with the indexed primary as its purpose-built logical
+origin rather than by manufacturing a fake `SubtreePath`.
+
+The secondary Merk uses node feature type
+`ProvableCountedAndProvableSummedMerkNode(1, count_value)`. Every row therefore
+contributes count `1`, while its carried sum preserves the count band's total
+as an authenticated scalar.
 
 The reason the secondary is a *provable* count tree (rather than the
 simpler `BasicMerkNode`) is that this lets the existing
@@ -426,8 +458,8 @@ Merk is not touched. The verifier receives the primary's root hash plus a
 ### Top-k by count
 
 ```rust
-// Shipped API on `GroveDb`:
-let entries: Vec<(u64, Vec<u8>)> = db
+// Public API on `GroveDb`:
+let entries: Vec<IndexedAxisEntry<u64>> = db
     .indexed_count_top_k(path, k, /* descending: */ true, transaction, grove_version)?
     .expect("top-k");
 
@@ -436,14 +468,11 @@ let proof_bytes = db
     .prove_indexed_count_top_k(path, k, /* descending: */ true, transaction, grove_version)?
     .expect("prove");
 let result = GroveDb::verify_indexed_count_top_k(&proof_bytes, &path, k)?;
-// result.entries: Vec<(u64, Vec<u8>)>, result.root_hash: [u8; 32]
+// result.entries: Vec<IndexedAxisEntry<u64>>, result.root_hash: [u8; 32]
 ```
 
-The query returns `(count, key)` pairs. To resolve a primary value the
-caller follows up with `db.get(path, key, ...)`; the dedicated proof
-shape carries only the secondary range proof + a 32-byte attestation
-of the primary's root hash. Workloads that don't need values
-(leaderboards, ranking views) pay nothing for data they wouldn't read.
+Each entry contains `ordering_value`, `primary_key`, and the resolved terminal
+`Element`; callers do not perform a second primary lookup per result.
 
 Internally:
 
@@ -453,19 +482,24 @@ Internally:
 3. Run a **descending range query** with `limit = k` over the full
    secondary keyspace. This yields the k highest-count entries, with a
    standard Merk range proof.
-4. *(only if `resolve_values: true`)* For each `(c_be ‖ k)` in the
-   result, open the **primary** Merk and query for `k`. Each resolution
-   is one extra Merk read with one extra Merk inclusion proof.
-
-The default keeps the proof minimal: secondary range proof + a 32-byte
-attestation of the primary's root hash. Workloads that don't need the
-values (leaderboards, ranking views, "top N usernames") pay nothing for
-data they wouldn't read.
+4. Resolve each row and attach a compact target-shape witness. The verifier
+   reconstructs the immediate primary commitment directly from the target
+   bytes and shape data, then checks it against the hash already committed by
+   the secondary row. Reference-shaped primaries carry a bounded chain to the
+   terminal value; nodes outside the indexed primary retain ordinary root
+   authentication.
+
+For ordinary direct primary values, the target witness does not repeat a
+GroveDB inclusion proof per row. The canonical row's combined-reference hash is
+the authentication anchor, so proof growth is the resolved value plus its shape
+commitment rather than another root-to-primary path. A primary that is itself a
+reference pays for authentication only after its chain leaves that immediate
+row binding.
 
 ### Range by count
 
 ```rust
-let entries: Vec<(u64, Vec<u8>)> = db
+let entries: Vec<IndexedAxisEntry<u64>> = db
     .indexed_count_range(
         path,
         min,                       // u64, inclusive
@@ -601,31 +635,38 @@ existing GroveDB layer proofs, with these additions:
 graph TD
     L0["Layer proof: root → … → CountIndexedTree element<br/><i>standard, unchanged</i>"]
     EL["Element bytes: (primary_root_key, secondary_root_key, count_value, flags)<br/>actual_value_hash = Blake3(varint(len) || element_bytes)"]
-    L1A["Primary Merk proof<br/><i>only if primary values were touched</i>"]
+    PR["Primary root hash attestation"]
     L1B["Secondary Merk range proof<br/><i>over (count_be ‖ key) keys</i>"]
+    TW["Per-row compact target witness<br/><i>value bytes + commitment shape; root proofs only after reference hops</i>"]
+    ROW["Canonical secondary row<br/>binds H(reference bytes) + immediate primary commitment"]
     COMB["combined_value_hash = Blake3(actual_value_hash || primary_root_hash || secondary_root_hash)<br/><i>order is primary, then secondary</i>"]
 
     L0 --> EL
-    EL --> L1A
+    EL --> PR
     EL --> L1B
-    L1A --> COMB
+    L1B --> ROW
+    TW --> ROW
+    PR --> COMB
     L1B --> COMB
 ```
 
 Verifier obligations:
 
 - Parent layer verifies the element bytes (carrying both root keys) up to
   the GroveDB root.
-- Each Merk proof produces its own root hash (`primary_root_hash` and/or
-  `secondary_root_hash`).
+- The secondary Merk proof produces `secondary_root_hash`; the envelope carries
+  the untouched `primary_root_hash` attestation already committed by the outer
+  indexed-tree element.
+- Every returned row is checked for its canonical reference bytes and binds the
+  immediate primary commitment reconstructed from its target witness.
 - The verifier reconstructs `combined_value_hash` from
   `actual_value_hash`, `primary_root_hash`, `secondary_root_hash` (in
   that order) and checks it matches the value hash committed in the
   parent layer.
 
-Both root hashes must be made available to the verifier — when a query
-touches only one of the two trees, the proof carries the *other* tree's
-root hash as a 32-byte attestation (it is hashed but not traversed).
+Both root hashes must be available to the verifier. The primary root is hashed
+but not traversed for direct rows; the secondary row itself authenticates their
+immediate primary commitments.
 
 ## When to use which element type
 

grovedb-element/src/element/mod.rs

@@ -293,11 +293,13 @@ pub enum Element {
     /// `ProvableCountedAndProvableSummedMerkNode` (both count AND sum
     /// baked into node hash) and carries a TLV list of 1..=3 secondary
     /// Merks — one per selected axis (count, sum, avg). Each secondary
-    /// lives at its own derived storage prefix; the count axis is a
-    /// `ProvableCountTree` while the sum and avg axes are
-    /// `ProvableCountProvableSumTree`s, so every axis carries a
-    /// hash-bound count (enabling count-bound offset pagination) and
-    /// the sum/avg axes can additionally produce sum-on-range proofs.
+    /// lives at its own derived storage prefix and is a
+    /// `ProvableCountProvableSumTree`, so every axis carries a
+    /// hash-bound count (enabling count-bound offset pagination). Its
+    /// canonical rows are `ReferenceWithSumItem` values that point to
+    /// the corresponding primary key and bind the immediate primary
+    /// node's committed value hash; sum/avg axes can additionally
+    /// produce sum-on-range proofs.
     ///
     /// Fields: `(primary_root_key, count_value, sum_value, axes, flags)`
     /// - `primary_root_key`: root key of the primary

grovedb-element/src/indexed/mod.rs

@@ -17,7 +17,7 @@ pub use sort_keys::{
     encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key, AVG_FIXED_POINT_SCALE,
 };
 
-use crate::error::ElementError;
+use crate::{error::ElementError, reference_path::ReferencePathType, Element};
 
 /// Axis tag for a `ProvableCountProvableSumIndexedTree` secondary entry.
 ///
@@ -50,6 +50,32 @@ pub type IndexedTreeAxisEntry = (u8, Option<Vec<u8>>);
 /// encoding.
 pub type IndexedTreeAxes = Vec<IndexedTreeAxisEntry>;
 
+/// Build the canonical secondary row for an indexed-tree axis.
+///
+/// The row is always a one-hop sibling reference to the primary key. Its
+/// explicit sum preserves the secondary's dual count/sum aggregates.
+pub fn canonical_axis_reference(
+    axis: IndexAxis,
+    primary_key: &[u8],
+    count: u64,
+    sum: i64,
+) -> Result<Element, ElementError> {
+    let axis_sum = match axis {
+        IndexAxis::Count => i64::try_from(count).map_err(|_| {
+            ElementError::CorruptedData(format!(
+                "count value {count} exceeds i64::MAX and cannot be mirrored into an indexed \
+                 count-axis secondary"
+            ))
+        })?,
+        IndexAxis::Sum | IndexAxis::Avg => sum,
+    };
+    Ok(Element::new_reference_with_sum_item_with_hops(
+        ReferencePathType::SiblingReference(primary_key.to_vec()),
+        Some(1),
+        axis_sum,
+    ))
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

grovedb-element/src/lib.rs

@@ -10,7 +10,7 @@ pub mod reference_path;
 pub(crate) mod visualize_helpers;
 
 pub use indexed::{
-    compute_avg_fixed_point, decode_avg_sort_key, decode_count_sort_key, decode_sum_sort_key,
-    encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key, IndexAxis, IndexedTreeAxes,
-    IndexedTreeAxisEntry, AVG_FIXED_POINT_SCALE,
+    canonical_axis_reference, compute_avg_fixed_point, decode_avg_sort_key, decode_count_sort_key,
+    decode_sum_sort_key, encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key,
+    IndexAxis, IndexedTreeAxes, IndexedTreeAxisEntry, AVG_FIXED_POINT_SCALE,
 };

grovedb/src/batch/estimated_costs/average_case_costs.rs

@@ -2343,7 +2343,7 @@ mod tests {
     }
 
     /// The ≥-actual contract must hold for the MULTI-axis variant, not just
-    /// PCIT. The sum and avg axes' secondary rows (`SumItem`,
+    /// PCIT. The sum and avg axes' secondary rows (`ReferenceWithSumItem`,
     /// `ItemWithSumItem`) are larger than the count axis's empty `Item`;
     /// sizing all three as an empty item put the PCPSIT estimate ~25 bytes
     /// per key UNDER actual `added_bytes` — and the PCIT-only test above

grovedb/src/batch/indexed_tree/mirror.rs

@@ -22,9 +22,9 @@ use grovedb_merk::{
 use grovedb_storage::StorageContext;
 use grovedb_version::version::GroveVersion;
 
-use super::{read_entry_aggregates, AggregatePair, AggregateTransition};
+use super::{read_entry_state, AggregateTransition, MaybeEntryState};
 use crate::{
-    operations::indexed_tree::{axis_row_payload, make_axis_secondary_key},
+    operations::indexed_tree::{axis_row_reference, make_axis_secondary_key},
     Element, Error,
 };
 
@@ -63,17 +63,17 @@ fn enforce_axis_item_key_bound<'a>(
 /// what makes each axis's assembled batch deterministic.
 pub(crate) fn read_post_apply_transitions<'db, S: StorageContext<'db>>(
     primary_merk: &Merk<S>,
-    pre: &BTreeMap<Vec<u8>, AggregatePair>,
+    pre: &BTreeMap<Vec<u8>, MaybeEntryState>,
     grove_version: &GroveVersion,
 ) -> CostResult<Vec<AggregateTransition>, Error> {
     let mut cost = OperationCost::default();
     let mut transitions: Vec<AggregateTransition> = Vec::with_capacity(pre.len());
-    for (key, old_aggregates) in pre {
-        let new_aggregates = cost_return_on_error!(
+    for (key, old_state) in pre {
+        let new_state = cost_return_on_error!(
             &mut cost,
-            read_entry_aggregates(primary_merk, key, "post", grove_version)
+            read_entry_state(primary_merk, key, "post", grove_version)
         );
-        transitions.push((key.clone(), *old_aggregates, new_aggregates));
+        transitions.push((key.clone(), *old_state, new_state));
     }
     Ok(transitions).wrap_with_cost(cost)
 }
@@ -92,31 +92,31 @@ fn build_axis_mirror_batch(
     let mut cost = OperationCost::default();
     let secondary_tree_type = crate::operations::indexed_tree::axis_secondary_tree_type(axis);
     let mut secondary_batch: Vec<BatchEntry<Vec<u8>>> = Vec::with_capacity(transitions.len() * 2);
-    for (key, old_aggregates, new_aggregates) in transitions {
-        let entry_for = |aggregates: &AggregatePair| -> Result<_, Error> {
-            aggregates
-                .map(|(c, s)| {
-                    Ok((
-                        make_axis_secondary_key(axis, c, s, key),
-                        axis_row_payload(axis, c, s)?,
-                    ))
-                })
-                .transpose()
-        };
-        let old_entry = cost_return_on_error_no_add!(cost, entry_for(old_aggregates));
-        let new_entry = cost_return_on_error_no_add!(cost, entry_for(new_aggregates));
-        if old_entry == new_entry {
+    for (key, old_state, new_state) in transitions {
+        if old_state == new_state {
             continue;
         }
+        let old_key =
+            old_state.map(|state| make_axis_secondary_key(axis, state.count, state.sum, key));
+        let new_entry = new_state
+            .map(|state| {
+                Ok((
+                    make_axis_secondary_key(axis, state.count, state.sum, key),
+                    axis_row_reference(axis, key, state.count, state.sum)?,
+                    state.value_hash,
+                ))
+            })
+            .transpose();
+        let new_entry = cost_return_on_error_no_add!(cost, new_entry);
         // Delete the old row ONLY if the new one lands on a different key.
         // On the avg axis a change can alter the payload while leaving the
         // sort key fixed — (count, sum) going (1, 5) -> (2, 10) keeps
         // avg = 5 — and emitting a delete plus a put for one key in a single
         // Merk batch is rejected outright ("Keys in batch must be unique"),
         // failing the whole GroveDB batch. Where the key is unchanged the put
         // alone overwrites the payload, which is what the row needs.
-        let new_secondary_key_ref = new_entry.as_ref().map(|(key, _)| key);
-        if let Some((old_secondary_key, _)) = &old_entry
+        let new_secondary_key_ref = new_entry.as_ref().map(|(key, ..)| key);
+        if let Some(old_secondary_key) = &old_key
             && Some(old_secondary_key) != new_secondary_key_ref
         {
             cost_return_on_error!(
@@ -131,7 +131,7 @@ fn build_axis_mirror_batch(
                 .map_err(Error::MerkError)
             );
         }
-        if let Some((new_secondary_key, entry)) = new_entry {
+        if let Some((new_secondary_key, entry, target_hash)) = new_entry {
             let feature_type = cost_return_on_error_no_add!(
                 cost,
                 entry
@@ -141,8 +141,9 @@ fn build_axis_mirror_batch(
             cost_return_on_error!(
                 &mut cost,
                 entry
-                    .insert_into_batch_operations(
+                    .insert_reference_into_batch_operations(
                         new_secondary_key,
+                        target_hash,
                         &mut secondary_batch,
                         feature_type,
                         grove_version,