Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
852c5ef
feat(drive): boolean HAVING range queries on ranked index axes
QuantumExplorer Aug 12, 2026
25cce54
test(drive): having-range proof round-trip with identifier group keys
QuantumExplorer Aug 12, 2026
1134037
test(drive): pin single-property group_by contract on the having path
QuantumExplorer Aug 12, 2026
0f8068e
feat(drive): per-prefix ranked aggregates on compound indexes
QuantumExplorer Aug 13, 2026
9e72bdc
fix(drive): exact float AVG bound translation and honest having conti…
QuantumExplorer Aug 13, 2026
f1bfabc
Merge remote-tracking branch 'origin/claude/having-evaluation-feasibi…
QuantumExplorer Aug 13, 2026
0c80416
docs(dapi-grpc): document the PV14 having-range grammar on the wire
QuantumExplorer Aug 13, 2026
3ad095f
Merge remote-tracking branch 'origin/claude/having-evaluation-feasibi…
QuantumExplorer Aug 13, 2026
c418afc
docs(dapi-grpc): wire docs cover compound-index equality pins
QuantumExplorer Aug 13, 2026
39e14e5
docs(dapi-grpc): having-range ORDER BY uses the ranked order-key spel…
QuantumExplorer Aug 13, 2026
6ba3d00
Merge remote-tracking branch 'origin/claude/having-evaluation-feasibi…
QuantumExplorer Aug 13, 2026
9ec210b
fix(dpp): satisfy clippy type_complexity on the compound-ranked test …
QuantumExplorer Aug 13, 2026
8a5ca12
Merge remote-tracking branch 'origin/v4.2-dev' into claude/having-eva…
QuantumExplorer Aug 13, 2026
047c95d
Merge remote-tracking branch 'origin/claude/having-evaluation-feasibi…
QuantumExplorer Aug 13, 2026
5dd1671
Merge remote-tracking branch 'origin/v4.2-dev' into claude/gracious-m…
QuantumExplorer Aug 13, 2026
869dfba
fix(drive): adapt post-merge tests to pinned-prefix query APIs
QuantumExplorer Aug 13, 2026
b17dfcf
fix(dpp)!: ranked key ceiling binds only the terminal property; null …
QuantumExplorer Aug 13, 2026
650416f
refactor(dpp): give the ranked prefix-overlap rule its own file
QuantumExplorer Aug 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions book/src/drive/document-ranked-trees.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ One asymmetry is worth knowing when authoring: **the meta-schema demands the lit

Two structural rules, both enforced at contract-parse time in rs-dpp:

- **Single-property indexes only.** `ranked aggregates are only supported on single-property indexes in this protocol version`. Two reasons, both relaxable at a future protocol version. First, a compound index whose *prefix* level also terminates an aggregating index would need its ranked terminal tree wrapped in a `NonCounted` / `NotSummed` shell so it contributes zero to the parent's aggregate — and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard for this is `INDEXED_INNER_UNWRAPPABLE`.) Second, the ranked query surface has no equality-prefix routing: with more than one property there would be a prefix to fix before ranking, and nothing to express it with.
- **No aggregating index on a compound ranked index's full prefix.** Ranked flags are allowed on compound indexes, with **per-prefix** semantics: a ranked `[identityId, class]` puts the indexed tree at each prefix value's terminal `class` property-name level — one ordered secondary per `identityId`, each ranking only that identity's `class` groups. There is deliberately no global cross-prefix ordering; the query surfaces require every leading property to be pinned by an equality `where` clause. The one shape that stays impossible — and is rejected per document type, where all indexes are visible (`validate_no_ranked_prefix_overlap`) — is a countable/summable index terminating at exactly the compound's leading prefix: its aggregating value trees would demand the `NonCounted` / `NotSummed` shell around the ranked terminal tree, and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard behind the parse-time check is `INDEXED_INNER_UNWRAPPABLE`.) Only the exact `n-1` prefix conflicts: an aggregating index at a shorter prefix wraps a plain intermediate tree, and one extending past the ranked terminal lives inside its value trees — both supported.
- **Non-unique indexes only.** `ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rank`. Contested indexes are covered transitively — a contested index is unique by construction, so it hits the same check rather than needing its own.

### Version Gate
Expand Down Expand Up @@ -231,9 +231,9 @@ Every ranked read — and, on the prove path, every ranked proof — is issued a
/ <last_index_property_name: utf-8> // e.g. b"restaurantId"
```

The children of that tree are the *groups*: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a `string` property, its UTF-8 bytes — e.g. `b"alpha"`). The secondary entries a top-k read returns are keyed by those same group keys. A compound index `[a, b]` inserts `<a> / <value_of_a>` between the doctype and the terminal `<b>` level — which is exactly the shape ranked indexes don't support yet.
The children of that tree are the *groups*: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a `string` property, its UTF-8 bytes — e.g. `b"alpha"`). The secondary entries a top-k read returns are keyed by those same group keys. A compound index `[a, b]` inserts `<a> / <encoded pinned value of a>` between the doctype and the terminal `<b>` level — the value segment comes from the request's equality `where` pin on `a`, encoded with the same `serialize_value_for_key` the write path used to key that prefix's value tree, so the walk lands on **that prefix's own** indexed tree and secondary.

Prover and verifier build this path through the same function, `DriveDocumentRankedQuery::indexed_property_name_tree_path`, which is why they agree on the root hash by construction.
Prover and verifier build this path through the same function, `DriveDocumentRankedQuery::indexed_property_name_tree_path` (with the pinned prefix values encoded by the shared resolver, `resolve_ranked_query_for_mode`), which is why they agree on the root hash by construction.

## Write-Path Cost: The Grove v4 Cleanup Gates

Expand Down Expand Up @@ -265,7 +265,7 @@ A demoted `CountSumTree` value tree contributes its `(count, sum)` to a ranked i
- each group's value tree demotes from `ProvableCountProvableSumTree` to `CountSumTree`;
- the `chefId` continuation inside it goes in `Element::NonCounted`, contributing zero to the group's count and sum.

The one place the two changes genuinely collide is the case the single-property rule already forbids: a ranked *terminal* level sitting inside an aggregating value tree would need a wrapper, and an indexed tree can never be wrapped. That is the `INDEXED_INNER_UNWRAPPABLE` guard, and it fails closed.
The one place the two changes genuinely collide is the case the prefix-overlap rule already forbids at contract-parse time: a ranked *terminal* level sitting inside an aggregating value tree would need a wrapper, and an indexed tree can never be wrapped. That is the `INDEXED_INNER_UNWRAPPABLE` guard, and it fails closed.

## Storage-Layout Invariants

Expand Down Expand Up @@ -337,11 +337,11 @@ Note that the fixture puts each shape on its **own document type**. That's not a

| You want | Set |
|---|---|
| Top / bottom K groups by document count | `rankedCountable: true` on a single-property, non-unique index that already has `countable` + `rangeCountable: true` |
| Top / bottom K groups by document count | `rankedCountable: true` on a non-unique index that already has `countable` + `rangeCountable: true` |
| Top / bottom K groups by sum of a property | `rankedSummable: true` on an index with `summable: "<prop>"` + `rangeSummable: true` |
| Top / bottom K groups by average of a property | `rankedAverageable: true` on an index with `averageable: "<prop>"` + `rangeAverageable: true` (or the count+sum longhand) |
| Two rankings on one index (e.g. by count *and* by average) | Both keywords. The tree is a PCPSIT carrying both axes in its TLV; you pay one secondary Merk per axis on every write. |
| A ranking filtered by another property (`top 5 restaurants in London`) | Not available. Ranked indexes are single-property and ranked queries take no `where` clause — the secondary is sorted by aggregate, not by group key, so it cannot express a filtered subset. Model the filter as part of the grouping property, or rank client-side over a range query. |
| A ranking filtered by another property (`top 5 restaurants in London`) | A **compound ranked index** with the filter property leading: `[city, restaurantId]` with the ranked flags. Each city gets its own secondary; the query pins the prefix with an equality `where` (`WHERE city == "London" GROUP BY restaurantId ORDER BY <agg> DESC LIMIT 5`). Only equality pins — a range or `IN` on the prefix is rejected, and there is no cross-prefix (global) ordering on a compound ranked index. |
| A ranking on a unique or contested index | Not available, and not meaningful: every group holds at most one document. |
| Range aggregates without ranking (the 4.0 surface) | Just the `range*` flags. Ranking is strictly additive — adding it never changes what a range query returns. |
| Nothing ranking-aware (default) | Don't set any `ranked*` flag. The terminal property-name tree keeps the type its range flags give it. |
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions packages/dapi-grpc/protos/platform/v0/platform.proto
Original file line number Diff line number Diff line change
Expand Up @@ -893,16 +893,16 @@ message GetDocumentsRequest {
// - a is the In field AND b is the range field, in that order → existing compound distinct shape; entries carry both `in_key` (= a's value) and `key` (= b's value).
//
// `select=<COUNT(*)|SUM(f)|AVG(f)>, group_by=[p], order_by=[<the selected aggregate>]` (protocol v14+) — **ranked mode**:
// - exactly one `group_by` property, exactly one `order_by` clause naming the select's aggregate (`f` for `SUM(f)` / `AVG(f)`, the `$count` sentinel for `COUNT(*)`), a `limit` in `1 ..= 100`, an optional `offset`, and no `where` / `having` / `start_at`, on an index declaring the matching `rankedCountable` / `rankedSummable` / `rankedAverageable` axis → ranked executor, answered in `ResultData.ranked`.
// - exactly one `group_by` property, exactly one `order_by` clause naming the select's aggregate (`f` for `SUM(f)` / `AVG(f)`, the `$count` sentinel for `COUNT(*)`), a `limit` in `1 ..= 100`, an optional `offset`, and no `having` / `start_at`, on an index declaring the matching `rankedCountable` / `rankedSummable` / `rankedAverageable` axis → ranked executor, answered in `ResultData.ranked`. On a single-property ranked index no `where` is accepted; on a compound ranked index every leading index property must be pinned with an `EQUAL` where clause (one per property, `group_by` names the trailing property), selecting which prefix's ranking is read.
// - `DESC` is the "top n" reading (walk the axis from the largest aggregate down), `ASC` the "bottom n" reading. Worked example: `SELECT AVG(grade) GROUP BY restaurantId ORDER BY grade DESC LIMIT 1 OFFSET 4` is the 5th-best restaurant.
//
// `select=<COUNT(*)|SUM(f)|AVG(f)>, group_by=[p], having=[<the selected aggregate> <op> <value>]` (protocol v14+) — **having-range mode**:
// - exactly one `group_by` property, exactly one `having` clause whose aggregate is the select's aggregate, an operator describing one contiguous range (`EQUAL`, `GREATER_THAN[_OR_EQUALS]`, `LESS_THAN[_OR_EQUALS]`, `BETWEEN*`; `NOT_EQUAL` / `IN` rejected), a `limit` in `1 ..= 100`, an optional `order_by` naming the same aggregate (walk direction; ascending by default), and no `where` / `offset` / `start_at` / `start_after`, on an index declaring the matching ranked axis → having-range executor, answered in `ResultData.ranked`.
// - exactly one `group_by` property, exactly one `having` clause whose aggregate is the select's aggregate, an operator describing one contiguous range (`EQUAL`, `GREATER_THAN[_OR_EQUALS]`, `LESS_THAN[_OR_EQUALS]`, `BETWEEN*`; `NOT_EQUAL` / `IN` rejected), a `limit` in `1 ..= 100`, an optional `order_by` naming the same aggregate (walk direction; ascending by default), and no `offset` / `start_at` / `start_after`, on an index declaring the matching ranked axis → having-range executor, answered in `ResultData.ranked`. `where` follows the same rule as ranked mode: none on a single-property ranked index; exactly one `EQUAL` pin per leading index property on a compound ranked index.
// - no offset or cursor pagination: a page cut at `limit` continues only by tightening the bound past the last *distinct* aggregate value seen; a cut inside a tie (several groups sharing the boundary aggregate) cannot be continued, so size `limit` above the widest expected tie.
//
// **Rejected shapes** (return `Unsupported`):
// - any non-empty `having` on protocol v13 and earlier; at v14+, any `having` shape outside having-range mode above (multiple clauses, an aggregate other than the select's, `NOT_EQUAL` / `IN`, or a carried `where` / `offset` / cursor).
// - at v14+: a ranked-shaped request carrying a `where` clause, a `start_at` / `start_after` cursor, more than one `order_by`, or an `order_by` naming anything but the selected aggregate.
// - any non-empty `having` on protocol v13 and earlier; at v14+, any `having` shape outside having-range mode above (multiple clauses, an aggregate other than the select's, `NOT_EQUAL` / `IN`, a `where` shape other than the compound-index equality pins above, or a carried `offset` / cursor).
// - at v14+: a ranked-shaped request carrying a `where` shape other than the compound-index equality pins above (a non-`EQUAL` operator, a repeated or non-leading property, or a missing pin), a `start_at` / `start_after` cursor, more than one `order_by`, or an `order_by` naming anything but the selected aggregate.
// - `select=DOCUMENTS` with non-empty `group_by`.
// - `select=COUNT` with `group_by` on a field that is not constrained by an `In` or range where clause.
// - `select=COUNT` with `group_by.len() > 2`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,27 @@ pub(super) fn no_ranked_index_key_length_check(
Ok(())
}

/// RANKED: the cross-index structural check a generation runs over a
/// document type's parsed indices, before the merged index tree is built.
///
/// Generation 3's implementation rejects the compound-ranked prefix-overlap
/// shape the storage layer cannot lay out; earlier generations pass
/// [`no_ranked_index_structure_check`] — their grammar rejects the
/// `ranked*` keywords, so no index they parse can carry a ranking axis.
/// Unlike [`RankedIndexKeyLengthCheck`] this runs on **every** parse path,
/// not only under `full_validation`: a contract admitted through a
/// non-validating parse would brick the first document insert.
pub(super) type RankedIndexStructureCheck =
fn(&BTreeMap<String, Index>) -> Result<(), ProtocolError>;

/// The [`RankedIndexStructureCheck`] for a generation that has no ranking
/// axes to constrain.
pub(super) fn no_ranked_index_structure_check(
_indices: &BTreeMap<String, Index>,
) -> Result<(), ProtocolError> {
Ok(())
}

/// Everything the shared parsing steps need to know about *which* generation is
/// running them.
///
Expand Down Expand Up @@ -169,6 +190,8 @@ pub(super) struct ParserGeneration {
pub admit_ranked: bool,
/// See [`RankedIndexKeyLengthCheck`].
pub ranked_index_key_length_check: RankedIndexKeyLengthCheck,
/// See [`RankedIndexStructureCheck`].
pub ranked_index_structure_check: RankedIndexStructureCheck,
}

/// Reject a document type whose name is not a non-empty ASCII
Expand Down Expand Up @@ -947,6 +970,12 @@ fn parse_indices(
.transpose()?
.unwrap_or_default();

// Cross-index structural check owned by the generation, exactly like
// the per-property key-length check above: generations whose index
// grammar rejects the `ranked*` keywords pass the no-op, so the shared
// core never branches on a version.
(ctx.generation.ranked_index_structure_check)(&indices)?;

let index_structure =
IndexLevel::try_from_indices(indices.values(), ctx.name, ctx.platform_version)?;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ impl DocumentTypeV1 {
// therefore has no ranked key ceiling to enforce.
admit_ranked: false,
ranked_index_key_length_check: common::no_ranked_index_key_length_check,
ranked_index_structure_check: common::no_ranked_index_structure_check,
},
platform_version,
)
Expand Down
Loading
Loading