fix(drive): serve unproved ranked reads through the paginated prover

The `prove = false` arm of the ranked query called grovedb's direct read,
whose offset skip is a per-entry walk: one iterator step and one decode
per skipped entry, so the skip alone cost Theta(min(offset, population)).
`OFFSET` has no ceiling, ranked queries carry no fee, the dispatcher does
no cost accounting, a `spawn_blocking` query cannot be cancelled by client
disconnect or stream reset, and the gateway rate-limits per source IP
across the whole Platform service rather than per method. Measured on a
1,000,000-group fixture: 15 us at OFFSET 0, 457 ms at OFFSET 4e9. At
~1,950 req/min from one unauthenticated IP that is roughly 7x
oversubscription of a 2-vCPU node shared with consensus.

`execute_top_k_no_proof` now generates the same envelope the prove path
returns and extracts the page from it with grovedb's verifier, so the
skipped region is attested from counted subtree commitments instead of
walked. The full prove -> verify -> extract round trip measures
78/90/121/129 us at N = 1e3/1e4/1e5/1e6; verification is flat in offset
and in N and scales only with k. The deep-offset lever becomes a flat
48 us. The cost is +113 us absolute in one corner -- offset 0 with a small
k -- and from N = 100k the round trip is cheaper than the direct read
outright.

grovedb's verifier is called rather than drive's `verify::document_ranked`
wrapper because drive-abci builds drive without the `verify` feature, so
the wrapper is not compiled there. No Cargo edits, no feature flags, no
grovedb change, no proto field change, and no platform-version gate:
queries do not touch state, the only production caller is the v1 query
handler with `transaction: None`, and no query cost feeds block execution
or fees.

BEHAVIOUR CHANGE, wire-visible on unproved responses

`RankedPage::skipped` -- `GetDocumentsResponseV1.ResultData.Ranked.skipped`
on the wire -- stops echoing the request and starts reporting the truth.
grovedb's direct read returns an empty vector when the walk exhausts
during the skip and never reports how far it got, so drive had no choice
but to echo the requested offset. Through the envelope it gets the
attested count. On a five-group ranking, `LIMIT 2 OFFSET 9` with
`prove = false` now reports `skipped = 5` rather than `9`; the proved path
is unchanged. Only the past-the-end case differs, and the two paths no
longer diverge at all -- what proving adds is that the value is attested,
not that it differs. A client asserting `skipped == requested_offset`
will see a different value; a client using it as the rank base for
`entries[i]`, its documented purpose, is unaffected and gains a
population count it previously had to prove for.

The contract is corrected in every place that stated the old behaviour:
`platform.proto`, the Objective-C generated client that carries proto
prose verbatim, the developer book, and the Rust docs.

COMMENTS CORRECTED

Three comments asserted things the code did not do, and two of them were
policy: "there is no denial-of-service lever here to cap" justified the
uncapped OFFSET. They land with the fix rather than before it, because
rewriting them into an accurate description while shipping no mitigation
would have published an unmitigated unauthenticated remote DoS, with a
worked recipe, in a public repo. A fourth surfaced during review: the
drive-abci empty-ranking rejection told callers to retry with
`prove = false`, a route this change removes, and blamed a prover that is
retired and not on the path.

TESTS

Two defects, two independent pins, both verified red before green.

- Result equality: 144 cases (3 axes x 2 directions x k in {1, 4, 100} x
  offsets {0, 1, 5, 11, 12, 13, 100, 4e9}) against a population where
  three groups share every aggregate value, the values straddle zero, and
  the averages are non-integral -- so tie ordering, the sign boundary and
  the fixed point's rounding are exercised at every offset. The oracle is
  a live call to the old grovedb primitives; `skipped` is asserted from
  arithmetic, `min(offset, population)`.
  Reverting the implementation: 4 tests red in drive plus the wire
  assertion in drive-abci, all on `skipped`.

- Cost: asserted through drive's own executor via a new
  `execute_top_k_no_proof_with_cost` (public API unchanged), on both
  `seek_count` and `storage_loaded_bytes` -- the same `OperationCost`
  fields `fees/op.rs` already reads to compute charged credits -- against
  a bound derived from the secondary's AVL height rather than a chosen
  constant, plus a guard that fails if the fixture stops being able to
  catch a walk, plus a size tripwire for the "proof size is
  offset-independent" claim that nothing previously pinned. Over 200
  groups the direct read costs 5 -> 105 -> 204 -> 204 seeks and
  635 -> 30,835 -> 60,671 -> 60,671 bytes at offsets 0/100/200/4e9; the
  executor stays at 36 -> 35 -> 23 -> 23 and 4,458 -> 4,515 -> 2,414 ->
  2,414.
  This pin was verified against the case that defeats a naive one -- a
  direct read that also reports a truthful `skipped` by following its
  empty result with a count aggregate, satisfying every `skipped`
  assertion while restoring the walk. It goes red: "the executor's seeks
  must not grow with the offset: 105 at offset 100 against 5 at offset 0".
  Without it the security defect had no regression evidence; only the
  behaviour change did.

- `empty_ranking_proof_rejection` gets its first test, including the
  negative case pinning how deliberately narrow its match is.

drive --lib 3324 passed; drive-abci --lib query:: 607 passed; clippy and
fmt clean. drive also compiles server-only, verify-only, and for
wasm32-unknown-unknown via wasm-drive-verify.

Verification here discards the reconstructed root hash rather than
comparing it against a locally read one: query reads are not
snapshot-isolated, `finalize_block` makes a commit visible before
publishing the height the query layer's retry guard compares, and a
locally read root is not an independent trust boundary in any case. The
reasoning is recorded at the call site.

Author: shumkov

book/src/drive/ranked-index-examples.md

@@ -321,11 +321,11 @@ The response carries the skip back in `RankedEntries.skipped` (see [The Response
 
 Three properties worth stating plainly:
 
-- **The skip is attested, not walked.** grovedb proves the skipped region from the counted subtree commitments (`HashWithCount` / `HashWithCountAndSum`) rather than by traversing it. Both the prover's work and the proof's size stay `O(log n + k)` **at any offset**.
-- **There is therefore no offset ceiling.** An offset of 4 and an offset of four billion cost the same, so there is no denial-of-service lever a cap would close — and a cap would only stop honest deep pagination.
+- **The skip is attested, not walked.** grovedb proves the skipped region from the counted subtree commitments (`HashWithCount` / `HashWithCountAndSum`) rather than by traversing it entry by entry. Both executors go through that prover, so the work and the proof's size stay `O(log n + k)` **at any offset**.
+- **There is therefore no offset ceiling.** An offset of 4 and an offset of four billion cost the same order of work — on either path — so there is no denial-of-service lever a cap would close, and a cap would only stop honest deep pagination.
 - **An offset past the end is a positive answer.** `entries` comes back empty and `skipped` is the ranking's *entire attested population*. "There are only 12 groups" is more information than a bare empty list.
 
-On the **unproven** read there is nothing to attest and grovedb's read API does not report a short walk, so `skipped` simply echoes the requested offset. The proved and unproven paths therefore disagree in exactly one case — an offset past the end, where the unproven read reports the request and the proved one reports the truth. **Callers who need the population must prove.**
+On the **unproven** read the server derives the same number the same way — it extracts its page from that same counted envelope rather than walking the secondary — so the two paths no longer disagree anywhere, including past the end. What proving adds is that the value is *attested*: nothing in an unproven response is, so it is only as good as the node that sent it. **Callers who need to trust the population, rather than merely receive it, must still prove.**
 
 ## The Response
 

packages/dapi-grpc/clients/platform/v0/objective-c/Platform.pbobjc.h

@@ -1407,12 +1407,18 @@ message GetDocumentsResponse {
       // attested count, re-derived by the verifier from the counted
       // subtree commitments in the proof bytes rather than trusted
       // from this field; a proving client should use the verified
-      // value. On the unproven read there is nothing to attest and
-      // grovedb's read API does not report a short walk, so the server
-      // echoes the requested offset. The two therefore disagree in
-      // exactly one case — an offset past the end, where the unproven
-      // read reports the request and the proved one reports the truth.
-      // Callers who need the population must prove.
+      // value. An unproven response carries the same number — the
+      // server extracts its page from that same counted envelope — so
+      // the two no longer disagree anywhere, including past the end.
+      // What proving adds is that the value is *attested*: nothing in
+      // an unproven response is, so it is only as good as the node
+      // that sent it. Callers who need to trust the population, rather
+      // than merely receive it, must still prove.
+      //
+      // Do not assume this field equals the offset you requested. It
+      // equals the offset only when the skip succeeded; when the walk
+      // ran out of groups first it is smaller, and that is the answer
+      // rather than an inconsistency.
       optional uint64 skipped = 2 [jstype = JS_STRING];
     }
 

packages/dapi-grpc/protos/platform/v0/platform.proto

@@ -1344,7 +1344,10 @@ impl<C> Platform<C> {
                     Some(rejection) => {
                         return Ok(QueryValidationResult::new_with_error(rejection));
                     }
-                    None => return Err(e.into()),
+                    None => {
+                        annotate_ranked_chain_mismatch(&e);
+                        return Err(e.into());
+                    }
                 },
             };
 
@@ -1445,6 +1448,11 @@ fn into_v1_ranked_entry(e: DriveRankedEntry) -> RankedEntry {
 /// ancestor chain — not a single call site, and because the cost of
 /// keeping it is one string comparison on an error path.
 ///
+/// It is applied to both `prove` settings, and has to be: an unproven
+/// ranked page is decoded out of an internally generated proof, so
+/// anything that stops the envelope from being built stops both forms of
+/// the request.
+///
 /// Historically: the non-paginated prover had no absence-proof shape
 /// for "this axis secondary has no entries", so proving a ranking over
 /// an index that held no documents failed with a merk-level "Cannot
@@ -1463,6 +1471,45 @@ fn into_v1_ranked_entry(e: DriveRankedEntry) -> RankedEntry {
 /// narrow: any other `CorruptedData` still propagates as an internal
 /// error, because for every other cause that classification is
 /// correct.
+/// Name the benign cause of a ranked chain-mismatch error in the log,
+/// without reclassifying it.
+///
+/// A ranked page is committed to by an envelope built from several
+/// independent storage reads, which are not isolated from a concurrent
+/// block commit. A commit landing inside that window leaves the envelope's
+/// ancestor chain unreconcilable, grovedb rejects it with a "chain
+/// mismatch", and the request fails. The condition is benign and the query
+/// layer re-runs a query whose committed height moved — but there is a
+/// small window where it does not, and then this surfaces as an internal
+/// error whose log line is indistinguishable from real storage corruption.
+///
+/// It stays an internal error on purpose: a chain mismatch is also what
+/// genuine corruption looks like, and reclassifying it as retriable would
+/// hide that. Only the operator-facing explanation is added, so whoever
+/// reads the line at three in the morning knows which of the two to
+/// suspect first, and that a *lone* occurrence under load is the race.
+///
+/// Applies to both `prove` settings. The proved path has always had this
+/// exposure; unproved reads share it now that they are served from the
+/// same envelope.
+fn annotate_ranked_chain_mismatch(error: &drive::error::Error) {
+    let drive::error::Error::GroveDB(grove_error) = error else {
+        return;
+    };
+    let drive::query::GroveError::CorruptedData(message) = grove_error.as_ref() else {
+        return;
+    };
+    if !message.contains("chain mismatch") {
+        return;
+    }
+    tracing::warn!(
+        error = message.as_str(),
+        "ranked query failed to reconcile its proof's ancestor chain; an isolated \
+         occurrence under load is a query racing a block commit rather than corrupted \
+         state, but repeated occurrences — especially without load — are not"
+    );
+}
+
 fn empty_ranking_proof_rejection(error: &drive::error::Error) -> Option<QueryError> {
     let drive::error::Error::GroveDB(grove_error) = error else {
         return None;
@@ -1474,11 +1521,12 @@ fn empty_ranking_proof_rejection(error: &drive::error::Error) -> Option<QueryErr
         return None;
     }
     Some(QueryError::InvalidArgument(
-        "this ranking has no groups yet, and an empty ranking cannot be proved: \
-         grovedb has no absence-proof shape for an empty axis secondary. Retry \
-         with `prove = false` — the unproven read answers the same request with \
-         an empty entry list. Once the index holds at least one document, the \
-         proved form works."
+        "this ranking could not be committed to: some merk along its path holds no \
+         entries, and merk cannot prove a key in an empty tree. An empty *ranking* is \
+         not itself the problem — a ranked index with no documents yet proves fine, as \
+         an empty page — so this is about the state of the indexed tree rather than the \
+         shape of the request. Dropping `prove` does not route around it: an unproven \
+         ranked page is extracted from the same envelope."
             .to_string(),
     ))
 }

packages/rs-drive-abci/src/query/document_query/v1/mod.rs

@@ -2502,17 +2502,24 @@ mod ranked_tests {
         assert_eq!(tail.skipped, Some(3));
 
         // A window entirely past the end is an empty page, not an
-        // error. On this *unproven* path grovedb's read API doesn't
-        // report the short walk, so `skipped` echoes the request; the
-        // proved path is where it becomes the attested population.
+        // error, and `skipped` collapses to the population the walk
+        // actually found. That reaches the wire on this *unproven* path
+        // too — drive extracts the page from an internally generated
+        // paginated envelope, so the collapsed skip comes with it. It
+        // used to echo the requested 9 back instead.
         let past_end = ranked_page(&platform, &state, paged(2, 9), version);
         assert!(
             past_end.entries.is_empty(),
             "there is no rank 9 in a five-group ranking, and asking for one is not an \
              error — got {:?}",
             group_keys(&past_end.entries)
         );
-        assert_eq!(past_end.skipped, Some(9));
+        assert_eq!(
+            past_end.skipped,
+            Some(5),
+            "the response reports the five groups the ranking holds, not the offset that \
+             was asked for"
+        );
 
         // And the same page proves.
         let result = platform
@@ -3031,4 +3038,55 @@ mod ranked_tests {
             assert_ne!(label, "ranked", "no ORDER BY means no ranked routing");
         }
     }
+
+    /// The empty-tree mapper turns one grovedb failure into a
+    /// caller-facing argument error, and its value is entirely in how
+    /// *narrow* it is: every other `CorruptedData` must stay an internal
+    /// error, because for every other cause that classification is
+    /// correct. Nothing pinned either half before.
+    ///
+    /// It also applies to both `prove` settings, which is not an
+    /// oversight — an unproven ranked page is decoded out of an
+    /// internally generated envelope, so whatever stops the envelope from
+    /// being built stops both forms of the request. That is why the
+    /// message must not tell a caller to retry without `prove`.
+    #[test]
+    fn the_empty_tree_mapper_matches_only_its_own_failure() {
+        use drive::error::Error as DriveError;
+        use drive::query::GroveError;
+
+        let mapped = empty_ranking_proof_rejection(&DriveError::GroveDB(Box::new(
+            GroveError::CorruptedData("Cannot create proof for empty tree".to_string()),
+        )))
+        .expect("the empty-tree failure is a caller-facing condition");
+        let QueryError::InvalidArgument(message) = mapped else {
+            panic!("an empty-tree failure must be an argument error, not a server fault");
+        };
+        assert!(
+            !message.contains("Retry with `prove = false`"),
+            "the message must not send callers down a path that no longer exists — an \
+             unproven read is served from the same envelope: {message}"
+        );
+
+        // Everything else stays an internal error.
+        for other in [
+            GroveError::CorruptedData("some unrelated corruption".to_string()),
+            GroveError::PathNotFound("no such subtree".to_string()),
+        ] {
+            let label = format!("{other:?}");
+            assert!(
+                empty_ranking_proof_rejection(&DriveError::GroveDB(Box::new(other))).is_none(),
+                "only the empty-tree failure may be reclassified, not {label}"
+            );
+        }
+        assert!(
+            empty_ranking_proof_rejection(&DriveError::Drive(
+                drive::error::drive::DriveError::CorruptedDriveState(
+                    "Cannot create proof for empty tree".to_string()
+                )
+            ))
+            .is_none(),
+            "the marker string alone must not reclassify a non-grovedb error"
+        );
+    }
 }

packages/rs-drive-abci/src/query/document_query/v1/tests.rs

@@ -1542,10 +1542,13 @@ fn an_offset_window_spanning_the_end_returns_the_short_tail() {
 /// The page comes back empty and `skipped` collapses below the requested
 /// offset — and *that shape* is the proof that the ranking holds exactly
 /// `skipped` groups in total, because the counted commitments cover the whole
-/// walk. It is the only way this surface reports a population, and the one
-/// place the proved and unproven paths differ: the unproven read cannot see
-/// the short walk (grovedb's read API returns an empty vector either way) and
-/// reports the requested offset.
+/// walk. It is the only way this surface reports a population.
+///
+/// Both paths report it. The unproven read gets its page out of an internally
+/// generated envelope rather than by walking the secondary, so it carries the
+/// same collapsed skip; it once echoed the requested offset back (grovedb's
+/// direct read returns an empty vector either way and never says how far it
+/// got), and this test is where that echo is pinned as gone.
 #[test]
 fn an_offset_past_the_end_returns_an_empty_page_whose_skip_attests_the_population() {
     let (drive, contract) = setup_restaurants();
@@ -1557,8 +1560,9 @@ fn an_offset_past_the_end_returns_an_empty_page_whose_skip_attests_the_populatio
         "there is no rank 12 in a five-group ranking"
     );
     assert_eq!(
-        page.skipped, 12,
-        "the unproven read echoes the requested offset — it has nothing to attest with"
+        page.skipped, 5,
+        "the unproven read reports the population it actually reached, not the requested \
+         offset of 12"
     );
 
     let verified = verified_ranked_avg_page(&drive, &contract, 3, 12);
@@ -1568,6 +1572,11 @@ fn an_offset_past_the_end_returns_an_empty_page_whose_skip_attests_the_populatio
         "skipped < offset with an empty page is a proof that the ranking holds exactly \
          five groups"
     );
+    assert_eq!(
+        page.skipped, verified.skipped,
+        "the two paths agree on the population; proving buys the attestation, not a \
+         different number"
+    );
 
     assert_grovedb_is_consistent(&drive);
 }