fix(drive): serve unproved ranked reads through the paginated prover

book/src/drive/ranked-index-examples.md

@@ -321,11 +321,11 @@ The response carries the skip back in `RankedEntries.skipped` (see [The Response
 
 Three properties worth stating plainly:
 
-- **The skip is attested, not walked.** grovedb proves the skipped region from the counted subtree commitments (`HashWithCount` / `HashWithCountAndSum`) rather than by traversing it. Both the prover's work and the proof's size stay `O(log n + k)` **at any offset**.
-- **There is therefore no offset ceiling.** An offset of 4 and an offset of four billion cost the same, so there is no denial-of-service lever a cap would close — and a cap would only stop honest deep pagination.
+- **The skip is attested, not walked.** grovedb proves the skipped region from the counted subtree commitments (`HashWithCount` / `HashWithCountAndSum`) rather than by traversing it entry by entry. Both executors go through that prover, so the work and the proof's size stay `O(log n + k)` **at any offset**.
+- **There is therefore no offset ceiling.** An offset of 4 and an offset of four billion cost the same order of work — on either path — so there is no denial-of-service lever a cap would close, and a cap would only stop honest deep pagination.
 - **An offset past the end is a positive answer.** `entries` comes back empty and `skipped` is the ranking's *entire attested population*. "There are only 12 groups" is more information than a bare empty list.
 
-On the **unproven** read there is nothing to attest and grovedb's read API does not report a short walk, so `skipped` simply echoes the requested offset. The proved and unproven paths therefore disagree in exactly one case — an offset past the end, where the unproven read reports the request and the proved one reports the truth. **Callers who need the population must prove.**
+On the **unproven** read the server derives the same number the same way — it extracts its page from that same counted envelope rather than walking the secondary — so the two paths no longer disagree anywhere, including past the end. What proving adds is that the value is *attested*: nothing in an unproven response is, so it is only as good as the node that sent it. **Callers who need to trust the population, rather than merely receive it, must still prove.**
 
 ## The Response
 

packages/dapi-grpc/protos/platform/v0/platform.proto

@@ -1407,12 +1407,18 @@ message GetDocumentsResponse {
       // attested count, re-derived by the verifier from the counted
       // subtree commitments in the proof bytes rather than trusted
       // from this field; a proving client should use the verified
-      // value. On the unproven read there is nothing to attest and
-      // grovedb's read API does not report a short walk, so the server
-      // echoes the requested offset. The two therefore disagree in
-      // exactly one case — an offset past the end, where the unproven
-      // read reports the request and the proved one reports the truth.
-      // Callers who need the population must prove.
+      // value. An unproven response carries the same number — the
+      // server extracts its page from that same counted envelope — so
+      // the two no longer disagree anywhere, including past the end.
+      // What proving adds is that the value is *attested*: nothing in
+      // an unproven response is, so it is only as good as the node
+      // that sent it. Callers who need to trust the population, rather
+      // than merely receive it, must still prove.
+      //
+      // Do not assume this field equals the offset you requested. It
+      // equals the offset only when the skip succeeded; when the walk
+      // ran out of groups first it is smaller, and that is the answer
+      // rather than an inconsistency.
       optional uint64 skipped = 2 [jstype = JS_STRING];
     }
 

packages/rs-drive-abci/src/query/document_query/v1/mod.rs

@@ -1317,36 +1317,63 @@ impl<C> Platform<C> {
                 document_type_name, contract_id
             ))));
 
-        let drive_request = DocumentRankedRequest {
-            contract: contract_ref,
-            document_type,
-            group_by: &group_by,
-            select,
-            having: &having,
-            order_by: &order_clauses,
-            where_clauses: &where_clauses,
-            limit,
-            offset,
-            has_start_at: start.is_some(),
-            prove,
-        };
+        // A ranked page is committed to by an envelope built from several
+        // independent storage reads, which are not isolated from a
+        // concurrent block commit. A commit landing inside that window
+        // leaves the envelope's ancestor chain unreconcilable and grovedb
+        // rejects it — a transient condition that a fresh envelope over
+        // settled state resolves, so it is retried here rather than
+        // returned. `query::service` already re-runs a query whose
+        // committed height moved, but there is a window between a commit
+        // becoming visible and that height being published where it does
+        // not, and this closes it.
+        //
+        // Deliberately narrow: only the chain mismatch is retried, and
+        // only a bounded number of times. Genuine corruption produces the
+        // same rejection on every attempt and so still surfaces, one
+        // envelope later.
+        let mut attempts_left = RANKED_CHAIN_MISMATCH_RETRIES;
+        let drive_response = loop {
+            // Rebuilt per attempt rather than cloned: every field is a
+            // borrow of state that outlives the loop, plus the request's
+            // one owned value.
+            let drive_request = DocumentRankedRequest {
+                contract: contract_ref,
+                document_type,
+                group_by: &group_by,
+                select: select.clone(),
+                having: &having,
+                order_by: &order_clauses,
+                where_clauses: &where_clauses,
+                limit,
+                offset,
+                has_start_at: start.is_some(),
+                prove,
+            };
 
-        let drive_response =
             match self
                 .drive
                 .execute_document_ranked_request(drive_request, None, platform_version)
             {
-                Ok(r) => r,
+                Ok(r) => break r,
                 Err(drive::error::Error::Query(qe)) => {
                     return Ok(QueryValidationResult::new_with_error(QueryError::Query(qe)));
                 }
                 Err(e) => match empty_ranking_proof_rejection(&e) {
                     Some(rejection) => {
                         return Ok(QueryValidationResult::new_with_error(rejection));
                     }
-                    None => return Err(e.into()),
+                    None => {
+                        if is_ranked_chain_mismatch(&e) && attempts_left > 0 {
+                            attempts_left -= 1;
+                            continue;
+                        }
+                        annotate_ranked_chain_mismatch(&e);
+                        return Err(e.into());
+                    }
                 },
-            };
+            }
+        };
 
         let response = match drive_response {
             DocumentRankedResponse::Entries(page) => GetDocumentsResponseV1 {
@@ -1445,6 +1472,11 @@ fn into_v1_ranked_entry(e: DriveRankedEntry) -> RankedEntry {
 /// ancestor chain — not a single call site, and because the cost of
 /// keeping it is one string comparison on an error path.
 ///
+/// It is applied to both `prove` settings, and has to be: an unproven
+/// ranked page is decoded out of an internally generated proof, so
+/// anything that stops the envelope from being built stops both forms of
+/// the request.
+///
 /// Historically: the non-paginated prover had no absence-proof shape
 /// for "this axis secondary has no entries", so proving a ranking over
 /// an index that held no documents failed with a merk-level "Cannot
@@ -1463,6 +1495,68 @@ fn into_v1_ranked_entry(e: DriveRankedEntry) -> RankedEntry {
 /// narrow: any other `CorruptedData` still propagates as an internal
 /// error, because for every other cause that classification is
 /// correct.
+/// How many times a ranked request is rebuilt when its envelope loses the
+/// race against a block commit.
+///
+/// The condition needs a commit to land inside one envelope's read window,
+/// which takes microseconds, so a single retry over settled state is
+/// already overwhelmingly likely to succeed; two bounds the cost at three
+/// envelopes (~400 µs) for a request that would otherwise have failed.
+/// Raising it would trade real work against a vanishing tail.
+const RANKED_CHAIN_MISMATCH_RETRIES: u8 = 2;
+
+/// Whether a drive error is the ancestor-chain reconciliation failure that
+/// a concurrent commit produces.
+///
+/// Detected by variant plus marker substring rather than by a typed error,
+/// for the same reason [`empty_ranking_proof_rejection`] is: grovedb
+/// flattens the failure into a `CorruptedData(String)` at the indexed-axis
+/// proof boundary. "chain mismatch" is the substring both of grovedb's
+/// reconciliation errors carry (the deepest layer's and the intermediate
+/// ancestors').
+fn is_ranked_chain_mismatch(error: &drive::error::Error) -> bool {
+    let drive::error::Error::GroveDB(grove_error) = error else {
+        return false;
+    };
+    let drive::query::GroveError::CorruptedData(message) = grove_error.as_ref() else {
+        return false;
+    };
+    message.contains("chain mismatch")
+}
+
+/// Name the benign cause of a ranked chain-mismatch error in the log,
+/// without reclassifying it.
+///
+/// A ranked page is committed to by an envelope built from several
+/// independent storage reads, which are not isolated from a concurrent
+/// block commit. A commit landing inside that window leaves the envelope's
+/// ancestor chain unreconcilable, grovedb rejects it with a "chain
+/// mismatch", and the request fails. The condition is benign and the query
+/// layer re-runs a query whose committed height moved — but there is a
+/// small window where it does not, and then this surfaces as an internal
+/// error whose log line is indistinguishable from real storage corruption.
+///
+/// It stays an internal error on purpose: a chain mismatch is also what
+/// genuine corruption looks like, and reclassifying it as retriable would
+/// hide that. Only the operator-facing explanation is added, so whoever
+/// reads the line at three in the morning knows which of the two to
+/// suspect first, and that a *lone* occurrence under load is the race.
+///
+/// Applies to both `prove` settings. The proved path has always had this
+/// exposure; unproved reads share it now that they are served from the
+/// same envelope.
+fn annotate_ranked_chain_mismatch(error: &drive::error::Error) {
+    if !is_ranked_chain_mismatch(error) {
+        return;
+    }
+    tracing::warn!(
+        error = %error,
+        "ranked query failed to reconcile its proof's ancestor chain on every attempt; \
+         a burst under load is queries racing block commits, but a persistent or \
+         unloaded occurrence is not and should be treated as suspected corruption"
+    );
+}
+
 fn empty_ranking_proof_rejection(error: &drive::error::Error) -> Option<QueryError> {
     let drive::error::Error::GroveDB(grove_error) = error else {
         return None;
@@ -1474,11 +1568,12 @@ fn empty_ranking_proof_rejection(error: &drive::error::Error) -> Option<QueryErr
         return None;
     }
     Some(QueryError::InvalidArgument(
-        "this ranking has no groups yet, and an empty ranking cannot be proved: \
-         grovedb has no absence-proof shape for an empty axis secondary. Retry \
-         with `prove = false` — the unproven read answers the same request with \
-         an empty entry list. Once the index holds at least one document, the \
-         proved form works."
+        "this ranking could not be committed to: some merk along its path holds no \
+         entries, and merk cannot prove a key in an empty tree. An empty *ranking* is \
+         not itself the problem — a ranked index with no documents yet proves fine, as \
+         an empty page — so this is about the state of the indexed tree rather than the \
+         shape of the request. Dropping `prove` does not route around it: an unproven \
+         ranked page is extracted from the same envelope."
             .to_string(),
     ))
 }

packages/rs-drive-abci/src/query/document_query/v1/tests.rs

@@ -2502,17 +2502,24 @@ mod ranked_tests {
         assert_eq!(tail.skipped, Some(3));
 
         // A window entirely past the end is an empty page, not an
-        // error. On this *unproven* path grovedb's read API doesn't
-        // report the short walk, so `skipped` echoes the request; the
-        // proved path is where it becomes the attested population.
+        // error, and `skipped` collapses to the population the walk
+        // actually found. That reaches the wire on this *unproven* path
+        // too — drive extracts the page from an internally generated
+        // paginated envelope, so the collapsed skip comes with it. It
+        // used to echo the requested 9 back instead.
         let past_end = ranked_page(&platform, &state, paged(2, 9), version);
         assert!(
             past_end.entries.is_empty(),
             "there is no rank 9 in a five-group ranking, and asking for one is not an \
              error — got {:?}",
             group_keys(&past_end.entries)
         );
-        assert_eq!(past_end.skipped, Some(9));
+        assert_eq!(
+            past_end.skipped,
+            Some(5),
+            "the response reports the five groups the ranking holds, not the offset that \
+             was asked for"
+        );
 
         // And the same page proves.
         let result = platform
@@ -3031,4 +3038,99 @@ mod ranked_tests {
             assert_ne!(label, "ranked", "no ORDER BY means no ranked routing");
         }
     }
+
+    /// The empty-tree mapper turns one grovedb failure into a
+    /// caller-facing argument error, and its value is entirely in how
+    /// *narrow* it is: every other `CorruptedData` must stay an internal
+    /// error, because for every other cause that classification is
+    /// correct. Nothing pinned either half before.
+    ///
+    /// It also applies to both `prove` settings, which is not an
+    /// oversight — an unproven ranked page is decoded out of an
+    /// internally generated envelope, so whatever stops the envelope from
+    /// being built stops both forms of the request. That is why the
+    /// message must not tell a caller to retry without `prove`.
+    #[test]
+    fn the_empty_tree_mapper_matches_only_its_own_failure() {
+        use drive::error::Error as DriveError;
+        use drive::query::GroveError;
+
+        let mapped = empty_ranking_proof_rejection(&DriveError::GroveDB(Box::new(
+            GroveError::CorruptedData("Cannot create proof for empty tree".to_string()),
+        )))
+        .expect("the empty-tree failure is a caller-facing condition");
+        let QueryError::InvalidArgument(message) = mapped else {
+            panic!("an empty-tree failure must be an argument error, not a server fault");
+        };
+        assert!(
+            !message.contains("Retry with `prove = false`"),
+            "the message must not send callers down a path that no longer exists — an \
+             unproven read is served from the same envelope: {message}"
+        );
+
+        // Everything else stays an internal error.
+        for other in [
+            GroveError::CorruptedData("some unrelated corruption".to_string()),
+            GroveError::PathNotFound("no such subtree".to_string()),
+        ] {
+            let label = format!("{other:?}");
+            assert!(
+                empty_ranking_proof_rejection(&DriveError::GroveDB(Box::new(other))).is_none(),
+                "only the empty-tree failure may be reclassified, not {label}"
+            );
+        }
+        assert!(
+            empty_ranking_proof_rejection(&DriveError::Drive(
+                drive::error::drive::DriveError::CorruptedDriveState(
+                    "Cannot create proof for empty tree".to_string()
+                )
+            ))
+            .is_none(),
+            "the marker string alone must not reclassify a non-grovedb error"
+        );
+    }
+
+    /// The chain-mismatch detector decides whether a ranked request is
+    /// rebuilt or surfaced, so its narrowness is the whole safety property:
+    /// retrying the wrong error class would spend three envelopes on a
+    /// failure that cannot improve, and — worse — retrying *everything*
+    /// would turn a genuine fault into three of them.
+    ///
+    /// The message it matches is grovedb's, wrapped: the reconciliation
+    /// failure arrives as `CorruptedData` with the marker embedded in
+    /// grovedb's own prefix, never as the bare marker, which is why this
+    /// matches a substring rather than the whole string.
+    #[test]
+    fn only_a_grovedb_chain_mismatch_is_retried() {
+        use drive::error::Error as DriveError;
+        use drive::query::GroveError;
+
+        let wrapped = "indexed-axis paginated proof: intermediate layer at depth 2 chain \
+                       mismatch — parent recorded value_hash ab, computed cd";
+        assert!(
+            is_ranked_chain_mismatch(&DriveError::GroveDB(Box::new(GroveError::CorruptedData(
+                wrapped.to_string()
+            )))),
+            "the reconciliation failure must be recognised inside grovedb's wrapper text"
+        );
+
+        // Everything else is surfaced on the first attempt.
+        for other in [
+            GroveError::CorruptedData("Cannot create proof for empty tree".to_string()),
+            GroveError::CorruptedData("some unrelated corruption".to_string()),
+            GroveError::PathNotFound("no such subtree".to_string()),
+        ] {
+            let label = format!("{other:?}");
+            assert!(
+                !is_ranked_chain_mismatch(&DriveError::GroveDB(Box::new(other))),
+                "must not be retried: {label}"
+            );
+        }
+        assert!(
+            !is_ranked_chain_mismatch(&DriveError::Drive(
+                drive::error::drive::DriveError::CorruptedDriveState("chain mismatch".to_string())
+            )),
+            "the marker alone must not make a non-grovedb error retriable"
+        );
+    }
 }