ivanvyd · ivanvyd · Aug 5, 2026 · Aug 5, 2026
@@ -12,8 +12,11 @@ here rather than left to be discovered.
   its first chunk — correctly flagged as truncated, but short. The client now follows the
   `next_chunk_internal_link` Databricks supplies until the result is complete. Every way of
   failing to complete one still reports truncation, so a partial result is still never presented
-  as whole. Verified by contract tests; **not yet exercised against a live workspace**. See
-  [ADR 0004](docs/decisions/0004-complete-a-chunked-result-by-following-the-link-databricks-supplies.md).
+  as whole. The mechanism was verified against a live Azure workspace on 2026-08-05 — including
+  the permission question ADR 0004 recorded as unknown, which the answer to is yes — though Genie
+  emitting a multi-chunk result was not reproduced. See
+  [ADR 0004](docs/decisions/0004-complete-a-chunked-result-by-following-the-link-databricks-supplies.md)
+  and [compatibility.md](docs/compatibility.md).
 
 ### Added
 

@@ -206,8 +206,8 @@ promise; it is a record of evidence.
 | Unit and contract tests | Run on Windows and Linux in CI, on every push and pull request |
 | Azure Databricks, live workspace | `agents list`, `ask`, `pack run` and every output format verified against a real Genie Agent on 2026-08-01 |
 | `chat`, feedback, full-result download, visualizations | Contract tests only — **not** exercised live |
-| Chunked results assembled beyond the first chunk | Contract tests only — **not** exercised live. Whether a caller may read the remaining chunks of a Genie-executed statement is unverified; if not, the first chunk is returned flagged as truncated, as before |
-| Unattended service-principal authentication | Documented from Databricks' published flow, **not** executed by this project |
+| Chunked results assembled beyond the first chunk | Mechanism verified live on 2026-08-05 — chunk reads on a Genie-executed statement are permitted, and the link, response shape and row arithmetic all behave as the client assumes. Genie emitting a multi-chunk result was **not** reproduced |
+| Unattended service-principal authentication | Request shape verified against the live token endpoint; the exchange with real service-principal credentials was **not** run |
 | AWS Databricks | Not tested |
 | GCP Databricks | Not tested |
 | Windows / Linux | Covered by the CI matrix |

@@ -24,12 +24,12 @@ Status: in development.
   - `chat` — the REPL refuses to start without an interactive terminal, by design, so it cannot be
     driven from CI or an agent session. Its underlying follow-up call *is* covered live; the loop
     around it is not. Needs a human at a terminal.
-  - Chunked results — now assembled by following the link Databricks supplies
-    ([ADR 0004](docs/decisions/0004-complete-a-chunked-result-by-following-the-link-databricks-supplies.md)),
-    but still unverified live: Genie writes its own SQL and generally bounds it, so producing a
-    result large enough to span chunks is not reliably reachable through the Conversation API.
-    Covered by contract tests against a stubbed multi-chunk response. Whether a caller may read the
-    remaining chunks of a Genie-executed statement is the specific unknown.
+  - Chunked results — assembled by following the link Databricks supplies
+    ([ADR 0004](docs/decisions/0004-complete-a-chunked-result-by-following-the-link-databricks-supplies.md)).
+    The mechanism was verified live on 2026-08-05, including the permission question that was the
+    stated unknown. What remains unreached is Genie *itself* emitting a multi-chunk result: it
+    writes its own SQL and generally bounds it, so this needs a Genie Agent over a large table and
+    still cannot be forced. Covered by contract tests meanwhile.
   - `QUERY_RESULT_EXPIRED` recovery — the cache expires on Databricks' schedule, hours later. No
     way to force it; covered by contract tests only.
 - [x] Command reference and authentication guide

@@ -76,21 +76,45 @@ Not a live workspace. Recorded because the packaged surface is the only surface
 | Packaged tool installed to an isolated `--tool-path` | `lakespeak --version` → `0.1.0+b934d6b…`, matching the merge commit |
 | Throwaway consumer against `LakeSpeak.Genie.0.1.0.nupkg` | Compiles and reads the new `MaxResultRows`, default `100000` — proving the new public member is in the *package*, not just the build output |
 
-## Chunked result assembly — implemented, unverified live
+## Chunked result assembly — mechanism verified live, 2026-08-05
 
-The client now follows `next_chunk_internal_link` to assemble a chunked result
+The client follows `next_chunk_internal_link` to assemble a chunked result
 ([ADR 0004](decisions/0004-complete-a-chunked-result-by-following-the-link-databricks-supplies.md)).
-This has **never run against a real workspace**, and one specific thing is unknown: whether a
-caller may read the remaining chunks of a statement Genie executed on their behalf. That link
-resolves to `/api/2.0/sql/statements/…`, which is not a Genie path, and no documentation states
-whether the caller's identity carries there.
+Every assumption that design rests on was checked against the live Azure workspace.
 
-If the answer is no, the client returns the first chunk flagged as truncated — exactly the
-behaviour that preceded the change. Nothing regresses; the improvement simply does not arrive.
+**The permission question is answered: yes.** This was the named unknown — whether a caller may
+read the remaining chunks of a statement Genie executed on their behalf, given the link resolves to
+`/api/2.0/sql/statements/…`, which is not a Genie path.
 
-Contract tests cover assembly across three chunks, an unreachable chunk, a repeated link, a link
-resolving off-workspace, and the row cap. What they cannot cover is Databricks' real answer to that
-permission question.
+| Probe | Result |
+|---|---|
+| Genie `start-conversation` → completed message → `query.statement_id` | `01f19102-25bf-…` |
+| `GET /api/2.0/sql/statements/{that id}` with the caller's own token | **HTTP 200**, manifest, `SUCCEEDED` |
+| `GET .../result/chunks/0` for that Genie-executed statement | **HTTP 200**, rows returned |
+
+**The wire contract behaves as the client assumes.** A deliberately chunked statement
+(`SELECT id, repeat('x',120) FROM range(60000)`, run through the Statement Execution API so no
+table was created):
+
+| Observation | Value |
+|---|---|
+| `manifest.total_chunk_count` | `2` |
+| `manifest.truncated` on a merely-chunked result | **`false`** — the original defect's premise, confirmed rather than assumed |
+| Chunk 0 `row_count` | `41250` of `60000` |
+| `next_chunk_internal_link` | `/api/2.0/sql/statements/{id}/result/chunks/1` — workspace-relative, the shape the client's host validation accepts |
+| Following that link | HTTP 200, `row_offset: 41250`, `row_count: 18750`, no `next_chunk_index` |
+| Assembled total | `41250 + 18750 = 60000`, matching `total_row_count` exactly |
+
+That covers the link's shape, the chunk response's shape, the loop's termination condition, and the
+row arithmetic the truncation flag depends on.
+
+**What is still not proven:** Genie itself emitting a result large enough to span chunks. The demo
+Agent's table has six rows, and Genie generally bounds its own SQL, so the two halves above are each
+verified while their composition is not. If Genie never emits a multi-chunk result, this code simply
+never engages; if it does, every mechanism it needs has now been shown to work.
+
+Contract tests continue to cover the paths a live run cannot reach on demand: an unreachable chunk,
+a repeated link, a link resolving off-workspace, and the row cap.
 
 ## What has been verified, and how
 
@@ -104,10 +128,10 @@ permission question.
 | Credential redaction, both signature fields | Unit tests using realistic JSON payloads |
 | Question Pack validation, including path traversal | Unit tests, plus the CLI run by hand |
 | CLI parsing, help, exit codes, `config show` leaking nothing | The built binary run by hand |
-| Chunked result assembly, and every way it can stop short | Contract tests against a stubbed multi-chunk server. **Never run live** — see the section above for the specific unknown |
+| Chunked result assembly, and every way it can stop short | Contract tests against a stubbed multi-chunk server. The wire contract and the chunk-read permission were verified live on 2026-08-05 — see above; Genie emitting a multi-chunk result was not |
 | Documented CLI commands still existing | A test parses every fenced `lakespeak …` example in this repository against the real command tree |
 | The packaged public API, as opposed to the build output | A consumer project compiled against the `.nupkg` from `./artifacts`, 2026-08-05 |
-| Unattended service-principal auth | **Not verified.** `docs/authentication.md` documents Databricks' published M2M call and this project has not executed it |
+| Unattended service-principal auth | **Partly.** The token endpoint accepts the documented request — invalid credentials return `401 invalid_client`, not `404` — so the URL, method, Basic auth and form parameters are right. The exchange with *valid* service-principal credentials has not been run |
 | Against real Databricks | See "Live verification" above. `agents list`, `ask`, every output format, `pack run`, `export last` and `feedback last` were run against a live workspace; `chat`, chunked/external-link results and `QUERY_RESULT_EXPIRED` recovery were not. |
 
 ## How to update this file

@@ -57,8 +57,24 @@ A chunk that cannot be fetched degrades to the rows already in hand rather than
 answer text is the primary payload and a missing tail should not discard it; the truncation flag
 is what tells the caller the table is short.
 
-**This is verified by contract tests and has not been exercised against a live workspace.**
-Whether a caller may read the remaining chunks of a statement Genie executed on their behalf is a
-workspace permission question that no documentation settles. The unreachable-chunk path exists
-because the answer may well be "no", in which case behaviour is exactly what it was before this
-change. `docs/compatibility.md` records this as untested rather than implying otherwise.
+## Verification — 2026-08-05
+
+The premise this decision was least sure of has since been checked against a live Azure workspace,
+and it holds.
+
+Whether a caller may read the remaining chunks of a statement Genie executed on their behalf was
+recorded here as settled by no documentation. It is now settled by observation: a Genie-executed
+statement's id, taken from a completed message, returned **HTTP 200** from both
+`/api/2.0/sql/statements/{id}` and `/api/2.0/sql/statements/{id}/result/chunks/0` using the
+caller's own token.
+
+The wire contract was checked the same way, on a deliberately chunked statement: `total_chunk_count`
+of 2, `manifest.truncated` **false** on a merely-chunked result — the defect's premise, confirmed —
+a `next_chunk_internal_link` of `/api/2.0/sql/statements/{id}/result/chunks/1`, and following it
+returning the remaining rows with no `next_chunk_index`, summing exactly to `total_row_count`.
+
+What remains unproven is Genie *itself* emitting a multi-chunk result, which needs a large table in
+a Genie Agent and cannot be forced, since Genie bounds its own SQL. Both halves are verified; their
+composition is not. The unreachable-chunk path therefore stays — not because the permission answer
+is expected to be "no", but because a path that returns rows to a user must not depend on an
+assumption nobody re-checks. See [compatibility.md](../compatibility.md) for the evidence.
@@ -75,10 +75,11 @@ link that is not a workspace-relative path, a chunk the caller is not permitted
 
 Two caveats worth knowing. `MaxResultRows` defaults to 100,000 rows, because following a chunked
 result to its end is otherwise unbounded work held in memory — raise it if you would rather have
-the memory cost than the shortfall. And **this path has not been exercised against a live
-workspace**: whether a caller may read the remaining chunks of a statement Genie executed on their
-behalf is a permission question no documentation settles, and if the answer is no, behaviour is
-what it was before — the first chunk, flagged.
+the memory cost than the shortfall. And while the **mechanism** was verified against a live
+workspace on 2026-08-05 — chunk reads on a Genie-executed statement are permitted, and the link,
+response shape and row arithmetic all behave as assumed — **Genie producing a multi-chunk result
+was never reproduced**, because the Agent used for verification has a six-row table and Genie
+bounds its own SQL. See [compatibility.md](compatibility.md).
 
 The truncation flag itself was wrong until a post-ship review caught it: the client relied on
 `manifest.truncated`, which reports statement-level truncation by Databricks and is `false` for a