From cbee49704d18b157480c54f2daa653bf6741d3f2 Mon Sep 17 00:00:00 2001 From: Ivan Vydrin Date: Wed, 5 Aug 2026 21:55:04 +0300 Subject: [PATCH] Record what this cycle actually verified, and what it did not The cycle's theme was that a claim in prose drifts from reality because nothing owns it. It then left the repository's own evidence ledger describing the previous version of the code, which is the same defect wearing the same clothes. compatibility.md described chunk handling as though nothing had changed, and the README's verification table never mentioned chunked results at all -- so the headline change of the cycle appeared in neither place a reader checks before trusting it. Three things are now recorded rather than assumed. Chunked assembly exists and has never run against a real workspace. The specific unknown is named: the chunk link resolves to /api/2.0/sql/statements, which is not a Genie path, and no documentation says whether the caller's identity carries there. If it does not, the client returns the first chunk flagged as truncated -- the behaviour that preceded the change -- so nothing regresses and the improvement simply does not arrive. The service-principal recipe is documented from Databricks' published flow and has not been executed by this project. It is a set of instructions, not a tested path, and the ledger now says so where someone deciding whether to rely on it will read it. The 2026-08-05 local verification is recorded, including the one check that distinguishes a package from a build: a consumer compiled against the nupkg reads MaxResultRows, so the new public member is provably in the artifact a user installs. --- README.md | 2 ++ docs/compatibility.md | 40 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 027644e..869c077 100644 --- a/README.md +++ b/README.md @@ -206,6 +206,8 @@ promise; it is a record of evidence. | Unit and contract tests | Run on Windows and Linux in CI, on every push and pull request | | Azure Databricks, live workspace | `agents list`, `ask`, `pack run` and every output format verified against a real Genie Agent on 2026-08-01 | | `chat`, feedback, full-result download, visualizations | Contract tests only — **not** exercised live | +| Chunked results assembled beyond the first chunk | Contract tests only — **not** exercised live. Whether a caller may read the remaining chunks of a Genie-executed statement is unverified; if not, the first chunk is returned flagged as truncated, as before | +| Unattended service-principal authentication | Documented from Databricks' published flow, **not** executed by this project | | AWS Databricks | Not tested | | GCP Databricks | Not tested | | Windows / Linux | Covered by the CI matrix | diff --git a/docs/compatibility.md b/docs/compatibility.md index d6c5e4f..206f0ab 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -58,9 +58,39 @@ The live suite in `tests/LakeSpeak.LiveIntegrationTests` reproduces all of this. `DATABRICKS_HOST`, `DATABRICKS_TOKEN` and `LAKESPEAK_LIVE_AGENT` set: `dotnet test -c Release --filter "Category=Live"`. -What this still does **not** exercise: `chat` (needs an interactive terminal), full-result downloads -beyond the first chunk, visualizations, and `QUERY_RESULT_EXPIRED` recovery. Those remain covered by -contract tests only. +What this still does **not** exercise: `chat` (needs an interactive terminal), chunked result +assembly, the Genie full-result download endpoints, visualizations, and `QUERY_RESULT_EXPIRED` +recovery. Those remain covered by contract tests only. + +## Local verification — 2026-08-05 + +Not a live workspace. Recorded because the packaged surface is the only surface a user meets, and +"it built" is not evidence that it packaged. + +| Check | Result | +|---|---| +| Full suite, `Category!=Live` | 224 tests, 0 failed, across five projects | +| Build under warnings-as-errors | Clean | +| `dotnet format --verify-no-changes` | Clean | +| `dotnet restore --locked-mode` | Succeeds; no lock file drift | +| Packaged tool installed to an isolated `--tool-path` | `lakespeak --version` → `0.1.0+b934d6b…`, matching the merge commit | +| Throwaway consumer against `LakeSpeak.Genie.0.1.0.nupkg` | Compiles and reads the new `MaxResultRows`, default `100000` — proving the new public member is in the *package*, not just the build output | + +## Chunked result assembly — implemented, unverified live + +The client now follows `next_chunk_internal_link` to assemble a chunked result +([ADR 0004](decisions/0004-complete-a-chunked-result-by-following-the-link-databricks-supplies.md)). +This has **never run against a real workspace**, and one specific thing is unknown: whether a +caller may read the remaining chunks of a statement Genie executed on their behalf. That link +resolves to `/api/2.0/sql/statements/…`, which is not a Genie path, and no documentation states +whether the caller's identity carries there. + +If the answer is no, the client returns the first chunk flagged as truncated — exactly the +behaviour that preceded the change. Nothing regresses; the improvement simply does not arrive. + +Contract tests cover assembly across three chunks, an unreachable chunk, a repeated link, a link +resolving off-workspace, and the row cap. What they cannot cover is Databricks' real answer to that +permission question. ## What has been verified, and how @@ -74,6 +104,10 @@ contract tests only. | Credential redaction, both signature fields | Unit tests using realistic JSON payloads | | Question Pack validation, including path traversal | Unit tests, plus the CLI run by hand | | CLI parsing, help, exit codes, `config show` leaking nothing | The built binary run by hand | +| Chunked result assembly, and every way it can stop short | Contract tests against a stubbed multi-chunk server. **Never run live** — see the section above for the specific unknown | +| Documented CLI commands still existing | A test parses every fenced `lakespeak …` example in this repository against the real command tree | +| The packaged public API, as opposed to the build output | A consumer project compiled against the `.nupkg` from `./artifacts`, 2026-08-05 | +| Unattended service-principal auth | **Not verified.** `docs/authentication.md` documents Databricks' published M2M call and this project has not executed it | | Against real Databricks | See "Live verification" above. `agents list`, `ask`, every output format, `pack run`, `export last` and `feedback last` were run against a live workspace; `chat`, chunked/external-link results and `QUERY_RESULT_EXPIRED` recovery were not. | ## How to update this file