ivanvyd · ivanvyd · Aug 1, 2026 · Aug 1, 2026
@@ -125,7 +125,7 @@ spec:
 lakespeak pack run daily-platform-brief.yaml
 ```
 
-See the [Question Pack guide](docs/question-packs/) for the schema and failure semantics.
+See the [Question Pack guide](docs/question-packs.md) for the schema and failure semantics.
 
 ## What it does not do
 
@@ -158,8 +158,9 @@ promise; it is a record of evidence.
 
 | Area | Status |
 |---|---|
-| Unit and contract tests (mocked) | See CI |
-| Azure Databricks, live workspace | See [docs/compatibility.md](docs/compatibility.md) |
+| Unit and contract tests | 89 tests, run on Windows and Linux in CI |
+| Azure Databricks, live workspace | `agents list`, `ask`, `pack run` and every output format verified against a real Genie Agent on 2026-08-01 |
+| `chat`, feedback, full-result download, visualizations | Contract tests only — **not** exercised live |
 | AWS Databricks | Not tested |
 | GCP Databricks | Not tested |
 | Windows / Linux | Covered by the CI matrix |
@@ -170,13 +171,14 @@ Paths that have not been exercised against a real workspace are labelled as such
 
 ## Documentation
 
-- [Concepts](docs/concepts/) — Agents, conversations, attachments
-- [Commands](docs/commands/) — every command and flag
-- [Authentication](docs/authentication/) — profiles, U2M, M2M, and what is not supported
-- [Question Packs](docs/question-packs/)
-- [Architecture](docs/architecture/) and [decisions](docs/decisions/)
-- [Troubleshooting](docs/troubleshooting/)
-- [Limitations](docs/limitations.md)
+- [Commands](docs/commands.md) — every command, flag and exit code
+- [Authentication](docs/authentication.md) — profiles, environment tokens, and what is not supported
+- [Question Packs](docs/question-packs.md) — the schema and its failure semantics
+- [Troubleshooting](docs/troubleshooting.md)
+- [Limitations](docs/limitations.md) — read this one
+- [Decisions](docs/decisions/) — ADRs for the load-bearing choices
+- [Genie API surface](docs/planning/genie-api-surface.md) — every wire claim, labelled verified or not
+- [SOC 2 control mapping](docs/compliance/soc2-mapping.md)
 
 ## Contributing
 

@@ -0,0 +1,96 @@
+# Authentication
+
+LakeSpeak never stores a credential. It has no credential store, and its configuration file has no
+field capable of holding one.
+
+## Interactive use: the Databricks CLI
+
+The default. You log in once with the Databricks CLI, and LakeSpeak borrows the token it caches.
+
+```bash
+databricks auth login --profile company
+lakespeak chat --profile company
+```
+
+Under the hood LakeSpeak runs `databricks auth token --profile company` and uses the short-lived
+OAuth token it returns. The browser flow, the token cache and the refresh logic all stay inside a
+tool Databricks maintains — see [ADR 0003](decisions/0003-broker-credentials-through-the-databricks-cli.md)
+for why that trade was made.
+
+The CLI is invoked with an argument vector, never through a shell, because profile names can come
+from a config file or a Question Pack.
+
+## Unattended use: an environment token
+
+CI runners and containers have no browser, and often no Databricks CLI. Set `DATABRICKS_TOKEN` and
+LakeSpeak uses it directly, taking precedence over the CLI broker.
+
+```bash
+export DATABRICKS_HOST="https://adb-xxxxxxxxxxxx.n.azuredatabricks.net"
+export DATABRICKS_TOKEN="<token>"
+lakespeak ask --agent finance --format json "Revenue yesterday?"
+```
+
+Two honest caveats.
+
+A **personal access token** is a standing credential with no refresh and no expiry pressure.
+Databricks documents it as a local-debugging path rather than a production one, and `auth check`
+warns when it finds one in a profile.
+
+On **Azure**, a better option exists: an Entra ID access token for the Databricks resource works
+directly as a bearer token, for a user principal or a managed identity. It is short-lived, and a
+managed identity needs no Azure RBAC role for this.
+
+```bash
+export DATABRICKS_TOKEN=$(az account get-access-token \
+  --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --query accessToken -o tsv)
+```
+
+This is the path used to verify LakeSpeak against a live workspace; see
+[compatibility.md](compatibility.md).
+
+## What is not supported
+
+**OAuth M2M client-credential profiles.** `databricks auth token` covers user-to-machine profiles
+only, so a profile configured with `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET` cannot be
+brokered through the CLI. Native M2M is v0.2 work. Until then, use an environment token.
+
+**OIDC workload federation**, GitHub Actions OIDC, and Azure DevOps OIDC. Also v0.2 or later.
+
+## Resolving the workspace host
+
+First match wins:
+
+1. `GenieClientOptions.Host` set in code
+2. `DATABRICKS_HOST`
+3. the `host` of the named profile in `.databrickscfg`
+4. the `host` of the `DEFAULT` profile
+
+`https` is required. A non-https host is rejected rather than accepted, because a bearer token over
+plain HTTP is a disclosed token and the mistake is otherwise silent.
+
+`lakespeak config show` prints which value won and where it came from.
+
+## In a .NET application
+
+```csharp
+services.AddLakeSpeak(options => options.Profile = "production");
+```
+
+To supply your own token — from an application's existing OAuth flow, or a managed identity:
+
+```csharp
+services.AddGenieTokenProvider(async ct => await myTokenSource.GetAsync(ct));
+services.AddLakeSpeak(options => options.Host = new Uri("https://..."));
+```
+
+Register the provider **before** `AddLakeSpeak`; it uses `TryAdd`, so a provider you register wins.
+
+## What LakeSpeak guarantees
+
+- No token is written to disk, put in a URL, or passed as a process argument.
+- Authorization headers and both Databricks signature fields are redacted from all diagnostic
+  output, including `--verbose`, and from exception messages.
+- Presigned result URLs are fetched **without** the `Authorization` header, so a Databricks
+  credential is never handed to blob storage.
+- `auth check` and `config show` print no part of any credential.
@@ -0,0 +1,137 @@
+# Command reference
+
+Every command accepts the global options below. `lakespeak <command> --help` prints the same
+information at the terminal.
+
+## Global options
+
+| Option | Meaning |
+|---|---|
+| `--profile`, `-p` | Databricks CLI profile to authenticate with |
+| `--format`, `-f` | `text` (default), `table`, `markdown`, `json`, `jsonl`, `csv` |
+| `--quiet`, `-q` | Suppress progress output on stderr |
+| `--verbose` | Print diagnostics to stderr. Credentials are always redacted |
+
+Results go to **stdout**; progress, warnings and errors go to **stderr**. That split is what lets
+`lakespeak ask --format json … 2>/dev/null` produce clean JSON for a script.
+
+Colour and progress are suppressed automatically when output is redirected, when `NO_COLOR` is
+set, or when the format is machine-readable.
+
+## `lakespeak agents list`
+
+Lists the Genie Agents your identity can see.
+
+```bash
+lakespeak agents list
+lakespeak agents list --format json
+```
+
+An identity with no Genie grants sees nothing. That is reported as a warning with exit `0`, not as
+an error — the fix is a Databricks permission, not a change to how you invoked the command.
+
+## `lakespeak ask <question>`
+
+Asks a question in a new conversation and prints the answer.
+
+```bash
+lakespeak ask --agent sales "Who were our five fastest-growing customers?"
+lakespeak ask --agent finance --format json "What was recognized revenue yesterday?"
+lakespeak ask --agent finance --show-sql "Revenue by market and month"
+```
+
+| Option | Meaning |
+|---|---|
+| `--agent`, `-a` | Agent id, exact title, or a configured alias |
+| `--show-sql` | Print the generated SQL alongside the answer |
+
+`--agent` accepts an id, a title, or an alias from your config file. If a name matches more than
+one Agent, the command **fails** rather than picking one — answering against the wrong Finance
+Agent looks exactly like success.
+
+With `--format csv`, the *query result* is written. A narrative answer has no rows, so if the
+response carries no result you get a warning on stderr and nothing on stdout.
+
+## `lakespeak chat`
+
+An interactive, stateful conversation. Follow-up questions keep their context.
+
+```bash
+lakespeak chat
+lakespeak chat --agent platform-operations
+```
+
+With no `--agent`, a selector appears so you never have to paste an id.
+
+Requires an interactive terminal — it refuses to run against a pipe, because a chat loop reading
+EOF spins and a selector nobody can answer looks like a hang. Use `ask` for scripts.
+
+Ctrl+C cancels the question in flight and returns you to the prompt; it does not end the session.
+
+| Slash command | Meaning |
+|---|---|
+| `/help` | List commands |
+| `/agents` | List available Agents |
+| `/use <agent>` | Switch Agent and start a new conversation |
+| `/new` | Start a new conversation with the same Agent |
+| `/sql` | Show the SQL behind the last answer |
+| `/result` | Show the last query result |
+| `/export <path>` | Write the last result to CSV |
+| `/thumbs-up`, `/thumbs-down [comment]` | Send feedback to Databricks |
+| `/exit` | Leave |
+
+## `lakespeak pack`
+
+```bash
+lakespeak pack init my-brief.yaml       # write a starter pack
+lakespeak pack validate my-brief.yaml   # check it without running it
+lakespeak pack run my-brief.yaml        # run it and write the report
+```
+
+| Option (on `run`) | Meaning |
+|---|---|
+| `--output`, `-o` | Write the report here instead of the pack's configured path |
+| `--force` | Overwrite an existing report |
+
+`validate` reports **every** problem at once rather than stopping at the first, so fixing a pack is
+one pass instead of repeated guessing.
+
+See the [Question Pack guide](question-packs.md).
+
+## `lakespeak auth check`
+
+Verifies that a profile resolves, that a token can be obtained, and that the workspace answers.
+
+```bash
+lakespeak auth check --profile company
+```
+
+Prints the token's length, never any part of its value. Warns about profiles holding a legacy
+personal access token.
+
+## `lakespeak config show`
+
+Prints the effective configuration and, for each value, where it came from.
+
+```bash
+lakespeak config show
+```
+
+The output contains no credentials, so it is safe to paste into an issue.
+
+## Exit codes
+
+| Code | Meaning |
+|---:|---|
+| 0 | Success |
+| 1 | Unexpected failure |
+| 2 | Invalid command, configuration, or Question Pack |
+| 3 | Authentication failure |
+| 4 | Authorization failure |
+| 5 | Agent or conversation not found |
+| 6 | Genie could not answer |
+| 7 | Timeout or cancellation |
+| 8 | Question Pack finished with some questions failed |
+| 9 | Unsupported or malformed response |
+
+These are contractual. An existing code will not change meaning.
@@ -28,10 +28,33 @@ otherwise, please open an issue. Visualization retrieval is explicitly Beta and
 
 | Cloud | Status |
 |---|---|
-| Azure Databricks | **Not yet verified against a live workspace** |
+| Azure Databricks | **Verified against a live workspace on 2026-08-01** — see below |
 | AWS Databricks | Not tested |
 | GCP Databricks | Not tested |
 
+## Live verification, 2026-08-01
+
+Run against an Azure Databricks premium workspace in `eastus2`, on a serverless SQL warehouse,
+against a Genie Agent over a synthetic revenue table created for the purpose. Authentication was an
+Entra ID access token for resource `2ff814a6-3304-4ab8-85cb-cd0e6f879c1d`, supplied through
+`DATABRICKS_TOKEN`.
+
+| Path | Result |
+|---|---|
+| `agents list` (table and json) | Listed the live Agent with its real id |
+| `ask` with `--show-sql` | Real answer, real result table, real generated SQL, exit 0 |
+| `ask --format json` | Valid UTF-8, versioned schema, on stdout only |
+| `ask --format csv` | Clean CSV on stdout with diagnostics on stderr, verified via `2>/dev/null` |
+| `ask` against an unknown Agent | Real listing lookup, exit 2 |
+| `pack run` | Two questions, Markdown report written, exit 0 |
+| Decimal fidelity | `4500000.00`, `3350000.50`, `1780000.25` reached CSV, JSON and Markdown byte-identical to what Databricks returned |
+| Column types | `DECIMAL(22,2)` — precision and scale preserved, which `type_name` alone would have lost |
+| Non-ASCII | `€` intact in a file-written report |
+
+What this did **not** exercise: `chat` (needs an interactive terminal), feedback submission,
+full-result downloads, visualizations, cancellation against a genuinely long-running query, and
+`QUERY_RESULT_EXPIRED` recovery. Those remain covered by contract tests only.
+
 ## What has been verified, and how
 
 | Area | Evidence |

@@ -11,6 +11,22 @@ cancelled orders. The answer is still a confident sentence with a number in it.
 LakeSpeak preserves the generated SQL, the query result and the message identifiers precisely so a
 person can check. It has no way to judge correctness, and neither does any other client.
 
+## Genie may answer with a question
+
+Genie sometimes responds by asking for clarification rather than answering — for example, "Would
+you prefer to see the active customers for exactly the quarter '2026-Q1' instead of any quarter
+containing '2026-Q1'?" This is observed behaviour, not a hypothetical; it happened on the second
+question of the first Question Pack ever run against a live workspace.
+
+The message still completes successfully and the clarification arrives as the answer text, because
+that is genuinely what Genie returned. LakeSpeak does not try to detect it: distinguishing "a
+question back" from "an answer phrased as a question" is a judgement call, and guessing wrong in
+either direction is worse than reporting faithfully.
+
+The consequence for automation is worth planning around. An unattended Question Pack can produce a
+report whose answer is a request for clarification, and it will exit `0` because nothing failed.
+Phrase pack questions to be unambiguous, and read reports rather than trusting the exit code alone.
+
 ## It cannot see more than you can
 
 Every request carries your identity, and Unity Catalog decides what that identity sees. There is no