diff --git a/README.md b/README.md index ea33172..f18fd84 100644 --- a/README.md +++ b/README.md @@ -125,7 +125,7 @@ spec: lakespeak pack run daily-platform-brief.yaml ``` -See the [Question Pack guide](docs/question-packs/) for the schema and failure semantics. +See the [Question Pack guide](docs/question-packs.md) for the schema and failure semantics. ## What it does not do @@ -158,8 +158,9 @@ promise; it is a record of evidence. | Area | Status | |---|---| -| Unit and contract tests (mocked) | See CI | -| Azure Databricks, live workspace | See [docs/compatibility.md](docs/compatibility.md) | +| Unit and contract tests | 89 tests, run on Windows and Linux in CI | +| Azure Databricks, live workspace | `agents list`, `ask`, `pack run` and every output format verified against a real Genie Agent on 2026-08-01 | +| `chat`, feedback, full-result download, visualizations | Contract tests only — **not** exercised live | | AWS Databricks | Not tested | | GCP Databricks | Not tested | | Windows / Linux | Covered by the CI matrix | @@ -170,13 +171,14 @@ Paths that have not been exercised against a real workspace are labelled as such ## Documentation -- [Concepts](docs/concepts/) — Agents, conversations, attachments -- [Commands](docs/commands/) — every command and flag -- [Authentication](docs/authentication/) — profiles, U2M, M2M, and what is not supported -- [Question Packs](docs/question-packs/) -- [Architecture](docs/architecture/) and [decisions](docs/decisions/) -- [Troubleshooting](docs/troubleshooting/) -- [Limitations](docs/limitations.md) +- [Commands](docs/commands.md) — every command, flag and exit code +- [Authentication](docs/authentication.md) — profiles, environment tokens, and what is not supported +- [Question Packs](docs/question-packs.md) — the schema and its failure semantics +- [Troubleshooting](docs/troubleshooting.md) +- [Limitations](docs/limitations.md) — read this one +- [Decisions](docs/decisions/) — ADRs for the load-bearing choices +- [Genie API surface](docs/planning/genie-api-surface.md) — every wire claim, labelled verified or not +- [SOC 2 control mapping](docs/compliance/soc2-mapping.md) ## Contributing diff --git a/docs/authentication.md b/docs/authentication.md new file mode 100644 index 0000000..dedc052 --- /dev/null +++ b/docs/authentication.md @@ -0,0 +1,96 @@ +# Authentication + +LakeSpeak never stores a credential. It has no credential store, and its configuration file has no +field capable of holding one. + +## Interactive use: the Databricks CLI + +The default. You log in once with the Databricks CLI, and LakeSpeak borrows the token it caches. + +```bash +databricks auth login --profile company +lakespeak chat --profile company +``` + +Under the hood LakeSpeak runs `databricks auth token --profile company` and uses the short-lived +OAuth token it returns. The browser flow, the token cache and the refresh logic all stay inside a +tool Databricks maintains — see [ADR 0003](decisions/0003-broker-credentials-through-the-databricks-cli.md) +for why that trade was made. + +The CLI is invoked with an argument vector, never through a shell, because profile names can come +from a config file or a Question Pack. + +## Unattended use: an environment token + +CI runners and containers have no browser, and often no Databricks CLI. Set `DATABRICKS_TOKEN` and +LakeSpeak uses it directly, taking precedence over the CLI broker. + +```bash +export DATABRICKS_HOST="https://adb-xxxxxxxxxxxx.n.azuredatabricks.net" +export DATABRICKS_TOKEN="" +lakespeak ask --agent finance --format json "Revenue yesterday?" +``` + +Two honest caveats. + +A **personal access token** is a standing credential with no refresh and no expiry pressure. +Databricks documents it as a local-debugging path rather than a production one, and `auth check` +warns when it finds one in a profile. + +On **Azure**, a better option exists: an Entra ID access token for the Databricks resource works +directly as a bearer token, for a user principal or a managed identity. It is short-lived, and a +managed identity needs no Azure RBAC role for this. + +```bash +export DATABRICKS_TOKEN=$(az account get-access-token \ + --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --query accessToken -o tsv) +``` + +This is the path used to verify LakeSpeak against a live workspace; see +[compatibility.md](compatibility.md). + +## What is not supported + +**OAuth M2M client-credential profiles.** `databricks auth token` covers user-to-machine profiles +only, so a profile configured with `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET` cannot be +brokered through the CLI. Native M2M is v0.2 work. Until then, use an environment token. + +**OIDC workload federation**, GitHub Actions OIDC, and Azure DevOps OIDC. Also v0.2 or later. + +## Resolving the workspace host + +First match wins: + +1. `GenieClientOptions.Host` set in code +2. `DATABRICKS_HOST` +3. the `host` of the named profile in `.databrickscfg` +4. the `host` of the `DEFAULT` profile + +`https` is required. A non-https host is rejected rather than accepted, because a bearer token over +plain HTTP is a disclosed token and the mistake is otherwise silent. + +`lakespeak config show` prints which value won and where it came from. + +## In a .NET application + +```csharp +services.AddLakeSpeak(options => options.Profile = "production"); +``` + +To supply your own token — from an application's existing OAuth flow, or a managed identity: + +```csharp +services.AddGenieTokenProvider(async ct => await myTokenSource.GetAsync(ct)); +services.AddLakeSpeak(options => options.Host = new Uri("https://...")); +``` + +Register the provider **before** `AddLakeSpeak`; it uses `TryAdd`, so a provider you register wins. + +## What LakeSpeak guarantees + +- No token is written to disk, put in a URL, or passed as a process argument. +- Authorization headers and both Databricks signature fields are redacted from all diagnostic + output, including `--verbose`, and from exception messages. +- Presigned result URLs are fetched **without** the `Authorization` header, so a Databricks + credential is never handed to blob storage. +- `auth check` and `config show` print no part of any credential. diff --git a/docs/commands.md b/docs/commands.md new file mode 100644 index 0000000..a16225f --- /dev/null +++ b/docs/commands.md @@ -0,0 +1,137 @@ +# Command reference + +Every command accepts the global options below. `lakespeak --help` prints the same +information at the terminal. + +## Global options + +| Option | Meaning | +|---|---| +| `--profile`, `-p` | Databricks CLI profile to authenticate with | +| `--format`, `-f` | `text` (default), `table`, `markdown`, `json`, `jsonl`, `csv` | +| `--quiet`, `-q` | Suppress progress output on stderr | +| `--verbose` | Print diagnostics to stderr. Credentials are always redacted | + +Results go to **stdout**; progress, warnings and errors go to **stderr**. That split is what lets +`lakespeak ask --format json … 2>/dev/null` produce clean JSON for a script. + +Colour and progress are suppressed automatically when output is redirected, when `NO_COLOR` is +set, or when the format is machine-readable. + +## `lakespeak agents list` + +Lists the Genie Agents your identity can see. + +```bash +lakespeak agents list +lakespeak agents list --format json +``` + +An identity with no Genie grants sees nothing. That is reported as a warning with exit `0`, not as +an error — the fix is a Databricks permission, not a change to how you invoked the command. + +## `lakespeak ask ` + +Asks a question in a new conversation and prints the answer. + +```bash +lakespeak ask --agent sales "Who were our five fastest-growing customers?" +lakespeak ask --agent finance --format json "What was recognized revenue yesterday?" +lakespeak ask --agent finance --show-sql "Revenue by market and month" +``` + +| Option | Meaning | +|---|---| +| `--agent`, `-a` | Agent id, exact title, or a configured alias | +| `--show-sql` | Print the generated SQL alongside the answer | + +`--agent` accepts an id, a title, or an alias from your config file. If a name matches more than +one Agent, the command **fails** rather than picking one — answering against the wrong Finance +Agent looks exactly like success. + +With `--format csv`, the *query result* is written. A narrative answer has no rows, so if the +response carries no result you get a warning on stderr and nothing on stdout. + +## `lakespeak chat` + +An interactive, stateful conversation. Follow-up questions keep their context. + +```bash +lakespeak chat +lakespeak chat --agent platform-operations +``` + +With no `--agent`, a selector appears so you never have to paste an id. + +Requires an interactive terminal — it refuses to run against a pipe, because a chat loop reading +EOF spins and a selector nobody can answer looks like a hang. Use `ask` for scripts. + +Ctrl+C cancels the question in flight and returns you to the prompt; it does not end the session. + +| Slash command | Meaning | +|---|---| +| `/help` | List commands | +| `/agents` | List available Agents | +| `/use ` | Switch Agent and start a new conversation | +| `/new` | Start a new conversation with the same Agent | +| `/sql` | Show the SQL behind the last answer | +| `/result` | Show the last query result | +| `/export ` | Write the last result to CSV | +| `/thumbs-up`, `/thumbs-down [comment]` | Send feedback to Databricks | +| `/exit` | Leave | + +## `lakespeak pack` + +```bash +lakespeak pack init my-brief.yaml # write a starter pack +lakespeak pack validate my-brief.yaml # check it without running it +lakespeak pack run my-brief.yaml # run it and write the report +``` + +| Option (on `run`) | Meaning | +|---|---| +| `--output`, `-o` | Write the report here instead of the pack's configured path | +| `--force` | Overwrite an existing report | + +`validate` reports **every** problem at once rather than stopping at the first, so fixing a pack is +one pass instead of repeated guessing. + +See the [Question Pack guide](question-packs.md). + +## `lakespeak auth check` + +Verifies that a profile resolves, that a token can be obtained, and that the workspace answers. + +```bash +lakespeak auth check --profile company +``` + +Prints the token's length, never any part of its value. Warns about profiles holding a legacy +personal access token. + +## `lakespeak config show` + +Prints the effective configuration and, for each value, where it came from. + +```bash +lakespeak config show +``` + +The output contains no credentials, so it is safe to paste into an issue. + +## Exit codes + +| Code | Meaning | +|---:|---| +| 0 | Success | +| 1 | Unexpected failure | +| 2 | Invalid command, configuration, or Question Pack | +| 3 | Authentication failure | +| 4 | Authorization failure | +| 5 | Agent or conversation not found | +| 6 | Genie could not answer | +| 7 | Timeout or cancellation | +| 8 | Question Pack finished with some questions failed | +| 9 | Unsupported or malformed response | + +These are contractual. An existing code will not change meaning. diff --git a/docs/compatibility.md b/docs/compatibility.md index 92e158a..c68e504 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -28,10 +28,33 @@ otherwise, please open an issue. Visualization retrieval is explicitly Beta and | Cloud | Status | |---|---| -| Azure Databricks | **Not yet verified against a live workspace** | +| Azure Databricks | **Verified against a live workspace on 2026-08-01** — see below | | AWS Databricks | Not tested | | GCP Databricks | Not tested | +## Live verification, 2026-08-01 + +Run against an Azure Databricks premium workspace in `eastus2`, on a serverless SQL warehouse, +against a Genie Agent over a synthetic revenue table created for the purpose. Authentication was an +Entra ID access token for resource `2ff814a6-3304-4ab8-85cb-cd0e6f879c1d`, supplied through +`DATABRICKS_TOKEN`. + +| Path | Result | +|---|---| +| `agents list` (table and json) | Listed the live Agent with its real id | +| `ask` with `--show-sql` | Real answer, real result table, real generated SQL, exit 0 | +| `ask --format json` | Valid UTF-8, versioned schema, on stdout only | +| `ask --format csv` | Clean CSV on stdout with diagnostics on stderr, verified via `2>/dev/null` | +| `ask` against an unknown Agent | Real listing lookup, exit 2 | +| `pack run` | Two questions, Markdown report written, exit 0 | +| Decimal fidelity | `4500000.00`, `3350000.50`, `1780000.25` reached CSV, JSON and Markdown byte-identical to what Databricks returned | +| Column types | `DECIMAL(22,2)` — precision and scale preserved, which `type_name` alone would have lost | +| Non-ASCII | `€` intact in a file-written report | + +What this did **not** exercise: `chat` (needs an interactive terminal), feedback submission, +full-result downloads, visualizations, cancellation against a genuinely long-running query, and +`QUERY_RESULT_EXPIRED` recovery. Those remain covered by contract tests only. + ## What has been verified, and how | Area | Evidence | diff --git a/docs/limitations.md b/docs/limitations.md index d473657..ddd1591 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -11,6 +11,22 @@ cancelled orders. The answer is still a confident sentence with a number in it. LakeSpeak preserves the generated SQL, the query result and the message identifiers precisely so a person can check. It has no way to judge correctness, and neither does any other client. +## Genie may answer with a question + +Genie sometimes responds by asking for clarification rather than answering — for example, "Would +you prefer to see the active customers for exactly the quarter '2026-Q1' instead of any quarter +containing '2026-Q1'?" This is observed behaviour, not a hypothetical; it happened on the second +question of the first Question Pack ever run against a live workspace. + +The message still completes successfully and the clarification arrives as the answer text, because +that is genuinely what Genie returned. LakeSpeak does not try to detect it: distinguishing "a +question back" from "an answer phrased as a question" is a judgement call, and guessing wrong in +either direction is worse than reporting faithfully. + +The consequence for automation is worth planning around. An unattended Question Pack can produce a +report whose answer is a request for clarification, and it will exit `0` because nothing failed. +Phrase pack questions to be unambiguous, and read reports rather than trusting the exit code alone. + ## It cannot see more than you can Every request carries your identity, and Unity Catalog decides what that identity sees. There is no diff --git a/docs/question-packs.md b/docs/question-packs.md new file mode 100644 index 0000000..9b91aed --- /dev/null +++ b/docs/question-packs.md @@ -0,0 +1,116 @@ +# Question Packs + +A Question Pack turns a set of business questions into a repeatable, reviewable report. It lives in +your repository, goes through code review like anything else, and produces the same structure every +time it runs. + +```bash +lakespeak pack init daily-brief.yaml +lakespeak pack validate daily-brief.yaml +lakespeak pack run daily-brief.yaml +``` + +## A pack + +```yaml +apiVersion: lakespeak.dev/v1alpha1 +kind: QuestionPack + +metadata: + name: daily-platform-brief + description: Daily summary of Databricks platform health + +spec: + agent: platform-operations + + questions: + - id: failed-jobs + title: Failed production jobs + ask: > + Which production jobs failed during the last 24 hours? + Include job name, failure time, and latest error category. + timeout: 90s + + output: + format: markdown + path: reports/daily-platform-brief.md + + behavior: + continueOnQuestionFailure: true + includeGeneratedSql: false + includeTimings: true + includeIdentifiers: false +``` + +The full schema is published at +[`schemas/question-pack-v1alpha1.schema.json`](../schemas/question-pack-v1alpha1.schema.json). + +## Fields worth understanding + +**`spec.agent` is required and never inferred.** A report that silently ran against a different +Agent is worse than one that failed. + +**`behavior.continueOnQuestionFailure`** (default `true`) finishes the run and records failures in +place, exiting `8`. Set it to `false` to stop at the first failure. For a scheduled report, partial +results usually beat none. + +**`behavior.includeIdentifiers`** (default `false`) adds conversation and message ids so an answer +can be traced back in Databricks. It is off by default because those ids identify a conversation +containing governed data, and reports get committed. + +**`spec.output.path`** is relative to the pack file. Absolute paths and anything resolving outside +the pack's directory are rejected at load time — a pack can arrive in a pull request, so its output +path is attacker-influenced. + +## How a pack runs + +Questions run **sequentially**, and each gets a **fresh conversation**. + +Sequentially because each question occupies a SQL warehouse, and ten concurrent questions degrade +a shared warehouse for everyone on it. Reports are not latency-sensitive. + +Fresh conversations because Genie is stateful: reusing one would let the answer to question three +depend on questions one and two, making the report order-dependent in a way no reader would +suspect. + +## Guarantees + +- **A pack is data, never code.** It cannot execute a command, read a file, or widen the + permissions of the identity running it. +- **Validation is strict.** Unknown keys fail rather than being ignored, so a typo in + `continueOnQuestionFailure` cannot silently leave the intended behaviour off. +- **All errors at once.** `validate` reports every problem in one pass. +- **No prompts.** A pack run never asks a question interactively, so it cannot hang a cron job. + An ambiguous Agent name fails instead. +- **Deterministic output.** The same pack against the same data produces a byte-identical report + apart from the timestamp and timings, which is what makes a committed report reviewable in a + diff. +- **Capped at 50 questions.** Beyond that it is a scheduled job, not a report. + +## Reports carry a warning + +Every generated report states that answers come from natural-language questions and can be wrong in +ways that read as plausible, and tells the reader to check the generated SQL before acting on +anything consequential. That line is not configurable. + +## Something to plan around + +Genie sometimes replies with a **clarifying question** instead of an answer. The message completes +successfully and the clarification becomes the answer text, so the pack exits `0` with a report +whose answer is a question. Phrase pack questions unambiguously, and read reports rather than +trusting the exit code alone. See [limitations](limitations.md). + +## In CI + +```yaml +- name: Daily brief + env: + DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }} + DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }} + run: | + dotnet tool install --global LakeSpeak.Cli + lakespeak pack run packs/daily-brief.yaml --force +``` + +Remember that job logs are usually readable by everyone with repository access, and a report +contains governed data. Treat the artifact accordingly. diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md new file mode 100644 index 0000000..d3c4309 --- /dev/null +++ b/docs/security/threat-model.md @@ -0,0 +1,147 @@ +# Threat model + +What LakeSpeak is trying to prevent, what it is not, and where the boundaries sit. + +## What this is + +A CLI and library that runs on a workstation or a CI runner. It holds no data at rest, has no users +of its own, exposes no network service, and stores no credentials. It authenticates as **you** and +carries data between Databricks and your terminal or a file. + +That shape decides the threat model. There is no server to attack, no session to hijack, and no +credential store to exfiltrate. What is left is: the credential passing through, the data passing +through, the untrusted content coming back, and the artifact you installed. + +## Trust boundaries + +| Boundary | Trusted? | Why it matters | +|---|---|---| +| The user running the tool | Yes | They already hold the Databricks credential. LakeSpeak grants them nothing new. | +| The Databricks workspace | Partially | Trusted to authenticate and authorise. **Not** trusted for content: responses are model-generated and table-derived. | +| A Question Pack file | **No** | Can arrive in a pull request or a shared repository. Treated as untrusted input. | +| The local filesystem | Yes for reading config, **no** for write targets | Export paths are attacker-influenceable through packs. | +| The Databricks CLI binary | Yes | If it is compromised, so is `databricks` itself; nothing LakeSpeak does helps. | +| The terminal | Sink | Cannot validate what it renders, so LakeSpeak must not send it anything dangerous. | + +## Threats and what is done about them + +### T1 — Access token disclosure + +*A token reaches a log, a CI transcript, a bug report, or another user's process list.* + +The token is never persisted, never placed in a URL, and never passed as a process argument +(readable by other local users on most platforms). It is attached in exactly one place, a +`DelegatingHandler`, so no call site handles a raw token. + +Authorization headers, both Databricks signature fields (`download_id_signature`, +`statement_id_signature`), and token-shaped strings are redacted from all diagnostic output. +`GenieException` scrubs its own message at construction, so a token cannot escape through an +exception even if a call site forgets. + +A denylist cannot be complete, so it is the last line rather than the only one. + +**Residual risk:** a token supplied via `DATABRICKS_TOKEN` is visible in the environment of the +process and to anything that can read it. That is inherent to environment credentials. + +### T2 — Token disclosure to a third party + +*The Databricks bearer token is sent somewhere that is not Databricks.* + +Query results can arrive as presigned links to cloud storage. Sending the Databricks token to blob +storage would hand a credential to a third party. Those requests are issued **without** the +`Authorization` header. Databricks rejects such requests with HTTP 400, so the mistake would be +loud — but the client must not make it in the first place. + +`https` is enforced on the workspace host. A bearer token over plain HTTP is a disclosed token, and +the mistake is otherwise silent. + +### T3 — Command injection through a profile or Agent name + +*A crafted profile name in a config file or Question Pack executes a shell command.* + +The Databricks CLI is invoked with an argument vector and `UseShellExecute = false`. No +user-supplied value reaches a shell interpreter. There is no other subprocess execution anywhere in +the tool. + +### T4 — Terminal injection from response content + +*A table cell or a model-generated answer contains ANSI escapes.* + +Genie returns model output and cells drawn from your data. Both are untrusted for rendering: a +crafted value can move the cursor, clear the screen, recolour later output, or draw something +resembling this tool's own prompt to solicit input. + +Control characters are replaced at the rendering boundary, preserving only newline, carriage return +and tab. Sanitising once at the boundary is more reliable than auditing every write site. + +### T5 — Formula injection into a spreadsheet + +*An exported CSV opens in Excel and a cell executes.* + +A cell beginning `=`, `+`, `-` or `@` is evaluated as a formula by common spreadsheet software. Such +values are prefixed with a single quote on export — visibly, not silently. + +### T6 — Path traversal through a Question Pack + +*A pack writes its report outside its own directory.* + +Output paths are resolved against the pack's directory and rejected if they escape it or are +absolute. A pack can arrive in a pull request, so `../../.ssh/authorized_keys` is a realistic input +rather than a hypothetical. + +### T7 — Governed data leaking through artifacts + +*Query results end up somewhere they should not.* + +LakeSpeak writes nothing to disk except files you explicitly request, caches nothing, and copies no +conversation content into its config. Export warns that the file contains governed data and asks +before overwriting. Question Pack reports omit conversation and message ids by default, because +those identify a conversation containing governed data and reports get committed. + +**Residual risk, and it is the largest one.** Once you export a file or run in CI, the data is +where you put it. Job logs are usually readable by everyone with repository access. No client-side +control can fix that; the documentation says so plainly rather than implying protection. + +### T8 — Supply chain compromise + +*The package you install is not the code that was reviewed.* + +Dependencies are centrally pinned with lock files, and CI restores in `--locked-mode`, so a drifted +transitive version fails the build instead of shipping. A moderate-or-higher advisory fails CI. +GitHub Actions are pinned by commit SHA. Builds are deterministic. Releases carry an SBOM, SHA-256 +checksums and build provenance attestation. Publishing requires a protected environment, so it is a +decision rather than a side effect of pushing a tag. + +Secret scanning with push protection is enabled on the repository — it has already blocked one +commit, correctly, on a synthetic test fixture. + +### T9 — Untrusted pull requests reaching credentials + +*A fork PR runs live tests and exfiltrates workspace secrets.* + +Live tests are excluded from the default run by trait and never execute for pull requests. Jobs that +need workspace credentials are gated on the PR originating from this repository. + +### T10 — A wrong answer treated as authoritative + +*Someone acts on a plausible but incorrect number.* + +Not a security control, and listed here because it is the most likely real-world harm. + +Generated SQL can be wrong in ways that read as correct. LakeSpeak preserves the SQL, the result and +the message ids so a person can check, and every Question Pack report carries a non-configurable +warning. It cannot judge correctness, and the SOC 2 mapping says explicitly that no control makes a +generated answer true. + +## Out of scope + +- **A malicious Databricks workspace.** A workspace you authenticate to can return anything. + Response *shape* is validated; response *content* is trusted. +- **A compromised local machine.** If an attacker runs code as you, they have your credentials + regardless. +- **Denial of service.** LakeSpeak exposes no service. Warehouse capacity is a Databricks concern. +- **Unity Catalog correctness.** LakeSpeak cannot widen or narrow what your identity can see. + +## Reporting + +See [SECURITY.md](../../SECURITY.md). Report privately; do not open a public issue. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..4545588 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,124 @@ +# Troubleshooting + +Start with: + +```bash +lakespeak auth check +lakespeak config show +``` + +Between them they tell you which profile is in play, which workspace it resolves to, whether a +token can actually be obtained, and whether the workspace answers. Neither prints any part of a +credential, so both are safe to paste into an issue. + +## "Could not run the Databricks CLI" + +LakeSpeak brokers tokens through the Databricks CLI and could not start it. + +Install it, then log in: + +```bash +databricks auth login --profile company +``` + +If the CLI is installed but not on `PATH` for the process running LakeSpeak — common on Windows +after a `winget` install in an already-open shell — restart the shell. + +For CI, where there is usually no CLI at all, set `DATABRICKS_TOKEN` instead. See +[authentication](authentication.md). + +## "No Databricks host configured" + +Nothing supplied a workspace URL. Set one of, in precedence order: `DATABRICKS_HOST`, a `host` in +the named `.databrickscfg` profile, or a `host` in the `DEFAULT` profile. + +`lakespeak config show` reports which source won. + +## "Databricks rejected the credentials (HTTP 401)" + +The token expired or was never valid. For a CLI profile: + +```bash +databricks auth login --profile company +``` + +If you are supplying `DATABRICKS_TOKEN` yourself, it is probably stale — an Entra token lasts about +an hour. + +## "The authenticated identity is not permitted to do this (HTTP 403)" + +Your identity cannot use that Agent, or cannot read a table behind it. This is a Databricks grant, +and nothing LakeSpeak can change: it cannot widen your access. + +## "No Genie Agents are visible to this identity" + +Not an error — the call succeeded and returned nothing. Either no Genie Agents exist in the +workspace, or none are shared with your identity. Both are fixed in Databricks. + +Confirm you are pointed at the workspace you think you are with `lakespeak config show`. + +## "'x' matches N Agents" + +Two Agents share a title. LakeSpeak refuses to guess, because answering against the wrong Agent +looks exactly like success. Use the id from `lakespeak agents list`, or add an alias to your config +so scripts do not carry raw ids. + +## "Genie did not finish within Ns" + +The question exceeded the timeout. Usually a cold SQL warehouse — the first question of the day +can take minutes while it starts. + +Raise it per question in a pack (`timeout: 5m`) or in `defaults.timeout` in your config. The error +names the last state observed, which tells you whether it was waiting on a warehouse +(`PendingWarehouse`) or actually running a query (`ExecutingQuery`). + +## "The cached query result has expired" + +The message succeeded and its answer and SQL are still valid; only the cached result aged out. +Ask again to regenerate it. This is `QUERY_RESULT_EXPIRED`, which LakeSpeak deliberately does not +treat as a failure. + +## Output looks broken when piped + +Results go to stdout and diagnostics to stderr. If you are seeing progress messages mixed into +your data, you are capturing both: + +```bash +lakespeak ask --agent finance --format json "…" 2>/dev/null # data only +``` + +If you are seeing escape sequences, force plain output with `NO_COLOR=1`. LakeSpeak already +suppresses colour when output is redirected or the format is machine-readable. + +## Non-ASCII shows as `?` in my terminal + +The tool writes UTF-8 and pins its output encoding. A `?` almost always means the terminal or the +capture is on a legacy code page. Redirect to a file and inspect it — the bytes are usually fine. + +## `pack validate` fails and I fixed one thing + +It reports every problem at once, on purpose. Read the whole list before re-running. + +## "spec.output.path resolves outside the pack directory" + +Report paths are relative to the pack file, and traversal is rejected. A pack can arrive from a +pull request, so this is a guard rather than an inconvenience. Move the output inside the pack's +directory. + +## CI fails with NU1004 after I added a dependency + +The lock files no longer match the project graph. Regenerate and commit them: + +```bash +dotnet restore --force-evaluate +``` + +CI restores in `--locked-mode` so a drifted transitive version fails the build rather than silently +resolving differently on the runner. + +## Still stuck + +Open an issue with what you ran, what happened, and what you expected. Include +`lakespeak --verbose` output — credentials are redacted, but **read it first**: questions, answers +and query results can contain your organisation's data, and LakeSpeak cannot know which of your +table names are sensitive. diff --git a/src/LakeSpeak.Cli/Program.cs b/src/LakeSpeak.Cli/Program.cs index c651110..8e5bf4d 100644 --- a/src/LakeSpeak.Cli/Program.cs +++ b/src/LakeSpeak.Cli/Program.cs @@ -7,6 +7,18 @@ internal static class Program { internal static async Task Main(string[] args) { + // Windows consoles default to a legacy code page that cannot represent most non-ASCII + // characters, so a currency symbol or a non-Latin table value degrades to '?' on its way + // out. That is silent corruption of machine-readable output, so the encoding is pinned + // rather than inherited. Guarded because a redirected or closed handle throws here. + try + { + System.Console.OutputEncoding = System.Text.Encoding.UTF8; + } + catch (IOException) + { + } + var root = new RootCommand( "LakeSpeak.NET — talk to governed Databricks data from your terminal.\n" + "An independent open-source project, not affiliated with Databricks."); diff --git a/src/LakeSpeak.Genie/ServiceCollectionExtensions.cs b/src/LakeSpeak.Genie/ServiceCollectionExtensions.cs index bb042d3..5287354 100644 --- a/src/LakeSpeak.Genie/ServiceCollectionExtensions.cs +++ b/src/LakeSpeak.Genie/ServiceCollectionExtensions.cs @@ -37,6 +37,16 @@ public static IServiceCollection AddLakeSpeak( services.TryAddSingleton(sp => { var options = sp.GetRequiredService>().Value; + + // DATABRICKS_TOKEN wins when set. Unattended environments — CI, a container, a + // scheduled Question Pack — have no browser and often no Databricks CLI, so the + // environment has to be a real path rather than a documented one that silently + // falls through to a CLI that is not installed. + if (Environment.GetEnvironmentVariable(EnvironmentTokenProvider.TokenVariable) is { Length: > 0 }) + { + return new EnvironmentTokenProvider(); + } + return new DatabricksCliTokenProvider(options.Profile); }); diff --git a/tests/LakeSpeak.Genie.Tests/LakeSpeak.Genie.Tests.csproj b/tests/LakeSpeak.Genie.Tests/LakeSpeak.Genie.Tests.csproj index e2131b4..7bcef65 100644 --- a/tests/LakeSpeak.Genie.Tests/LakeSpeak.Genie.Tests.csproj +++ b/tests/LakeSpeak.Genie.Tests/LakeSpeak.Genie.Tests.csproj @@ -5,6 +5,7 @@ + diff --git a/tests/LakeSpeak.Genie.Tests/OutputFidelityTests.cs b/tests/LakeSpeak.Genie.Tests/OutputFidelityTests.cs new file mode 100644 index 0000000..b54e906 --- /dev/null +++ b/tests/LakeSpeak.Genie.Tests/OutputFidelityTests.cs @@ -0,0 +1,137 @@ +using LakeSpeak.Genie; +using LakeSpeak.Rendering; + +namespace LakeSpeak.Genie.Tests; + +/// +/// Values must reach the output byte-for-byte as Databricks returned them. These tests exist +/// because the failure mode is silent: a reformatted number or a mangled character still looks +/// like a working report. +/// +public class OutputFidelityTests +{ + private static GenieQueryResult Result(params (string Name, string? Value)[] cells) => + new( + cells.Select(c => new GenieColumn(c.Name, "STRING", "STRING")).ToList(), + [cells.Select(c => c.Value).ToList()], + IsTruncated: false, + TotalRowCount: 1); + + [Theory] + [InlineData("4500000.00")] + [InlineData("3350000.50")] + [InlineData("0.000000000000000001")] + [InlineData("-0.0")] + [InlineData("1E+40")] + [InlineData("99999999999999999999999999.99")] + public void Csv_carries_numbers_through_unchanged(string value) + { + var csv = CsvWriter.Write(Result(("amount", value))); + + // Not 4.5E6, not 4,500,000.00, not 4500000. Parsing a DECIMAL in order to print it is + // how a client silently changes someone's revenue figure. + csv.ShouldContain(value); + } + + [Theory] + [InlineData("€2.4M")] + [InlineData("Ökonomie")] + [InlineData("東京")] + [InlineData("Ω≈ç√∫")] + [InlineData("emoji 🙂 in a cell")] + public void Non_ascii_survives_every_writer(string value) + { + var result = Result(("label", value)); + var response = new GenieResponse( + "agent", "conversation", "message", GenieMessageState.Completed, + value, null, result, [], new GenieResponseMetadata(TimeSpan.Zero, 1)); + + CsvWriter.Write(result).ShouldContain(value); + MarkdownWriter.Write(response).ShouldContain(value); + TerminalSafety.Sanitize(value).ShouldBe(value); + + // JSON is asserted after a round trip rather than by substring. Characters outside the + // BMP are legitimately written as escaped surrogate pairs, which no consumer ever sees + // because every parser decodes them — so a substring check would fail on correct output. + using var parsed = System.Text.Json.JsonDocument.Parse(MachineOutput.ToJson(response)); + parsed.RootElement.GetProperty("answer").GetString().ShouldBe(value); + } + + // A SQL NULL and the empty string are different values, and a format that renders them + // identically loses information the caller cannot recover. + [Fact] + public void Csv_distinguishes_null_from_empty_string() + { + var csv = CsvWriter.Write(Result(("a", null), ("b", string.Empty))); + var dataLine = csv.Split('\n')[1].TrimEnd('\r'); + + dataLine.ShouldBe(","); + } + + [Theory] + [InlineData("=1+1")] + [InlineData("+1")] + [InlineData("-1+1")] + [InlineData("@SUM(A1)")] + public void Csv_defuses_spreadsheet_formulas(string value) + { + var csv = CsvWriter.Write(Result(("payload", value))); + + // A leading =, +, - or @ makes Excel and Sheets evaluate the cell. The value is still + // present and readable; it just cannot execute. + csv.ShouldContain($"\"'{value}\""); + } + + [Fact] + public void Csv_quotes_values_containing_delimiters_and_quotes() + { + var csv = CsvWriter.Write(Result(("a", "has,comma"), ("b", "has\"quote"))); + + csv.ShouldContain("\"has,comma\""); + csv.ShouldContain("\"has\"\"quote\""); + } + + // Genie returns model-generated prose and cells drawn from your tables. A crafted value must + // not be able to move the cursor or draw something resembling this tool's own prompt. + [Theory] + [InlineData("cleared")] + [InlineData("bell")] + [InlineData("carriage\rreturn")] + public void Control_characters_are_neutralised(string value) + { + var sanitized = TerminalSafety.SanitizeCell(value); + + sanitized.ShouldNotContain(""); + sanitized.ShouldNotContain(""); + sanitized.ShouldNotContain("\r"); + } + + [Fact] + public void Newline_and_tab_survive_sanitising_prose() + { + const string prose = "line one\nline two\tcolumn"; + + TerminalSafety.Sanitize(prose).ShouldBe(prose); + } + + [Fact] + public void Markdown_escapes_pipes_so_a_cell_cannot_break_the_table() + { + var markdown = MarkdownWriter.Write(new GenieResponse( + "a", "c", "m", GenieMessageState.Completed, null, null, + Result(("col", "a|b")), [], new GenieResponseMetadata(TimeSpan.Zero, 1))); + + markdown.ShouldContain("a\\|b"); + } + + [Fact] + public void Json_uses_a_stable_lowercase_status_vocabulary() + { + var json = MachineOutput.ToJson(new GenieResponse( + "a", "c", "m", GenieMessageState.QueryResultExpired, null, null, null, [], + new GenieResponseMetadata(TimeSpan.Zero, 1))); + + json.ShouldContain("\"status\": \"queryresultexpired\""); + json.ShouldContain("\"schemaVersion\": \"1\""); + } +} diff --git a/tests/LakeSpeak.Genie.Tests/packages.lock.json b/tests/LakeSpeak.Genie.Tests/packages.lock.json index 8d643a6..165db34 100644 --- a/tests/LakeSpeak.Genie.Tests/packages.lock.json +++ b/tests/LakeSpeak.Genie.Tests/packages.lock.json @@ -1237,6 +1237,11 @@ "resolved": "1.0.5", "contentHash": "LaSDYOJDh2WncgRboqiWtk/Igqoim/LV7v808qBeWY/f36Ol5oEKguEYpKrWw5ap8KYP0SRXf7/v3zil9koY6Q==" }, + "Spectre.Console.Ansi": { + "type": "Transitive", + "resolved": "0.57.2", + "contentHash": "Y1+u73shwP+JYHmrkdN6bSt2kaGBEAafhi2nrecczbXdFRey+k7J/WF9YPfcB0rqivv2S8fmif6FSBq9m/AFvw==" + }, "Stef.Validation": { "type": "Transitive", "resolved": "0.3.0", @@ -1479,6 +1484,13 @@ "Microsoft.Extensions.Options": "[10.0.10, )" } }, + "lakespeak.rendering": { + "type": "Project", + "dependencies": { + "LakeSpeak.Genie": "[0.1.0, )", + "Spectre.Console": "[0.57.2, )" + } + }, "Microsoft.Extensions.Configuration": { "type": "CentralTransitive", "requested": "[10.0.10, )", @@ -1558,6 +1570,15 @@ "Microsoft.Extensions.Primitives": "10.0.10" } }, + "Spectre.Console": { + "type": "CentralTransitive", + "requested": "[0.57.2, )", + "resolved": "0.57.2", + "contentHash": "wzsB+P9i6F9G+cFOJxRotCiQWNSzZVAMxi8YMiHlGXXU1JLMXGp7zSSqBMmROZS2bzpK5jk+Saa7Evp8HwqhKg==", + "dependencies": { + "Spectre.Console.Ansi": "0.57.2" + } + }, "YamlDotNet": { "type": "CentralTransitive", "requested": "[18.1.0, )",