diff --git a/CHANGELOG.md b/CHANGELOG.md index d73886d..519bfac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,17 @@ here rather than left to be discovered. - The README no longer states a test count. The number had drifted from 89 to 175 without anyone noticing, which is what prose claims do. +- **Positioning corrected.** Databricks CLI v1.10.0 ships `databricks genie ask`, which holds a + conversation across calls, shows SQL and prints JSON. The README previously argued that this + project exists because the official CLI makes you manage six identifiers by hand; that argument + no longer holds. It now leads with the two things that do — there is no Databricks SDK for .NET, + and Question Packs have no equivalent — and points readers at the official command for a plain + terminal answer. `ROADMAP.md` named this exact event as one that would narrow the differentiator. +- **ADR 0001 corrected**, without changing its decision. It rejected an MCP server partly on the + grounds that one would duplicate Databricks' managed endpoints. Those endpoints are stateless, so + a stateful implementation would not duplicate them. The decision stands on scope instead, which + was always the real reason. `GOVERNANCE.md` and `ROADMAP.md` updated to match. +- NuGet version badges for both packages, so the README shows what is actually published. ## 0.1.0-preview.1 — 2026-08-01 diff --git a/GOVERNANCE.md b/GOVERNANCE.md index 2d7e7ad..53ec04a 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -40,9 +40,13 @@ rejected for being outside it. **Deferred rather than rejected** — reconsidered once there are real users asking: -- An MCP server mode. Databricks already ships managed MCP endpoints for Genie; a LakeSpeak MCP - server is only justified if it exposes something they do not, such as Question Packs. Re-exposing - `ask` over MCP is not a reason. +- An MCP server mode. Databricks ships managed MCP endpoints for Genie, and those are **stateless** + — every question starts over — so a stateful LakeSpeak MCP server, or one exposing Question + Packs, would offer something they do not. That makes this a scope decision rather than a + duplication one: an MCP server is a second product with its own transport and support burden. It + is deferred until a real user asks, not rejected. Re-exposing `ask` alone would still not be a + reason. See [ADR 0001](docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md) + and its correction. - Multi-Agent orchestration. ## New dependencies diff --git a/README.md b/README.md index 45a9e5b..4991ea6 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # LakeSpeak.NET [![CI](https://github.com/ivanvyd/LakeSpeak.NET/actions/workflows/ci.yml/badge.svg)](https://github.com/ivanvyd/LakeSpeak.NET/actions/workflows/ci.yml) +[![LakeSpeak.Cli](https://img.shields.io/nuget/v/LakeSpeak.Cli?label=LakeSpeak.Cli)](https://www.nuget.org/packages/LakeSpeak.Cli/) +[![LakeSpeak.Genie](https://img.shields.io/nuget/v/LakeSpeak.Genie?label=LakeSpeak.Genie)](https://www.nuget.org/packages/LakeSpeak.Genie/) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/ivanvyd/LakeSpeak.NET/badge)](https://scorecard.dev/viewer/?uri=github.com/ivanvyd/LakeSpeak.NET) @@ -35,7 +37,7 @@ dotnet tool install --global LakeSpeak.Cli --prerelease [Databricks Genie](https://docs.databricks.com/aws/en/genie/) answers questions about your data in plain English: you ask, it writes SQL against tables someone has curated, runs it on a SQL warehouse, and answers. A **Genie Agent** is one such configured surface. Genie normally lives in -the Databricks web UI — LakeSpeak puts it in your terminal, your scripts, and your .NET code, while +the Databricks web UI — LakeSpeak puts it in your .NET code, your scripts and your terminal, while keeping the generated SQL visible so you can check the answer. You need a Genie Agent to already exist in your workspace and be shared with you. LakeSpeak cannot @@ -43,20 +45,30 @@ create one, and sees only what your own Databricks identity can see. ## Why this exists -The Genie Conversation API is capable, and the official `databricks genie` CLI exposes it. But using -it means managing Agent ids, conversation ids, message ids, attachment ids, a polling loop, and a -two-step download workflow yourself: +Start with what you may not need this for. The official CLI has a good terminal experience for +Genie: ```bash -databricks genie list-spaces -databricks genie start-conversation "question" -databricks genie get-message -databricks genie get-message-attachment-query-result ... -databricks genie generate-download-full-query-result ... -databricks genie get-download-full-query-result ... +databricks genie ask -s sales --include-sql "How did revenue change last month?" ``` -LakeSpeak turns that into: +That holds a conversation across calls, shows the SQL, and prints JSON with `-o json`. If a terminal +answer is all you want, it ships with the Databricks CLI and it is fine. + +Two things it does not cover. + +**There is no Databricks SDK for .NET.** Python, Java, Go and R are covered; .NET is not. A .NET +service that needs a Genie answer in-process has no first-party option, and shelling out to a CLI +from a hosted service is not one. `LakeSpeak.Genie` is a typed client with DI registration, +cancellation, typed failures, and no credential of its own. + +**Question Packs have no equivalent.** A set of business questions, version-controlled, reviewed +like code, run on a schedule, producing a deterministic Markdown report. That is a different +artifact from a terminal command, and nothing else produces it. + +The terminal client exists because the library needed proving and because `--agent` with +[configured aliases](docs/configuration.md) targets a named Agent, which the official `ask` exposes +no flag for. ```bash lakespeak ask --agent sales "How did revenue change last month?" @@ -66,11 +78,12 @@ lakespeak ask --agent sales "How did revenue change last month?" The generated SQL is shown because the answer is only as trustworthy as the query behind it. -It is deliberately narrow. It is not a Databricks SDK for .NET, not a replacement for the official -CLI, and not another Genie MCP server — Databricks already ships [managed MCP endpoints for -Genie](https://docs.databricks.com/aws/en/generative-ai/mcp/), and duplicating them would add -nothing. What is missing is a good product experience for stateful conversations, and that is the -gap this fills. See [docs/decisions](docs/decisions/) for the reasoning. +It is deliberately narrow. It is not a Databricks SDK for .NET and not a replacement for the +official CLI. It is also not a Genie MCP server: Databricks ships [managed MCP endpoints for +Genie](https://docs.databricks.com/aws/en/generative-ai/mcp/), and although those are stateless — +every question starts over — building a stateful one would be a second product, and this is +deliberately one. See [ADR 0001](docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md), +including its correction. ## Install diff --git a/ROADMAP.md b/ROADMAP.md index dccdcd1..0294aab 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -53,14 +53,26 @@ Status: in development. ## Deliberately deferred -An MCP server mode. Databricks already ships managed MCP endpoints for Genie, and re-exposing `ask` -over MCP would add nothing. It becomes worth reconsidering only if LakeSpeak has something they do -not — Question Packs being the obvious candidate. +An MCP server mode. Databricks' managed Genie MCP endpoints are stateless, so a stateful LakeSpeak +one — or one exposing Question Packs — would offer something they do not. It is deferred on scope +rather than on duplication: a second product with its own transport and support burden, for one +maintainer. Reconsidered when a real user asks. See +[ADR 0001](docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md). + +## One of these already happened + +**The official CLI became conversational.** Databricks CLI v1.10.0 ships `databricks genie ask`, +which holds a conversation across calls with `-s`, shows SQL with `--include-sql`, and prints JSON. +This page previously listed that as a hypothetical that would narrow the differentiator to Question +Packs and the .NET library. It has occurred, and the plan followed: the README now leads with those +two and points readers at the official command for a plain terminal answer. + +Recorded here rather than quietly deleted, because a project that predicts something, is right, and +then says nothing has stopped paying attention. ## Things that would change the plan - **An official Databricks .NET SDK.** `LakeSpeak.Genie` would migrate onto it rather than compete - with it, and this project would keep only the product layer. -- **The official CLI becoming conversational.** The differentiator would narrow to Question Packs - and the .NET library, and the roadmap would follow. + with it, and this project would keep only the product layer. This is the remaining existential + one. - **Real users.** Everything above is a guess about what people want. Issues beat guesses. diff --git a/docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md b/docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md index afeb44d..bdeb316 100644 --- a/docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md +++ b/docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md @@ -45,3 +45,27 @@ authentication story. An MCP mode becomes worth reconsidering only when LakeSpeak has something the official endpoints do not, such as Question Packs. Re-exposing `ask` over MCP would not qualify. This is recorded in `GOVERNANCE.md` as deferred rather than rejected. + +## Correction — 2026-08-05 + +The decision stands. Two of the premises above do not, and leaving them uncorrected would let a +future reader inherit reasoning that has since been falsified. + +**"An MCP server for Genie would duplicate something Databricks maintains" is wrong.** Databricks' +managed Genie MCP endpoints invoke Genie as a stateless tool: conversation history is not carried +between calls, so every `query_space` call is a brand-new question. A stateful MCP server would not +duplicate theirs. It would do the one thing theirs cannot. + +**The second premise confuses the protocol with one implementation of it.** The context above uses +Genie-as-an-MCP-tool not carrying history to conclude that "MCP is the lossy one". That is a +property of Databricks' server, not of the protocol. Nothing in MCP prevents a stateful +implementation, and this project's own conversation handling is the hard part of building one. + +The decision is unchanged because the honest reason was never duplication — it is scope. An MCP +server is a second product with its own transport, surface and support burden, attached to a +project one person maintains. The trigger for reconsidering it, per `GOVERNANCE.md`, is a real user +asking. That has not happened. + +One further fact bearing on the "coding agents are served through the CLI's JSON output" +consequence: the official CLI now ships `databricks genie ask`, which is stateful across calls via +`-s` and emits JSON. The consequence still holds. It is simply no longer unique to LakeSpeak. diff --git a/docs/getting-started.md b/docs/getting-started.md index a31c5a6..ca20ffc 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -11,8 +11,10 @@ curated for it, runs that SQL on a SQL warehouse, and answers. A **Genie Agent** still calls it a *space* — is one such configured question-answering surface: a set of tables, some instructions, and a warehouse to run on. -Genie normally lives inside the Databricks web UI. **LakeSpeak puts it in your terminal and in your -.NET code.** That is the whole idea. +Genie normally lives inside the Databricks web UI. **LakeSpeak puts it in your .NET code, and in +your terminal.** The .NET half is the part nothing else does — Databricks publishes SDKs for +Python, Java, Go and R, but not for .NET. For a terminal answer alone, `databricks genie ask` ships +with the Databricks CLI and is worth trying first. ``` You LakeSpeak Databricks Genie