Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,17 @@ here rather than left to be discovered.

- The README no longer states a test count. The number had drifted from 89 to 175 without anyone
noticing, which is what prose claims do.
- **Positioning corrected.** Databricks CLI v1.10.0 ships `databricks genie ask`, which holds a
conversation across calls, shows SQL and prints JSON. The README previously argued that this
project exists because the official CLI makes you manage six identifiers by hand; that argument
no longer holds. It now leads with the two things that do — there is no Databricks SDK for .NET,
and Question Packs have no equivalent — and points readers at the official command for a plain
terminal answer. `ROADMAP.md` named this exact event as one that would narrow the differentiator.
- **ADR 0001 corrected**, without changing its decision. It rejected an MCP server partly on the
grounds that one would duplicate Databricks' managed endpoints. Those endpoints are stateless, so
a stateful implementation would not duplicate them. The decision stands on scope instead, which
was always the real reason. `GOVERNANCE.md` and `ROADMAP.md` updated to match.
- NuGet version badges for both packages, so the README shows what is actually published.

## 0.1.0-preview.1 — 2026-08-01

Expand Down
10 changes: 7 additions & 3 deletions GOVERNANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,13 @@ rejected for being outside it.

**Deferred rather than rejected** — reconsidered once there are real users asking:

- An MCP server mode. Databricks already ships managed MCP endpoints for Genie; a LakeSpeak MCP
server is only justified if it exposes something they do not, such as Question Packs. Re-exposing
`ask` over MCP is not a reason.
- An MCP server mode. Databricks ships managed MCP endpoints for Genie, and those are **stateless**
— every question starts over — so a stateful LakeSpeak MCP server, or one exposing Question
Packs, would offer something they do not. That makes this a scope decision rather than a
duplication one: an MCP server is a second product with its own transport and support burden. It
is deferred until a real user asks, not rejected. Re-exposing `ask` alone would still not be a
reason. See [ADR 0001](docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md)
and its correction.
- Multi-Agent orchestration.

## New dependencies
Expand Down
45 changes: 29 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# LakeSpeak.NET

[![CI](https://github.com/ivanvyd/LakeSpeak.NET/actions/workflows/ci.yml/badge.svg)](https://github.com/ivanvyd/LakeSpeak.NET/actions/workflows/ci.yml)
[![LakeSpeak.Cli](https://img.shields.io/nuget/v/LakeSpeak.Cli?label=LakeSpeak.Cli)](https://www.nuget.org/packages/LakeSpeak.Cli/)
[![LakeSpeak.Genie](https://img.shields.io/nuget/v/LakeSpeak.Genie?label=LakeSpeak.Genie)](https://www.nuget.org/packages/LakeSpeak.Genie/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/ivanvyd/LakeSpeak.NET/badge)](https://scorecard.dev/viewer/?uri=github.com/ivanvyd/LakeSpeak.NET)

Expand Down Expand Up @@ -35,28 +37,38 @@ dotnet tool install --global LakeSpeak.Cli --prerelease
[Databricks Genie](https://docs.databricks.com/aws/en/genie/) answers questions about your data in
plain English: you ask, it writes SQL against tables someone has curated, runs it on a SQL
warehouse, and answers. A **Genie Agent** is one such configured surface. Genie normally lives in
the Databricks web UI — LakeSpeak puts it in your terminal, your scripts, and your .NET code, while
the Databricks web UI — LakeSpeak puts it in your .NET code, your scripts and your terminal, while
keeping the generated SQL visible so you can check the answer.

You need a Genie Agent to already exist in your workspace and be shared with you. LakeSpeak cannot
create one, and sees only what your own Databricks identity can see.

## Why this exists

The Genie Conversation API is capable, and the official `databricks genie` CLI exposes it. But using
it means managing Agent ids, conversation ids, message ids, attachment ids, a polling loop, and a
two-step download workflow yourself:
Start with what you may not need this for. The official CLI has a good terminal experience for
Genie:

```bash
databricks genie list-spaces
databricks genie start-conversation <space-id> "question"
databricks genie get-message <space-id> <conversation-id> <message-id>
databricks genie get-message-attachment-query-result ...
databricks genie generate-download-full-query-result ...
databricks genie get-download-full-query-result ...
databricks genie ask -s sales --include-sql "How did revenue change last month?"
```

LakeSpeak turns that into:
That holds a conversation across calls, shows the SQL, and prints JSON with `-o json`. If a terminal
answer is all you want, it ships with the Databricks CLI and it is fine.

Two things it does not cover.

**There is no Databricks SDK for .NET.** Python, Java, Go and R are covered; .NET is not. A .NET
service that needs a Genie answer in-process has no first-party option, and shelling out to a CLI
from a hosted service is not one. `LakeSpeak.Genie` is a typed client with DI registration,
cancellation, typed failures, and no credential of its own.

**Question Packs have no equivalent.** A set of business questions, version-controlled, reviewed
like code, run on a schedule, producing a deterministic Markdown report. That is a different
artifact from a terminal command, and nothing else produces it.

The terminal client exists because the library needed proving and because `--agent` with
[configured aliases](docs/configuration.md) targets a named Agent, which the official `ask` exposes
no flag for.

```bash
lakespeak ask --agent sales "How did revenue change last month?"
Expand All @@ -66,11 +78,12 @@ lakespeak ask --agent sales "How did revenue change last month?"

The generated SQL is shown because the answer is only as trustworthy as the query behind it.

It is deliberately narrow. It is not a Databricks SDK for .NET, not a replacement for the official
CLI, and not another Genie MCP server — Databricks already ships [managed MCP endpoints for
Genie](https://docs.databricks.com/aws/en/generative-ai/mcp/), and duplicating them would add
nothing. What is missing is a good product experience for stateful conversations, and that is the
gap this fills. See [docs/decisions](docs/decisions/) for the reasoning.
It is deliberately narrow. It is not a Databricks SDK for .NET and not a replacement for the
official CLI. It is also not a Genie MCP server: Databricks ships [managed MCP endpoints for
Genie](https://docs.databricks.com/aws/en/generative-ai/mcp/), and although those are stateless —
every question starts over — building a stateful one would be a second product, and this is
deliberately one. See [ADR 0001](docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md),
including its correction.

## Install

Expand Down
24 changes: 18 additions & 6 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,26 @@ Status: in development.

## Deliberately deferred

An MCP server mode. Databricks already ships managed MCP endpoints for Genie, and re-exposing `ask`
over MCP would add nothing. It becomes worth reconsidering only if LakeSpeak has something they do
not — Question Packs being the obvious candidate.
An MCP server mode. Databricks' managed Genie MCP endpoints are stateless, so a stateful LakeSpeak
one — or one exposing Question Packs — would offer something they do not. It is deferred on scope
rather than on duplication: a second product with its own transport and support burden, for one
maintainer. Reconsidered when a real user asks. See
[ADR 0001](docs/decisions/0001-a-cli-and-a-library-rather-than-another-mcp-server.md).

## One of these already happened

**The official CLI became conversational.** Databricks CLI v1.10.0 ships `databricks genie ask`,
which holds a conversation across calls with `-s`, shows SQL with `--include-sql`, and prints JSON.
This page previously listed that as a hypothetical that would narrow the differentiator to Question
Packs and the .NET library. It has occurred, and the plan followed: the README now leads with those
two and points readers at the official command for a plain terminal answer.

Recorded here rather than quietly deleted, because a project that predicts something, is right, and
then says nothing has stopped paying attention.

## Things that would change the plan

- **An official Databricks .NET SDK.** `LakeSpeak.Genie` would migrate onto it rather than compete
with it, and this project would keep only the product layer.
- **The official CLI becoming conversational.** The differentiator would narrow to Question Packs
and the .NET library, and the roadmap would follow.
with it, and this project would keep only the product layer. This is the remaining existential
one.
- **Real users.** Everything above is a guess about what people want. Issues beat guesses.
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,27 @@ authentication story.
An MCP mode becomes worth reconsidering only when LakeSpeak has something the official endpoints do
not, such as Question Packs. Re-exposing `ask` over MCP would not qualify. This is recorded in
`GOVERNANCE.md` as deferred rather than rejected.

## Correction — 2026-08-05

The decision stands. Two of the premises above do not, and leaving them uncorrected would let a
future reader inherit reasoning that has since been falsified.

**"An MCP server for Genie would duplicate something Databricks maintains" is wrong.** Databricks'
managed Genie MCP endpoints invoke Genie as a stateless tool: conversation history is not carried
between calls, so every `query_space` call is a brand-new question. A stateful MCP server would not
duplicate theirs. It would do the one thing theirs cannot.

**The second premise confuses the protocol with one implementation of it.** The context above uses
Genie-as-an-MCP-tool not carrying history to conclude that "MCP is the lossy one". That is a
property of Databricks' server, not of the protocol. Nothing in MCP prevents a stateful
implementation, and this project's own conversation handling is the hard part of building one.

The decision is unchanged because the honest reason was never duplication — it is scope. An MCP
server is a second product with its own transport, surface and support burden, attached to a
project one person maintains. The trigger for reconsidering it, per `GOVERNANCE.md`, is a real user
asking. That has not happened.

One further fact bearing on the "coding agents are served through the CLI's JSON output"
consequence: the official CLI now ships `databricks genie ask`, which is stateful across calls via
`-s` and emits JSON. The consequence still holds. It is simply no longer unique to LakeSpeak.
6 changes: 4 additions & 2 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ curated for it, runs that SQL on a SQL warehouse, and answers. A **Genie Agent**
still calls it a *space* — is one such configured question-answering surface: a set of tables, some
instructions, and a warehouse to run on.

Genie normally lives inside the Databricks web UI. **LakeSpeak puts it in your terminal and in your
.NET code.** That is the whole idea.
Genie normally lives inside the Databricks web UI. **LakeSpeak puts it in your .NET code, and in
your terminal.** The .NET half is the part nothing else does — Databricks publishes SDKs for
Python, Java, Go and R, but not for .NET. For a terminal answer alone, `databricks genie ask` ships
with the Databricks CLI and is worth trying first.

```
You LakeSpeak Databricks Genie
Expand Down
Loading