Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,14 @@ on:
tags: ['v*']
workflow_dispatch:
inputs:
dry-run:
description: 'Build and attest, but do not publish to NuGet'
version:
description: 'Version to build, without the leading v (e.g. 0.1.0). Leave blank for a dev build.'
type: string
required: false
publish:
description: 'Publish to NuGet. Leave unticked to build, test and attest without releasing.'
type: boolean
default: true
default: false

permissions:
contents: read
Expand Down Expand Up @@ -36,7 +40,16 @@ jobs:
run: |
if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
v="${GITHUB_REF_NAME#v}"
elif [ -n "${{ inputs.version }}" ]; then
v="${{ inputs.version }}"
else
# Placeholder for a rehearsal run. Publishing this would put a permanent
# 0.0.0-dev package on nuget.org, which cannot be unpublished — so refuse
# rather than let a ticked box and an empty field combine into a release.
if [ "${{ inputs.publish }}" = "true" ]; then
echo "::error::'publish' was ticked but 'version' is empty. Supply an explicit version to publish." >&2
exit 1
fi
v="0.0.0-dev.${GITHUB_RUN_NUMBER}"
fi
echo "version=$v" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -85,7 +98,11 @@ jobs:

publish:
needs: build
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
# A tag push publishes. A manual run publishes only when explicitly ticked, so the default
# dispatch is a safe rehearsal that builds, tests and attests without releasing anything.
if: >-
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v'))
|| (github.event_name == 'workflow_dispatch' && inputs.publish)
runs-on: ubuntu-latest
# A protected environment turns publishing into a decision someone makes,
# rather than a side effect of pushing a tag.
Expand Down Expand Up @@ -114,7 +131,9 @@ jobs:
--source https://api.nuget.org/v3/index.json \
--skip-duplicate

# Only a tag has a release to attach artifacts to; a manual publish has no ref to name.
- uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3.0.2
if: startsWith(github.ref, 'refs/tags/v')
with:
files: |
artifacts/*.zip
Expand Down
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,39 @@ here rather than left to be discovered.
- **CI** covering build, cross-platform tests, packing, tool install, CodeQL, Scorecard, secret
scanning, dependency review, SBOM and build provenance attestation.

### Fixed

- `auth check` constructed the Databricks CLI token provider directly instead of using the one the
client actually resolves. With `DATABRICKS_TOKEN` set and no Databricks CLI installed — the
unattended setup the environment provider exists for — it reported failure for a working
configuration. It now tests the resolved provider and names which one answered.
- A manual release run with `publish` ticked and `version` left blank would have pushed a
placeholder `0.0.0-dev.<n>` to nuget.org permanently, despite `RELEASING.md` stating that
version is never published. The workflow now refuses that combination.
- Resolving an Agent by id paged the whole Agent listing to find it, costing one round trip per
page for the form scripts and Question Packs are told to prefer. Ids are now fetched directly,
gated on the id shape so resolving by name is unaffected.
- The published Question Pack schema offered `output.format: json`, which the loader rejects.
The schema now matches the loader.
- The `/quit` chat alias worked but appeared in neither `/help` nor the docs.
- Live integration tests reported **failure** rather than **skip** when no workspace was
configured, so a bare `dotnet test` gave a contributor without Databricks access 8 red tests.
They are now gated with `[Fact(SkipUnless = …)]`, which xunit evaluates before constructing the
class; throwing from the constructor never skipped.
- The release workflow declared a `dry-run` input it never read, and gated publishing on a tag
push — so a manual run could not publish and the toggle did nothing. Replaced with `version` and
`publish` inputs that are actually consulted.

### Documentation

- **`docs/getting-started.md`** — a zero-to-answer walkthrough covering what Databricks Genie is,
that a Genie Agent must already exist and be shared with you, scripting, and the .NET library.
- **`docs/configuration.md`** — the config file's location per platform, its full schema, alias
precedence and resolution order. Previously the file was referenced across several documents but
described in none, so its path and YAML shape could only be learned from the source.
- **`RELEASING.md`** — how a release is cut, how to rehearse one without publishing, and what
cannot be undone.

### Notes

Verified against a live Azure Databricks workspace on 2026-08-01: `agents list`, `ask`, every
Expand Down
19 changes: 19 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,25 @@ dotnet test -c Release --filter "Category!=Live"
The default test run needs no Databricks workspace and no credentials. That is deliberate: a
contributor should be able to make a change and prove it without an account or a bill.

## How the solution is laid out

Two projects ship to NuGet. The rest are internal to the CLI tool and are bundled into it, which
is why they are not separate packages — every extra public package is an API to support forever.

| Project | Ships? | What it is |
|---|---|---|
| `LakeSpeak.Genie` | **NuGet** | The client: wire contracts, polling, attachments, auth, typed failures |
| `LakeSpeak.Cli` | **NuGet** (dotnet tool) | Commands, console output, the `lakespeak` entry point |
| `LakeSpeak.Configuration` | internal | The config file, agent aliases, the last-answer pointer |
| `LakeSpeak.Application` | internal | Agent resolution — turning what you typed into one Agent |
| `LakeSpeak.Rendering` | internal | Terminal tables, CSV, Markdown, JSON, control-character safety |
| `LakeSpeak.QuestionPacks` | internal | Pack schema, validation, runner, report writer |

The wire/domain split matters: everything in `LakeSpeak.Genie/Wire/` is `internal` and mirrors the
Databricks response shapes exactly, including `space_id`. The public surface uses Agent
terminology. That translation happens once, at the serialization boundary — see
[`docs/planning/genie-api-surface.md`](docs/planning/genie-api-surface.md).

## Before opening a pull request

```bash
Expand Down
4 changes: 4 additions & 0 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@
attestation in .github/workflows/release.yml to mean anything. -->
<Deterministic>true</Deterministic>
<ContinuousIntegrationBuild Condition="'$(GITHUB_ACTIONS)' == 'true'">true</ContinuousIntegrationBuild>
<!-- No Microsoft.SourceLink.GitHub reference is needed: the .NET SDK has bundled Source Link
since .NET 8, so these two properties are live, not inert. Verified by packing and
reading the emitted .pdb, which carries a commit-pinned raw.githubusercontent.com URL. -->
<PublishRepositoryUrl>true</PublishRepositoryUrl>
<EmbedUntrackedSources>true</EmbedUntrackedSources>
<IncludeSymbols>true</IncludeSymbols>
Expand All @@ -54,6 +57,7 @@
<PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
<PackageProjectUrl>https://github.com/ivanvyd/lakespeak</PackageProjectUrl>
<PackageReadmeFile>README.md</PackageReadmeFile>
<PackageReleaseNotes>https://github.com/ivanvyd/lakespeak/blob/main/CHANGELOG.md</PackageReleaseNotes>
<Copyright>Copyright (c) LakeSpeak.NET contributors</Copyright>
<PackageTags>databricks;genie;cli;analytics;conversational</PackageTags>
</PropertyGroup>
Expand Down
4 changes: 4 additions & 0 deletions LakeSpeak.slnx
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
<Solution>
<Folder Name="/examples/" />
<Folder Name="/examples/dotnet-quickstart/">
<Project Path="examples/dotnet-quickstart/LakeSpeak.Examples.Quickstart.csproj" />
</Folder>
<Folder Name="/src/">
<Project Path="src/LakeSpeak.Application/LakeSpeak.Application.csproj" />
<Project Path="src/LakeSpeak.Cli/LakeSpeak.Cli.csproj" />
Expand Down
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
# LakeSpeak.NET

[![CI](https://github.com/ivanvyd/lakespeak/actions/workflows/ci.yml/badge.svg)](https://github.com/ivanvyd/lakespeak/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/ivanvyd/lakespeak/badge)](https://scorecard.dev/viewer/?uri=github.com/ivanvyd/lakespeak)

Talk to governed Databricks data from your terminal and .NET applications.

LakeSpeak.NET is an independent open-source .NET client, terminal application, and automation
toolkit for [Databricks Genie](https://docs.databricks.com/aws/en/genie/) Agents.

**New here?** [Getting started](docs/getting-started.md) walks from nothing to a working answer,
including what a Genie Agent is and how to tell whether you have one.

> **Not a Databricks product.** LakeSpeak.NET is an independent open-source community project. It
> is not an official Databricks product, is not endorsed by Databricks, and is not supported under
> any Databricks service-level agreement. "Databricks", "Databricks Genie" and "Unity Catalog" are
Expand All @@ -18,6 +25,17 @@ toolkit for [Databricks Genie](https://docs.databricks.com/aws/en/genie/) Agents
actually been tested and what has not. Nothing here is stable until `v1.0`; the CLI surface and the
`LakeSpeak.Genie` public API may both change.

## What this is, in one paragraph

[Databricks Genie](https://docs.databricks.com/aws/en/genie/) answers questions about your data in
plain English: you ask, it writes SQL against tables someone has curated, runs it on a SQL
warehouse, and answers. A **Genie Agent** is one such configured surface. Genie normally lives in
the Databricks web UI — LakeSpeak puts it in your terminal, your scripts, and your .NET code, while
keeping the generated SQL visible so you can check the answer.

You need a Genie Agent to already exist in your workspace and be shared with you. LakeSpeak cannot
create one, and sees only what your own Databricks identity can see.

## Why this exists

The Genie Conversation API is capable, and the official `databricks genie` CLI exposes it. But using
Expand Down Expand Up @@ -101,6 +119,11 @@ if (response.Query is not null)
}
```

Abbreviated for orientation. The complete version — DI registration, `using` directives, reading
the rows, and typed error handling on `GenieFailureKind` — is in
[Getting started → Using it from .NET](docs/getting-started.md#using-it-from-net), and it is
compiled against the library rather than written by hand.

## Question Packs

A Question Pack turns a set of business questions into a reviewable, version-controlled report.
Expand Down Expand Up @@ -171,14 +194,18 @@ Paths that have not been exercised against a real workspace are labelled as such

## Documentation

- [Getting started](docs/getting-started.md) — zero to a working answer
- [Commands](docs/commands.md) — every command, flag and exit code
- [Configuration](docs/configuration.md) — the config file, aliases and defaults
- [`examples/`](examples/) — a runnable .NET console sample and a complete Question Pack
- [Authentication](docs/authentication.md) — profiles, environment tokens, and what is not supported
- [Question Packs](docs/question-packs.md) — the schema and its failure semantics
- [Troubleshooting](docs/troubleshooting.md)
- [Limitations](docs/limitations.md) — read this one
- [Decisions](docs/decisions/) — ADRs for the load-bearing choices
- [Genie API surface](docs/planning/genie-api-surface.md) — every wire claim, labelled verified or not
- [SOC 2 control mapping](docs/compliance/soc2-mapping.md)
- [Releasing](RELEASING.md) — how a version is cut, and how to rehearse one

## Contributing

Expand Down
104 changes: 104 additions & 0 deletions RELEASING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Releasing

How a LakeSpeak release is cut. Written down because a release process that lives in one person's
head is a release process that stops when they do.

## What a release produces

| Artifact | Where it goes |
|---|---|
| `LakeSpeak.Genie` | NuGet — the library |
| `LakeSpeak.Cli` | NuGet — the `lakespeak` dotnet tool |
| `lakespeak-<version>-win-x64.zip` | GitHub release — self-contained, no SDK needed |
| `lakespeak-<version>-linux-x64.zip` | GitHub release |
| `lakespeak-<version>-osx-arm64.zip` | GitHub release (built, **not tested** — see `docs/compatibility.md`) |
| `SHA256SUMS.txt` | GitHub release |
| `sbom.json` | GitHub release — CycloneDX, the vendor list an adopter's review will ask for |
| Build provenance attestation | Attached to each `.nupkg` |

## Versioning

Semantic versioning. **Before `v1.0`, a minor version may break the API** — that is what `0.x`
means, and the README says so rather than leaving people to discover it.

The version comes from one place per situation:

- **A tag** `v1.2.3` → version `1.2.3`.
- **A manual run** with the `version` input → that value.
- **A manual run with no input** → `0.0.0-dev.<run number>`, which is never published.

`VersionPrefix` in `Directory.Build.props` is the local-development default only. CI overrides it.

## Rehearse first

The release workflow is manually runnable, and by default it does **not** publish. Use that.

1. Actions → **Release** → *Run workflow*.
2. Leave **publish** unticked. Optionally set **version** to the version you intend to cut.
3. Run it.

That builds, runs the full non-live test suite, packs, publishes the three self-contained
binaries as workflow artifacts, generates the SBOM and attests provenance — everything a real
release does except pushing to NuGet and creating a GitHub release. If the rehearsal is red, the
release would have been red.

Download the artifact and install the tool locally before trusting it:

```bash
dotnet tool install --global --add-source ./artifacts LakeSpeak.Cli --version <version>
lakespeak --version
```

CI already does this on every PR (the `tool-smoke` job), but doing it by hand once before a real
release is cheap.

## Cut the release

### Prerequisites, once

- `NUGET_API_KEY` as a repository secret, scoped to `LakeSpeak.*`, not a global key.
- A `nuget` **environment** in repository settings. This is what turns publishing into a decision
someone makes rather than a side effect of pushing a tag. Add yourself as a required reviewer.

### Steps

1. Update `CHANGELOG.md`. Move `Unreleased` entries under a new `## <version> — <date>` heading.
2. Confirm `docs/compatibility.md` reflects what has actually been verified for this version.
An entry there with no evidence behind it is worse than a missing one.
3. Merge those to `main`.
4. Tag and push:

```bash
git tag -a v1.2.3 -m "v1.2.3"
git push origin v1.2.3
```

5. The workflow runs and stops at the `nuget` environment gate. Approve it.
6. Check the GitHub release: three binaries, checksums, SBOM, generated notes.

### Publishing without a tag

A manual run with **publish** ticked will push to NuGet. It exists for the case where a tag has
already been pushed and the publish job failed for an environmental reason — a NuGet outage, an
expired key — and you want to retry without inventing a new version.

It does not create a GitHub release, because a manual run has no tag to attach one to.

## If something goes wrong

**NuGet does not allow unpublishing.** A package can be deprecated or delisted, never removed.
That is why the rehearsal step exists and why the environment gate is not optional.

- **Wrong version published** → publish a corrected higher version, then delist the wrong one.
Do not attempt to reuse the version number; NuGet will reject it and `--skip-duplicate` will
silently succeed without publishing anything.
- **Tag pushed too early** → delete the tag (`git push --delete origin v1.2.3`) *before* approving
the environment gate. After approval, the only path forward is a new version.
- **Release job red after publish** → the packages are already on NuGet. Fix the GitHub release
by hand rather than re-running the whole workflow.

## What is deliberately not automated

There is no auto-release on merge, no release-please, no auto-generated version from commit
messages. For a project this size those add a machine that has to be understood before a release
can be made, which is the opposite of what this file is for.
2 changes: 1 addition & 1 deletion docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Ctrl+C cancels the question in flight and returns you to the prompt; it does not
| `/result` | Show the last query result |
| `/export <path>` | Write the last result to CSV |
| `/thumbs-up`, `/thumbs-down [comment]` | Send feedback to Databricks |
| `/exit` | Leave |
| `/exit` | Leave (`/quit` does the same) |

## `lakespeak pack`

Expand Down
Loading
Loading