feat: provision local ClickHouse for Intake

README.md

@@ -51,7 +51,9 @@ Verify:
 nemo services status
 ```
 
-To permanently reset the database state: `rm -rf ~/.local/share/nemo`.
+To permanently reset local state, follow the explicitly confirmed, guarded
+sequence in [SETUP.md](SETUP.md#question-3--wipe-local-platform-data). It removes
+the managed ClickHouse container before deleting any bind-mounted data.
 
 <details>
 <summary>Useful CLI commands once setup completes</summary>

SETUP.md

@@ -33,7 +33,7 @@ If port 8080 is in use **or** a `nemo services run` process exists, do not silen
   - The user may already have the workspace / secret / provider seeded — check with `nemo workspaces list`, `nemo secrets list --workspace …`, and `nemo inference providers list --workspace …` before re-creating anything (creates may 409).
 - **(c) Abort and let the user investigate.**
 
-> ⚠️ **macOS unlinked-inode gotcha:** running `rm -rf ~/.local/share/nemo` while a `nemo services run` process is still alive does **not** reset state. The files get unlinked from the directory tree but the running process keeps writing to its open inode, and a freshly-spawned platform will see the on-disk file (a new, empty inode) while the old process still owns the data. Always kill the platform process **first**, then wipe the DB.
+> ⚠️ **macOS unlinked-inode gotcha:** running `rm -rf ~/.local/share/nemo` while a `nemo services run` process or its managed ClickHouse container is still alive does **not** reset state safely. A process can keep writing to the unlinked inode. Always stop the platform and remove its managed ClickHouse container **before** wiping the data directory.
 
 ## Question 2 — Local data directory?
 
@@ -55,15 +55,32 @@ export NMP_DATA_DIR=/custom/path/to/state
 
 `nemo setup` persists the choice to `~/.config/nmp/config.yaml` under `local_services.data_dir` and re-uses it on subsequent runs. If you're running services manually (not via `nemo setup`), set `NMP_DATA_DIR` yourself each session.
 
-## Question 3 — Wipe the local DB?
+## Question 3 — Wipe local platform data?
 
-Ask whether the user wants to wipe the local entity-store database before platform startup. Warn clearly that this deletes local platform state, including secret metadata and the local encryption key. Providers/secrets must be re-seeded afterward. If the database and encryption key get out of sync, later runs can fail with decryption errors such as `cryptography.exceptions.InvalidTag`. **The wipe only works if no `nemo services run` process is currently holding the file open** (see the macOS gotcha under Question 1). If the user confirms, run this before `nemo services run`:
+Ask whether the user wants to wipe local platform data before startup. This is a destructive operation that requires explicit confirmation. Warn clearly that it deletes the entity-store database, encryption key, files, job history, secrets, and Intake ClickHouse traces stored under the selected platform data directory. An explicitly configured ClickHouse data directory outside it is preserved. Providers and secrets must be re-seeded afterward. If the database and encryption key get out of sync, later runs can fail with decryption errors such as `cryptography.exceptions.InvalidTag`. **Stop every `nemo services run` process before wiping** (see the macOS gotcha under Question 1), and remove the managed ClickHouse container before deleting its bind-mounted data. If the user confirms, run this before `nemo services run`:
 
 ```bash
-rm -rf ~/.local/share/nemo
+DATA_DIR="${NMP_DATA_DIR:-${XDG_DATA_HOME:-$HOME/.local/share}/nemo}"
+case "$DATA_DIR" in
+  ""|"/"|"$HOME"|"$HOME/"|"."|"./"|"$PWD"|"$PWD/")
+    echo "REFUSING_UNSAFE_DATA_DIR: '$DATA_DIR' — abort"; exit 1 ;;
+esac
+lsof -iTCP:8080 -sTCP:LISTEN >/dev/null 2>&1 && { echo "PLATFORM_STILL_RUNNING — abort before wipe"; exit 1; }
+CLICKHOUSE_DATA_DIR="${NMP_INTAKE_CLICKHOUSE_DATA_DIR:-$DATA_DIR/intake-clickhouse}"
+if [ -f "$CLICKHOUSE_DATA_DIR/.nmp-clickhouse-identity" ]; then
+  NMP_DATA_DIR="$DATA_DIR" uv run python -m nmp.intake.local_clickhouse --remove || {
+    echo "CLICKHOUSE_CONTAINER_CLEANUP_FAILED — start Docker and retry; data was not deleted"
+    exit 1
+  }
+fi
+rm -rf "$DATA_DIR"
 ```
 
-Replace the path with whatever was chosen in Q2 (`$NMP_DATA_DIR`, `$XDG_DATA_HOME/nemo`, or the default `~/.local/share/nemo`).
+Replace the path with whatever was chosen in Q2 (`$NMP_DATA_DIR`,
+`$XDG_DATA_HOME/nemo`, or the default `~/.local/share/nemo`). The cleanup command
+validates and removes only the managed container, restoring host ownership when
+its data lives under the platform data directory. If cleanup fails because Docker
+is unavailable, start Docker and retry—do not proceed to `rm -rf`.
 
 ---
 
@@ -135,6 +152,20 @@ LOG_LEVEL=DEBUG uv run nemo services run \
 
 `nemo-switchyard` is auto-discovered via its `nemo.inference_middleware` entry point once dependencies are installed.
 
+### Local ClickHouse for Intake
+
+When the `intake` service is selected and `NMP_INTAKE_CLICKHOUSE_URL` is unset, Intake automatically
+provisions a ClickHouse container owned by the resolved NeMo data directory, with a Docker-assigned
+loopback port. Platform processes using that data directory reuse the same container; shutdown does
+not remove it. Only the explicitly confirmed reset in Question 3 or teardown options 2/3 delete its
+default data under the NeMo data directory, after removing the managed container. A separately
+configured `NMP_INTAKE_CLICKHOUSE_DATA_DIR` is preserved.
+
+Docker must already be running. If startup logs report `Docker daemon is unavailable`, start Docker
+Desktop on macOS/Windows or the Docker service on Linux, then rerun `nemo setup` or restart
+`nemo services run`. To use an external ClickHouse and bypass local Docker provisioning, export
+`NMP_INTAKE_CLICKHOUSE_URL` before starting the platform.
+
 ### Demo agent
 
 `make bootstrap` installs the NeMo agents plugin and the calculator-agent example through the root workspace, so no separate `uv pip install` is needed. After services start, `nemo setup` (or `nemo setup --auto --deploy-agent`) will deploy a demo `calculator-agent` in the default workspace. Verify with:
@@ -148,7 +179,8 @@ nemo agents invoke --agent calculator-agent --input "What is 12 * 8?"
 
 - **Port**: `8080` (CLI default — do NOT pass a custom `--base-url`).
 - **`export NMP_BASE_URL=http://localhost:8080` — required when targeting a local platform.** If your `~/.config/nmp/config.yaml` already points at a remote cluster, the CLI uses that base URL and ignores the local platform entirely. Setting this env var overrides the config file for the current shell session.
-- **Reset state:** `rm -rf ~/.local/share/nemo` (only with platform stopped — see the gotcha above).
+- **Reset state:** follow Question 3 above; it requires explicit confirmation and removes the managed
+  ClickHouse container before deleting platform data.
 
 ---
 

docs/agents/observability.mdx

@@ -109,13 +109,10 @@ deployment with authentication disabled, omit the `Authorization` header and the
 
 <Accordion title="Running from source (local development)">
 
-If you are bringing the pieces up yourself from a repository checkout, start them in this order.
-
-1. **ClickHouse** (the telemetry datastore):
-
-   ```bash
-   services/intake/scripts/spans/run_clickhouse.sh
-   ```
+If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or
+the Docker daemon is running, then start the backend. Intake automatically provisions and reuses a
+local ClickHouse container for the resolved NeMo data directory unless `NMP_INTAKE_CLICKHOUSE_URL`
+explicitly points to an external instance.
 
 1. **Backend services**: `intake` plus its `auth` (access checks) and `entities` (entity store)
    dependencies. `--port` defaults to `8080`; omit `uv run` if you installed the `nemo`

docs/evaluator/experiments.mdx

@@ -140,13 +140,10 @@ time.
 
 <Accordion title="Running from source (local development)">
 
-If you are bringing the pieces up yourself from a repository checkout, start them in this order.
-
-1. **ClickHouse** (required for the rollups):
-
-   ```bash
-   services/intake/scripts/spans/run_clickhouse.sh
-   ```
+If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or
+the Docker daemon is running. Intake automatically provisions and reuses a local ClickHouse
+container for the resolved NeMo data directory unless `NMP_INTAKE_CLICKHOUSE_URL` explicitly points
+to an external instance.
 
 1. **Backend services**: `intake` plus its `auth` and `entities` dependencies. `--port` defaults to
    `8080`; omit `uv run` if you installed the `nemo` command-line interface (CLI):

docs/get-started/example-agent.mdx

@@ -24,18 +24,16 @@ make bootstrap
 
 ## 2. Start NeMo Platform
 
-Agent traces are stored in ClickHouse, which runs as its own container. Start it first, from the repository root:
-
-```bash
-services/intake/scripts/spans/run_clickhouse.sh
-```
-
-Then run setup. This starts the platform services in the background and walks you through the configuration required to run the platform:
+Agent traces are stored in ClickHouse. Make sure Docker Desktop or the Docker daemon is running, then
+run setup. Intake automatically provisions and reuses a local ClickHouse container. Setup starts the
+platform services in the background and walks you through the configuration required to run the platform:
 
 ```bash
 uv run nemo setup
 ```
 
+To use an external ClickHouse instead, set `NMP_INTAKE_CLICKHOUSE_URL` before running setup.
+
 You should now be able to navigate to `http://localhost:8080` and see the NeMo Platform web UI.
 
 ## 3. Prepare τ-Bench and run the airline agent

e2e/services_pool.py

@@ -553,8 +553,8 @@ def _load_effective_e2e_config_from_layers(
 
 
 def e2e_services_data_dir(log_dir: Path, config_hash: str) -> Path:
-    """Return the persistent data directory for one pooled services instance."""
-    return log_dir / f"data-{config_hash}"
+    """Return persistent state outside the directory collected as test logs."""
+    return log_dir.parent / f"{log_dir.name}-data" / f"data-{config_hash}"
 
 
 def with_e2e_instance_paths(

packages/nemo_platform/pyproject.toml

@@ -180,6 +180,7 @@ intake-service = [
   "pydantic-settings>=2.6.1, <3.0.0",
   "nmp-common",
   "clickhouse-connect>=0.7,<1.0",
+  "docker>=7.1.0",
   "opentelemetry-proto>=1.27.0",
 ]
 

packages/nemo_platform_ext/src/nemo_platform_ext/skills/inference/SKILL.md

@@ -522,10 +522,8 @@ let translate do the conversion.
 
 ## Troubleshooting
 
-**DB disk I/O error on startup** — orphaned WAL journal files. Delete all three:
-```bash
-rm -rf ~/.local/share/nemo
-```
+**DB disk I/O error on startup** — use the `nemo-teardown` skill's **stop +
+wipe data** flow, then rerun setup. Do not delete live platform data directly.
 
 **`nemo-switchyard` fails to load at startup** — `switchyard.lib` not importable.
 Run `uv sync` from the repo root with default groups enabled to install the

packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md

@@ -53,10 +53,10 @@ Require:
 For a remote deployment, set `NMP_BASE_URL` to its HTTPS origin and skip local startup. Use the
 deployment's authentication mechanism. Do not send authentication across an HTTP redirect: validate
 the final HTTPS origin and do not add `curl -L` to authenticated requests. For a local source
-checkout, follow `SETUP.md`, then start ClickHouse before the backend:
+checkout, follow `SETUP.md`, make sure Docker is running, then start the backend. Intake
+automatically provisions local ClickHouse unless `NMP_INTAKE_CLICKHOUSE_URL` is set:
 
 ```bash
-services/intake/scripts/spans/run_clickhouse.sh
 uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080
 ```