From 34f44baf6fbc90d32189fdab45edc67354f93561 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 28 May 2026 16:38:30 -0700 Subject: [PATCH 01/19] fix(docs): Update `nemo-platform` README.md to be more public-facing Signed-off-by: Matthew Grossman --- packages/nemo_platform/{README.md => BUNDLING.md} | 4 +++- packages/nemo_platform/pyproject.toml | 3 +++ 2 files changed, 6 insertions(+), 1 deletion(-) rename packages/nemo_platform/{README.md => BUNDLING.md} (97%) diff --git a/packages/nemo_platform/README.md b/packages/nemo_platform/BUNDLING.md similarity index 97% rename from packages/nemo_platform/README.md rename to packages/nemo_platform/BUNDLING.md index 2df285e4d2..cba7095fda 100644 --- a/packages/nemo_platform/README.md +++ b/packages/nemo_platform/BUNDLING.md @@ -1,4 +1,6 @@ -# nemo-platform +# nemo-platform wheel bundling + +> Internal developer docs for the `nemo-platform` wrapper distribution. For an end-user introduction to NeMo Platform, see the [repo root README](../../README.md) or the [PyPI page](https://pypi.org/project/nemo-platform/). Wrapper distribution for NeMo Platform. When users run `pip install nemo-platform[services]`, this is the wheel they get. diff --git a/packages/nemo_platform/pyproject.toml b/packages/nemo_platform/pyproject.toml index f3e040db25..55c522c026 100644 --- a/packages/nemo_platform/pyproject.toml +++ b/packages/nemo_platform/pyproject.toml @@ -6,6 +6,9 @@ # # To regenerate the auto-generated sections: # make vendor +# +# README.md is the public PyPI landing page (top-level NeMo Platform intro). +# Internal bundling/wrapper-distribution developer docs live in BUNDLING.md. # ============================================================================= [project] From b36af3ed1a36bcd246705ca70836c22081a1f10a Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 28 May 2026 16:39:05 -0700 Subject: [PATCH 02/19] add readme Signed-off-by: Matthew Grossman --- packages/nemo_platform/README.md | 100 +++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 packages/nemo_platform/README.md diff --git a/packages/nemo_platform/README.md b/packages/nemo_platform/README.md new file mode 100644 index 0000000000..f07bcac706 --- /dev/null +++ b/packages/nemo_platform/README.md @@ -0,0 +1,100 @@ +# NeMo Platform + +[![License](https://img.shields.io/badge/license-Apache_2.0-D22128?style=flat-square)](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/LICENSE) +[![Python](https://img.shields.io/badge/python-3.11--3.13-3776AB?style=flat-square&logo=python&logoColor=white)](https://www.python.org/) +[![Docs](https://img.shields.io/badge/docs-nvidia--nemo.github.io-76B900?style=flat-square&logo=readthedocs&logoColor=white)](https://nvidia-nemo.github.io/nemo-platform/) + +Make the agents you ship faster, more accurate, and safer. + +NeMo Platform brings NVIDIA NeMo libraries together under one CLI, Python SDK, and web UI. Hardening, evaluation, and tuning for the agents you put in production. + +## What's here today + +- **Secure agents.** Guardrails (content safety, jailbreak detection, PII redaction), Auditor (red-teaming via garak), Anonymizer (PII handling for training data). +- **Evaluate agents.** LLM-as-judge, deterministic, agentic, and RAG benchmarks. Harbor-backed eval suites for regression testing. +- **Tune agents.** Skill optimization, prompt and hyperparameter tuning, Switchyard model routing. +- **Build agents.** NVIDIA NeMo Agent Toolkit (NAT) for LangGraph-based agents. Shared infrastructure: Inference Gateway, Secrets, Files, Entity Store, Jobs. +- **Generate synthetic data.** Generate synthetic data for training or evaluation purposes using Data Designer. +- **NeMo Studio (alpha).** Browser UI for chat, monitoring, and reviewing optimization suggestions. Studio's agent-focused features are still a work in progress; the CLI is the primary surface today. + +## What's coming soon + +- Fine-tuning +- Safe Synthesizer (synthetic data with privacy guarantees) +- Broader agent framework support. Today NeMo Platform optimizes LangGraph agents wrapped in NAT. If your agent is in another framework, you need to write the NAT wrapper. + +## Install + +**Prerequisites:** Python 3.11–3.13 and an API key for an inference provider (NVIDIA Build, OpenAI, Anthropic, Google Gemini, or a local Ollama instance). + +The `nemo-platform` distribution is a convenience wrapper that bundles the SDK, shared runtime packages, default first-party plugins, and platform services into a single wheel. Install just the SDK and CLI, or install everything needed to run the platform locally: + +```bash +# SDK + CLI only +pip install nemo-platform + +# SDK + CLI + all platform services and default plugins +pip install "nemo-platform[services]" +``` + +Then bring up the platform: + +```bash +nemo setup +``` + +`nemo setup` starts local services, registers your LLM provider, discovers available models, installs agent skills, and (optionally) deploys a sample agent. + +Verify: + +```bash +nemo services status +``` + +The recommended developer setup is to clone the repo and use `make bootstrap` instead of installing from PyPI — see the [setup playbook](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/SETUP.md) for the full walkthrough, including local data dir, DB reset, and troubleshooting. + +## Quick tour + +```bash +nemo --help # All commands +nemo models list # Available models +nemo chat # Chat directly with a model +nemo services status # Platform health +nemo skills list # Skills installed on the platform +``` + +Every capability is also available via REST API. Model inference uses the model IDs returned from `nemo models list` and is available at: + +```text +http://localhost:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions +``` + +## Use NeMo Platform from your coding agent + +After installation, launch your coding agent (Claude Code, Codex, Cursor, OpenCode, etc) from inside a NeMo Platform workspace. NeMo ships agent skills that let you drive the platform conversationally: + +- "Scaffold an agent from this spec and deploy it." +- "Run an evaluation against my agent." +- "Add content-safety guardrails to my agent." +- "Help me optimize my agent." +- "Show me what's running on the platform." +- "Shut down NeMo cleanly." + +Install or refresh skills with: + +```bash +nemo skills install --agent claude +``` + +## Links + +- **Source:** https://github.com/NVIDIA-NeMo/nemo-platform +- **Documentation:** https://nvidia-nemo.github.io/nemo-platform/ +- **Setup playbook:** https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/SETUP.md +- **CLI reference:** https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/docs/cli/index.md +- **API reference:** https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/docs/api/index.md +- **Issue tracker:** https://github.com/NVIDIA-NeMo/nemo-platform/issues + +## License + +NeMo Platform is licensed under the [Apache License 2.0](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/LICENSE). Third-party open-source dependencies have their own licenses; review them before use. From dbc2afbe914479e3bf52393e1f30b839891d341c Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Fri, 29 May 2026 11:38:59 -0700 Subject: [PATCH 03/19] fix up nemo-platform-plugin Signed-off-by: Matthew Grossman --- .../src/nemo_platform_plugin/README.md | 114 ++++++++++++++---- 1 file changed, 93 insertions(+), 21 deletions(-) diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md index 6f0b5365c9..db642911dd 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md @@ -1,8 +1,14 @@ # nemo-platform-plugin -`nemo-platform-plugin` is the only package NeMo Platform plugin authors install. It re-exports all base classes, schemas, and utilities needed to build fully-featured NeMo Platform plugins. +[![License](https://img.shields.io/badge/license-Apache_2.0-D22128?style=flat-square)](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/LICENSE) +[![Python](https://img.shields.io/badge/python-3.11--3.13-3776AB?style=flat-square&logo=python&logoColor=white)](https://www.python.org/) +[![Docs](https://img.shields.io/badge/docs-nvidia--nemo.github.io-76B900?style=flat-square&logo=readthedocs&logoColor=white)](https://nvidia-nemo.github.io/nemo-platform/) -## Surfaces +Build NeMo Platform plugins in Python. + +`nemo-platform-plugin` is the only package plugin authors install. It re-exports every base class, schema, and helper you need to contribute HTTP services, CLI commands, schedulable jobs, background controllers, typed configuration, and entity types to [NeMo Platform](https://pypi.org/project/nemo-platform/). Declare your surfaces in `pyproject.toml`, install the package, and the platform discovers and mounts them at startup — no registration code required. + +## What you can build | Surface | Base class | Entry-point group | Purpose | |---|---|---|---| @@ -10,36 +16,102 @@ | CLI | `NemoCLI` | `nemo.cli` | Contributes `nemo ` subcommands | | Job | `NemoJob` | `nemo.jobs` | Contributes schedulable, container-executable jobs. Auto-generates `run` / `submit` / `explain` CLI verbs. | | Controller | `NemoController` | `nemo.controllers` | Contributes background reconcile-loop controllers | -| Configuration | `NemoConfig` | (none)¹ | Typed plugin configuration with env var / YAML loading | -| Entity | `NemoEntity` | (none)¹ | Entity definitions stored in the NeMo Platform entity store | +| Configuration | `NemoConfig` | — | Typed plugin configuration with env var / YAML loading | +| Entity | `NemoEntity` | — | Entity definitions stored in the NeMo Platform entity store | + +## Install -¹ Not discovered via entry-points — used directly by plugin code. +```bash +pip install nemo-platform-plugin +``` -## Installation +To run a local NeMo Platform that loads and serves your plugin while you develop, install [`nemo-platform`](https://pypi.org/project/nemo-platform/) too — it ships the platform services, the `nemo` CLI, and the runtime that wires entity-client injection into your plugin's FastAPI app: ```bash -uv add nemo-platform-plugin nemo-platform +pip install "nemo-platform[all]" +``` + +## A minimal plugin + +A complete NeMo Platform plugin that contributes one HTTP route. Two files plus a `pyproject.toml`: + +```toml +# pyproject.toml +[project] +name = "nemo-my-plugin" +version = "0.1.0" +requires-python = ">=3.11" +dependencies = ["nemo-platform-plugin"] + +[project.entry-points."nemo.services"] +"my-plugin" = "nemo_my_plugin.service:MyService" + +# [build-system] is required by PEP 517 — without it, pip falls back to +# legacy setuptools. Any modern backend works; hatchling is what NeMo +# Platform itself uses and is the easiest fit for the src/ layout below. +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +# Required because the plugin uses a src/ layout with implicit namespace +# packages (no __init__.py). Hatchling needs to be told where the source is. +[tool.hatch.build.targets.wheel] +packages = ["src/nemo_my_plugin"] ``` -`nemo-platform` is required for entity client injection at runtime. +```python +# src/nemo_my_plugin/service.py +from typing import ClassVar -## How plugins are discovered +from fastapi import APIRouter +from nemo_platform_plugin.service import NemoService, RouterSpec -Declare surfaces in `pyproject.toml` entry-point groups. Install the package and the platform picks them up at startup — no registration code needed. See [QUICKSTART.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md) for the full template. -## The `name` rule +class MyService(NemoService): + name: ClassVar[str] = "my-plugin" + dependencies: ClassVar[list[str]] = [] -Every surface class requires a non-empty `name: ClassVar[str]`. Checked at class-definition time — missing raises `TypeError`. Must exactly match the entry-point key; mismatch logs a warning at startup. + def get_routers(self) -> list[RouterSpec]: + router = APIRouter() -## Next steps + @router.get("/hello") + async def hello() -> dict: + return {"message": "Hello from my plugin!"} -- [QUICKSTART.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md) — build your first plugin end-to-end -- [SERVICE.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/SERVICE.md) — HTTP routes, CRUD patterns, testing -- [JOB.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/JOB.md) — jobs, CLI commands, auto-generated job commands -- [CONTROLLER.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/CONTROLLER.md) — reconcile loops, state machines, service principal -- [CONFIG.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/CONFIG.md) — typed configuration, env vars, YAML, test overrides -- [ENTITY.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ENTITY.md) — entity store, CRUD client, pagination, optimistic locking -- [INFERENCE_MIDDLEWARE.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/INFERENCE_MIDDLEWARE.md) — inference request/response middleware, typed bodies, and response annotations -- [ARCHITECTURE.md](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md) — discovery, startup, all surfaces, SDK surface + return [RouterSpec(router, tag="My Plugin")] +``` + +Install and run the platform with your plugin discovered: + +```bash +pip install -e . +nemo services run +# GET http://localhost:8080/apis/my-plugin/hello → {"message": "Hello from my plugin!"} +``` + +That's a complete plugin. Adding a CLI command, a job, a controller, typed configuration, or entities follows the same shape — declare a class, register it under the matching entry-point group, install. See [QUICKSTART](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md) for the full template covering every surface. + +## Guides + +- [QUICKSTART](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md) — build your first plugin end-to-end (service + CLI + job + controller + config) +- [SERVICE](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/SERVICE.md) — HTTP routes, CRUD patterns, testing +- [JOB](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/JOB.md) — jobs, CLI commands, auto-generated job verbs +- [CONTROLLER](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/CONTROLLER.md) — reconcile loops, state machines, service principal +- [CONFIG](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/CONFIG.md) — typed configuration, env vars, YAML, test overrides +- [ENTITY](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ENTITY.md) — entity store, CRUD client, pagination, optimistic locking +- [INFERENCE_MIDDLEWARE](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/INFERENCE_MIDDLEWARE.md) — inference request/response middleware, typed bodies, response annotations +- [ARCHITECTURE](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md) — discovery, startup, all surfaces, SDK surface > **Tip:** These guides also ship inside the installed package at `nemo_platform_plugin/docs/`. + +## Links + +- **This package source:** https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/packages/nemo_platform_plugin +- **NeMo Platform on PyPI:** https://pypi.org/project/nemo-platform/ +- **NeMo Platform on GitHub:** https://github.com/NVIDIA-NeMo/nemo-platform +- **NeMo Platform docs:** https://nvidia-nemo.github.io/nemo-platform/ +- **Issue tracker:** https://github.com/NVIDIA-NeMo/nemo-platform/issues + +## License + +`nemo-platform-plugin` is licensed under the [Apache License 2.0](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/LICENSE). Third-party open-source dependencies have their own licenses; review them before use. From 72d7c8f605ce87ec52c8342b6072869f121c3e20 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Fri, 29 May 2026 11:49:53 -0700 Subject: [PATCH 04/19] apply code review feedback Signed-off-by: Matthew Grossman --- packages/nemo_platform/README.md | 35 +++++++++++++++----------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/packages/nemo_platform/README.md b/packages/nemo_platform/README.md index d309927131..b63798e057 100644 --- a/packages/nemo_platform/README.md +++ b/packages/nemo_platform/README.md @@ -53,12 +53,26 @@ nemo services status The recommended developer setup is to clone the repo and use `make bootstrap` instead of installing from PyPI — see the [setup playbook](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/SETUP.md) for the full walkthrough, including local data dir, DB reset, and troubleshooting. -## Quick tour +## Where to go next + +After `nemo setup` completes, the CLI prompts you to pick one of two paths. The same two paths are the recommended starting points if you came in via PyPI: + +- **Build and optimize agents.** Open a coding agent session inside your agent's project directory and ask: _"Build and optimize an agent using NeMo Platform."_ The shipped skills will scaffold the agent, deploy it, run evaluations, suggest optimizations, and add guardrails on request. +- **Explore the platform.** From any coding agent session, ask: _"What can I do with NeMo Platform?"_ to get a guided tour of the capabilities surfaced through skills. + +NeMo skills work with Claude Code, Codex, Cursor, OpenCode, and other coding agents. Install or refresh them with: + +```bash +nemo skills install --agent claude +``` + +## Operating the platform + +A few useful CLI commands once setup completes: ```bash nemo --help # All commands nemo models list # Available models -nemo chat # Chat directly with a model nemo services status # Platform health nemo skills list # Skills installed on the platform ``` @@ -69,23 +83,6 @@ Every capability is also available via REST API. Model inference uses the model http://localhost:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions ``` -## Use NeMo Platform from your coding agent - -After installation, launch your coding agent (Claude Code, Codex, Cursor, OpenCode, etc) from inside a NeMo Platform workspace. NeMo ships agent skills that let you drive the platform conversationally: - -- "Scaffold an agent from this spec and deploy it." -- "Run an evaluation against my agent." -- "Add content-safety guardrails to my agent." -- "Help me optimize my agent." -- "Show me what's running on the platform." -- "Shut down NeMo cleanly." - -Install or refresh skills with: - -```bash -nemo skills install --agent claude -``` - ## Links - **Source:** https://github.com/NVIDIA-NeMo/nemo-platform From b3ed16a243b5d9b0f48ebfe6e9a00d9c195ac655 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Fri, 29 May 2026 13:01:48 -0700 Subject: [PATCH 05/19] remove coming soon Signed-off-by: Matthew Grossman --- packages/nemo_platform/README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/packages/nemo_platform/README.md b/packages/nemo_platform/README.md index b63798e057..a321ceeeb8 100644 --- a/packages/nemo_platform/README.md +++ b/packages/nemo_platform/README.md @@ -17,12 +17,6 @@ NeMo Platform brings NVIDIA NeMo libraries together under one CLI, Python SDK, a - **Generate synthetic data.** Generate synthetic data for training or evaluation purposes using Data Designer. - **NeMo Studio (alpha).** Browser UI for chat, monitoring, and reviewing optimization suggestions. Studio's agent-focused features are still a work in progress; the CLI is the primary surface today. -## What's coming soon - -- Fine-tuning -- Safe Synthesizer (synthetic data with privacy guarantees) -- Broader agent framework support. Today NeMo Platform optimizes LangGraph agents wrapped in NAT. If your agent is in another framework, you need to write the NAT wrapper. - ## Install **Prerequisites:** Python 3.11–3.13 and an API key for an inference provider (NVIDIA Build, OpenAI, Anthropic, Google Gemini, or a local Ollama instance). From c9600efe03fa3ee1a67131100c104d1f48f3339c Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 08:37:37 -0700 Subject: [PATCH 06/19] code review Signed-off-by: Matthew Grossman --- packages/nemo_platform/README.md | 2 +- .../nemo_platform_plugin/src/nemo_platform_plugin/README.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/packages/nemo_platform/README.md b/packages/nemo_platform/README.md index a321ceeeb8..46b93c0a75 100644 --- a/packages/nemo_platform/README.md +++ b/packages/nemo_platform/README.md @@ -54,7 +54,7 @@ After `nemo setup` completes, the CLI prompts you to pick one of two paths. The - **Build and optimize agents.** Open a coding agent session inside your agent's project directory and ask: _"Build and optimize an agent using NeMo Platform."_ The shipped skills will scaffold the agent, deploy it, run evaluations, suggest optimizations, and add guardrails on request. - **Explore the platform.** From any coding agent session, ask: _"What can I do with NeMo Platform?"_ to get a guided tour of the capabilities surfaced through skills. -NeMo skills work with Claude Code, Codex, Cursor, OpenCode, and other coding agents. Install or refresh them with: +NeMo skills work with Claude Code, Codex, Cursor, OpenCode, and other coding agents. Install or refresh them with `nemo skills install --agent `, for example: ```bash nemo skills install --agent claude diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md index db642911dd..120193167e 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/README.md @@ -33,7 +33,7 @@ pip install "nemo-platform[all]" ## A minimal plugin -A complete NeMo Platform plugin that contributes one HTTP route. Two files plus a `pyproject.toml`: +A complete NeMo Platform plugin that contributes one HTTP route — a `pyproject.toml` and a service module: ```toml # pyproject.toml @@ -81,7 +81,7 @@ class MyService(NemoService): return [RouterSpec(router, tag="My Plugin")] ``` -Install and run the platform with your plugin discovered: +During development, `nemo-platform-plugin` is the only dependency your plugin needs. To actually run a local platform that loads your plugin, install `nemo-platform[all]` (see [Install](#install) above), then: ```bash pip install -e . From 7326046951899d9d0221c8461f72d0792ebc65a0 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:00:59 -0700 Subject: [PATCH 07/19] feat(ci): Add e2e tests Signed-off-by: Matthew Grossman --- Makefile | 7 ++- TESTING.md | 54 +++++++++++---------- conftest.py | 3 ++ e2e/conftest.py | 121 ++++++++++++++++++++++++++++++++++++++++++++++ e2e/test_smoke.py | 37 ++++++++++++++ 5 files changed, 193 insertions(+), 29 deletions(-) create mode 100644 e2e/conftest.py create mode 100644 e2e/test_smoke.py diff --git a/Makefile b/Makefile index e832e67369..de1e470f33 100644 --- a/Makefile +++ b/Makefile @@ -275,11 +275,10 @@ test-all-script: ## Run all unit tests using the helper script (with summary) @echo "Running all unit tests with summary..." uv run --frozen python tools/run_all_tests.py -# NOTE: disabled this in favour of the other test-e2e target (see below) .PHONY: test-e2e -# test-e2e: ## Run Python end-to-end tests (customer workflows and blueprints) -# @echo "Running Python end-to-end tests..." -# uv run --frozen pytest -v -m e2e +test-e2e: ## Run e2e tests against nemo services (starts/stops services automatically) + @echo "Running e2e tests..." + uv run --frozen pytest e2e -v --run-e2e $(PYTEST_EXTRA) .PHONY: test-regression test-regression: ## Run Python regression tests (functional microservice baseline tests) diff --git a/TESTING.md b/TESTING.md index 306a5c59c9..8cfdde69fa 100644 --- a/TESTING.md +++ b/TESTING.md @@ -113,25 +113,38 @@ def test_create_and_fetch_entity(client, db_session): ### 3. End-to-End (E2E) Tests -**Objective**: Ensure that customers can orchestrate services together for common workflows and blueprints on real deployed infrastructure. +**Objective**: Ensure that services work together correctly when running as a real platform process. **Characteristics**: -- Test complete customer workflows on actual deployments -- Uses testcontainers with Docker or Kubernetes backends -- Multiple services working together with real infrastructure -- Slowest tests (minutes to hours) -- Requires deployed infrastructure (no mocking of Jobs or Inference) +- Start the platform via `nemo services start` (real process, real ports) +- Hit services with an external HTTP client (the NeMoPlatform SDK) +- Test startup machinery, port binding, config resolution, and cross-service workflows +- Slower than integration tests (tens of seconds for startup) but faster than Docker/K8s e2e + +**How to run**: + +```bash +# Start services, run tests, stop services (all automatic) +make test-e2e + +# Or manually +uv run --frozen pytest e2e -v --run-e2e + +# If you already have services running +NMP_BASE_URL=http://localhost:8080 uv run --frozen pytest e2e -v --run-e2e +``` + +**Prerequisites**: `make bootstrap` must have been run. The harness runs `nemo services start` +on a free port with a unique `--instance` name, so it won't conflict with your dev instance. **When to Write E2E Tests**: Write E2E tests when you need to: - **Verify cross-service workflows**: Test operations that span multiple services (e.g., create workspace → upload file → run job → get results) -- **Test real infrastructure**: Validate jobs, inference, or storage backends that cannot be mocked -- **Validate customer scenarios**: Test complete workflows as users would experience them +- **Validate the real startup path**: Ensure config resolution, service discovery, and health checks work - **Test authentication/authorization**: Verify role-based access control across multiple services -- **Test async workflows**: Validate long-running operations, job lifecycles, and event propagation -- **Ensure service integration**: Verify that services work together correctly in production-like setups +- **Ensure service integration**: Verify that services work together correctly end-to-end **What NOT to E2E Test**: - Single service APIs (use integration tests) @@ -139,9 +152,7 @@ Write E2E tests when you need to: - Every permutation of inputs (E2E should focus on critical paths) - Implementation details (test user-visible behavior) -For detailed E2E test documentation, configuration options, and best practices, see: **[e2e/README.md](e2e/README.md)** - -**Location**: `e2e/` (root-level for deployed infrastructure tests) +**Location**: `e2e/` (root-level) ### 4. Infrastructure Tests @@ -395,19 +406,12 @@ uv run python tools/run_all_tests.py make test-integration uv run pytest -v -m integration -# End-to-end tests (Docker backend - recommended for local dev) -make test-e2e-docker -uv run pytest e2e --docker -v - -# End-to-end tests (Kubernetes backend: local minikube or custom cluster) -make test-e2e-minikube -uv run pytest e2e --kubernetes --cluster-url=https://my-cluster.example.com -v - -# E2E with custom registry and tag -uv run pytest e2e --docker --registry=my-registry --tag=v1.0.0 -v +# End-to-end tests (starts nemo services automatically) +make test-e2e +uv run --frozen pytest e2e -v --run-e2e -# E2E with custom config -uv run pytest e2e --docker --config=e2e/quickstart/custom.yaml -v +# E2E against an already-running instance +NMP_BASE_URL=http://localhost:8080 uv run --frozen pytest e2e -v --run-e2e # Regression tests make test-regression diff --git a/conftest.py b/conftest.py index 5d2c6809d8..87351128fc 100644 --- a/conftest.py +++ b/conftest.py @@ -285,6 +285,9 @@ def pytest_runtest_setup(item): if "slow" in [marker.name for marker in item.iter_markers()]: if not item.config.getoption("--run-slow"): skip_test("Skipping slow test (use --run-slow to run)") + if "e2e" in [marker.name for marker in item.iter_markers()]: + if not item.config.getoption("--run-e2e"): + skip_test("Skipping e2e test (use --run-e2e to run)") from xdist.scheduler.loadscope import LoadScopeScheduling # noqa: E402 diff --git a/e2e/conftest.py b/e2e/conftest.py new file mode 100644 index 0000000000..aa89a743f9 --- /dev/null +++ b/e2e/conftest.py @@ -0,0 +1,121 @@ +"""E2E test fixtures that run against a real ``nemo services`` process. + +Usage:: + + # Start services, run e2e tests, stop services + make test-e2e + + # Or manually + uv run --frozen pytest e2e -v --run-e2e + + # If you already have services running + NMP_BASE_URL=http://localhost:9090 uv run --frozen pytest e2e -v --run-e2e + +When ``NMP_BASE_URL`` is set the harness skips service startup/shutdown and +connects to the given URL. Otherwise it starts ``nemo services start`` on a +free port, waits for ``/health/ready``, and tears the instance down after the +session. +""" + +import logging +import os +import socket +import subprocess +import uuid +from collections.abc import Iterator + +import pytest +from nemo_platform import NeMoPlatform + +logger = logging.getLogger(__name__) + + +def _find_free_port() -> int: + """Bind to port 0 and let the OS assign a free port.""" + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +@pytest.fixture(scope="session") +def _services() -> Iterator[str]: + """Start ``nemo services`` and yield the base URL. + + Skipped when ``NMP_BASE_URL`` is already set (external services). + """ + external_url = os.environ.get("NMP_BASE_URL") + if external_url: + yield external_url + return + + port = _find_free_port() + instance = f"e2e-test-{port}" + url = f"http://127.0.0.1:{port}" + + logger.info("Starting nemo services on port %d (instance=%s)", port, instance) + + result = subprocess.run( + [ + "nemo", + "services", + "start", + "--service-group", + "all", + "--port", + str(port), + "--instance", + instance, + ], + capture_output=True, + text=True, + timeout=120, + ) + + if result.returncode != 0: + pytest.fail( + f"nemo services start failed (exit {result.returncode}):\nstdout: {result.stdout}\nstderr: {result.stderr}" + ) + + logger.info("Platform services started: %s", result.stdout.strip()) + + # Set NMP_BASE_URL so the SDK picks it up automatically + os.environ["NMP_BASE_URL"] = url + + yield url + + logger.info("Stopping nemo services (instance=%s)", instance) + stop_result = subprocess.run( + [ + "nemo", + "services", + "stop", + "--instance", + instance, + "--port", + str(port), + ], + capture_output=True, + text=True, + timeout=60, + ) + if stop_result.returncode != 0: + logger.warning( + "nemo services stop failed (exit %d): %s", + stop_result.returncode, + stop_result.stderr, + ) + + +@pytest.fixture(scope="session") +def sdk(_services: str) -> NeMoPlatform: + """Provide an SDK client connected to the running platform.""" + return NeMoPlatform(base_url=_services, max_retries=2) + + +@pytest.fixture(scope="function") +def workspace(sdk: NeMoPlatform) -> Iterator[str]: + """Create a unique workspace for each test, deleted on teardown.""" + name = f"e2e-{uuid.uuid4().hex[:8]}" + sdk.workspaces.create(name=name) + yield name + sdk.workspaces.delete(name) diff --git a/e2e/test_smoke.py b/e2e/test_smoke.py new file mode 100644 index 0000000000..46d3db4645 --- /dev/null +++ b/e2e/test_smoke.py @@ -0,0 +1,37 @@ +"""Smoke tests that verify the platform is reachable and core APIs respond. + +These are intentionally minimal — they validate the e2e harness works and +that services are up. Add more substantive tests in separate files. +""" + +from nemo_platform import NeMoPlatform + + +def test_health_ready(sdk: NeMoPlatform): + """GET /health/ready returns 200 when all services are up.""" + resp = sdk._client.get("/health/ready") + assert resp.status_code == 200 + + +def test_health_live(sdk: NeMoPlatform): + """GET /health/live returns 200 (liveness probe).""" + resp = sdk._client.get("/health/live") + assert resp.status_code == 200 + + +def test_create_and_delete_workspace(workspace: str, sdk: NeMoPlatform): + """Workspace CRUD round-trips through the platform. + + Uses the ``workspace`` fixture which creates a unique workspace + and deletes it on teardown. + """ + page = sdk.workspaces.list() + names = [w.name for w in page.data] + assert workspace in names + + +def test_list_workspaces(sdk: NeMoPlatform, workspace: str): + """Listing workspaces returns at least the test workspace.""" + page = sdk.workspaces.list() + names = [w.name for w in page.data] + assert workspace in names From 8aa78e5ed2606bf8e24f4223fced622571af93f9 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:15:09 -0700 Subject: [PATCH 08/19] fix conftest Signed-off-by: Matthew Grossman --- e2e/conftest.py | 107 ++++++++++++++++++++++++++---------------------- 1 file changed, 57 insertions(+), 50 deletions(-) diff --git a/e2e/conftest.py b/e2e/conftest.py index aa89a743f9..39995f4611 100644 --- a/e2e/conftest.py +++ b/e2e/conftest.py @@ -12,23 +12,30 @@ NMP_BASE_URL=http://localhost:9090 uv run --frozen pytest e2e -v --run-e2e When ``NMP_BASE_URL`` is set the harness skips service startup/shutdown and -connects to the given URL. Otherwise it starts ``nemo services start`` on a -free port, waits for ``/health/ready``, and tears the instance down after the -session. +connects to the given URL. Otherwise it spawns ``nemo services run`` as a +child process on a free port, polls ``/health/ready`` until ready, and +terminates the process after the session. """ import logging import os import socket import subprocess +import sys +import time import uuid from collections.abc import Iterator +from pathlib import Path +import httpx import pytest from nemo_platform import NeMoPlatform logger = logging.getLogger(__name__) +_HEALTH_TIMEOUT = 60 +_HEALTH_POLL_INTERVAL = 1.0 + def _find_free_port() -> int: """Bind to port 0 and let the OS assign a free port.""" @@ -37,11 +44,26 @@ def _find_free_port() -> int: return s.getsockname()[1] +def _wait_for_healthy(url: str, timeout: float = _HEALTH_TIMEOUT) -> bool: + """Poll /health/ready until it returns 200 or timeout expires.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + resp = httpx.get(f"{url}/health/ready", timeout=2.0) + if resp.status_code == 200: + return True + except httpx.RequestError: + pass + time.sleep(_HEALTH_POLL_INTERVAL) + return False + + @pytest.fixture(scope="session") def _services() -> Iterator[str]: - """Start ``nemo services`` and yield the base URL. + """Spawn ``nemo services run`` and yield the base URL. Skipped when ``NMP_BASE_URL`` is already set (external services). + The process is our direct child — on teardown we just terminate it. """ external_url = os.environ.get("NMP_BASE_URL") if external_url: @@ -49,61 +71,46 @@ def _services() -> Iterator[str]: return port = _find_free_port() - instance = f"e2e-test-{port}" url = f"http://127.0.0.1:{port}" - logger.info("Starting nemo services on port %d (instance=%s)", port, instance) - - result = subprocess.run( - [ - "nemo", - "services", - "start", - "--service-group", - "all", - "--port", - str(port), - "--instance", - instance, - ], - capture_output=True, + nemo_bin = str(Path(sys.executable).parent / "nemo") + args = [ + nemo_bin, + "services", + "run", + "--service-group", + "all", + "--port", + str(port), + ] + + logger.info("Starting nemo services on port %d", port) + + proc = subprocess.Popen( + args, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True, - timeout=120, ) - if result.returncode != 0: - pytest.fail( - f"nemo services start failed (exit {result.returncode}):\nstdout: {result.stdout}\nstderr: {result.stderr}" - ) + if not _wait_for_healthy(url): + # Grab whatever output we have for diagnostics + proc.terminate() + stdout, _ = proc.communicate(timeout=10) + pytest.fail(f"nemo services run did not become healthy within {_HEALTH_TIMEOUT}s.\nstdout:\n{stdout}") - logger.info("Platform services started: %s", result.stdout.strip()) - - # Set NMP_BASE_URL so the SDK picks it up automatically - os.environ["NMP_BASE_URL"] = url + logger.info("Platform services ready on port %d (pid %d)", port, proc.pid) yield url - logger.info("Stopping nemo services (instance=%s)", instance) - stop_result = subprocess.run( - [ - "nemo", - "services", - "stop", - "--instance", - instance, - "--port", - str(port), - ], - capture_output=True, - text=True, - timeout=60, - ) - if stop_result.returncode != 0: - logger.warning( - "nemo services stop failed (exit %d): %s", - stop_result.returncode, - stop_result.stderr, - ) + logger.info("Terminating nemo services (pid %d)", proc.pid) + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + logger.warning("Process did not exit after SIGTERM, sending SIGKILL") + proc.kill() + proc.wait(timeout=5) @pytest.fixture(scope="session") From b002fd1b31f1ae8eba9e1b6fa7bdfad2a023d460 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:26:55 -0700 Subject: [PATCH 09/19] fix wheel-build Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 132 +++++++++++++++++++++++++------------- Makefile | 2 +- e2e/conftest.py | 5 +- 3 files changed, 94 insertions(+), 45 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index a8913d3218..030fe2101e 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -202,14 +202,11 @@ jobs: coverage.xml coverage.json - # One job, one (package x python) matrix — kept as a single job rather - # than split per-package because the build path is identical and the - # matrix definition is the natural place to change what's covered. - # nemo-platform's wheel force-includes auth's policy.wasm at build time; - # nemo-platform-plugin doesn't, but plugin rows still wait on policy-wasm - # because needs: is per-job. ~5s of harmless wait per plugin row. - wheel-test: - name: ${{ matrix.package }} wheel build + test (py${{ matrix.python-version }}) + # Build wheels for all packages × python versions. Downstream jobs + # (wheel-smoke-test, python-e2e-test) download these artifacts instead + # of rebuilding. + wheel-build: + name: ${{ matrix.package }} wheel build (py${{ matrix.python-version }}) needs: [policy-wasm] runs-on: ubuntu-latest timeout-minutes: 25 @@ -236,29 +233,55 @@ jobs: package: ${{ matrix.package }} out-dir: ${{ github.workspace }}/dist cadence: nightly - # Sentinel epoch — the wheel version is meaningless for a CI test - # build, but stamp_sdk_version.py requires \d{14}. nightly-timestamp: "19700101000000" python-version: ${{ matrix.python-version }} + - name: Upload wheel + uses: actions/upload-artifact@v6 + with: + name: ${{ matrix.package }}-wheel-py${{ matrix.python-version }} + path: ${{ steps.build.outputs.wheel-path }} + retention-days: 7 + if-no-files-found: error + + # Smoke-test each wheel: install from the artifact and run basic CLI / + # import checks. This replaces the old combined wheel-test job. + wheel-smoke-test: + name: ${{ matrix.package }} wheel smoke test (py${{ matrix.python-version }}) + needs: [wheel-build] + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + strategy: + fail-fast: false + matrix: + package: [nemo-platform, nemo-platform-plugin] + python-version: ["3.11", "3.12", "3.13"] + steps: + - name: Checkout code + uses: actions/checkout@v6 + - name: Install uv + uses: astral-sh/setup-uv@v7 + with: + python-version: ${{ matrix.python-version }} + enable-cache: true + - name: Download wheel + uses: actions/download-artifact@v8 + with: + name: ${{ matrix.package }}-wheel-py${{ matrix.python-version }} + path: ${{ runner.temp }}/wheelcheck - # The two install+test steps below intentionally share a small preamble - # (mkdir + cp). Hoisting it into a separate step would cost more in - # workflow indirection than the 3 duplicated lines save. - name: Install + test nemo-platform CLI if: matrix.package == 'nemo-platform' shell: bash env: - WHEEL: ${{ steps.build.outputs.wheel-path }} PYTHON_VERSION: ${{ matrix.python-version }} NMP_DATA_DIR: ${{ runner.temp }}/nemo-data + _TYPER_FORCE_DISABLE_TERMINAL: "1" run: | set -euo pipefail - # Copy the wheel out of $GITHUB_WORKSPACE so the test cd's away - # from the source tree and Python can't reach back into it. - mkdir -p "${RUNNER_TEMP}/wheelcheck" - cp "${WHEEL}" "${RUNNER_TEMP}/wheelcheck/" - uv tool install --force --python "${PYTHON_VERSION}" \ - "${RUNNER_TEMP}/wheelcheck/$(basename "${WHEEL}")[services]" + WHEEL="$(ls ${RUNNER_TEMP}/wheelcheck/*.whl)" + uv tool install --force --python "${PYTHON_VERSION}" "${WHEEL}[services]" cd "${RUNNER_TEMP}/wheelcheck" unset PYTHONPATH VIRTUAL_ENV bash "${GITHUB_WORKSPACE}/script/test-nemo-cli.sh" @@ -267,22 +290,14 @@ jobs: if: matrix.package == 'nemo-platform-plugin' shell: bash env: - WHEEL: ${{ steps.build.outputs.wheel-path }} PYTHON_VERSION: ${{ matrix.python-version }} run: | set -euo pipefail - mkdir -p "${RUNNER_TEMP}/wheelcheck" - cp "${WHEEL}" "${RUNNER_TEMP}/wheelcheck/" + WHEEL="$(ls ${RUNNER_TEMP}/wheelcheck/*.whl)" cd "${RUNNER_TEMP}/wheelcheck" uv venv .venv --python "${PYTHON_VERSION}" - uv pip install --python .venv/bin/python "$(basename "${WHEEL}")" + uv pip install --python .venv/bin/python "${WHEEL}" unset PYTHONPATH VIRTUAL_ENV - # Import the package + a couple of representative submodules - # (cli, commands). These are the surfaces a plugin author would - # touch first; if any of them fail to import, the wheel is - # broken in a way that surfaces immediately on day one. If - # the plugin's public API surface changes substantially, this - # list should be revisited. .venv/bin/python -c " import nemo_platform_plugin import nemo_platform_plugin.cli @@ -290,21 +305,56 @@ jobs: print('nemo_platform_plugin', getattr(nemo_platform_plugin, '__version__', '')) " - - name: Upload wheel + # E2E tests: install nemo-platform from the wheel, start services, and + # run the e2e suite against the real process. Only runs on push to main, + # merge queue, and manual dispatch (not on PRs). + python-e2e-test: + name: Python e2e tests + needs: [wheel-build] + if: github.event_name != 'pull_request' + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout code + uses: actions/checkout@v6 + - name: Install uv + uses: astral-sh/setup-uv@v7 + with: + python-version: "3.11" + enable-cache: true + - name: Download wheel + uses: actions/download-artifact@v8 + with: + name: nemo-platform-wheel-py3.11 + path: ${{ runner.temp }}/wheelcheck + - name: Install nemo-platform from wheel + shell: bash + env: + NMP_DATA_DIR: ${{ runner.temp }}/nemo-data + run: | + set -euo pipefail + WHEEL="$(ls ${RUNNER_TEMP}/wheelcheck/*.whl)" + uv tool install --force --python 3.11 "${WHEEL}[services]" + - name: Run e2e tests + run: | + uv run --frozen pytest e2e -v --run-e2e --junitxml=report.xml + env: + _TYPER_FORCE_DISABLE_TERMINAL: "1" + - name: Upload test artifacts if: always() uses: actions/upload-artifact@v6 with: - name: ${{ matrix.package }}-wheel-py${{ matrix.python-version }} - path: ${{ steps.build.outputs.wheel-path }} - retention-days: 7 - if-no-files-found: error + name: python-e2e-test-results + retention-days: 30 + path: | + report.xml # Required-check pin: branch protection should reference this aggregator # rather than the per-row matrix jobs, so the matrix can grow or shrink # without admin intervention. Skipped counts as pass. wheel-test-aggregate: name: Wheel build + test - needs: [wheel-test] + needs: [wheel-smoke-test] if: always() runs-on: ubuntu-latest permissions: @@ -313,20 +363,16 @@ jobs: - name: Check matrix result shell: bash env: - MATRIX_RESULT: ${{ needs.wheel-test.result }} - # Per-row results, useful for debugging which (package, python) - # combination failed. GitHub doesn't expose individual matrix - # results by name; the workflow run UI is the canonical place - # to look. We surface a pointer in the failure message. + MATRIX_RESULT: ${{ needs.wheel-smoke-test.result }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | set -euo pipefail case "${MATRIX_RESULT}" in success|skipped) - echo "wheel-test matrix: ${MATRIX_RESULT}" + echo "wheel-smoke-test matrix: ${MATRIX_RESULT}" ;; *) - echo "::error::wheel-test matrix concluded ${MATRIX_RESULT}" + echo "::error::wheel-smoke-test matrix concluded ${MATRIX_RESULT}" echo "::error::see per-row results: ${RUN_URL}" exit 1 ;; diff --git a/Makefile b/Makefile index 202edfc314..7709b62d2d 100644 --- a/Makefile +++ b/Makefile @@ -269,7 +269,7 @@ test-all-script: ## Run all unit tests using the helper script (with summary) .PHONY: test-e2e test-e2e: ## Run e2e tests against nemo services (starts/stops services automatically) @echo "Running e2e tests..." - uv run --frozen pytest e2e -v --run-e2e $(PYTEST_EXTRA) + uv run --frozen pytest e2e -v --run-e2e --junitxml=report.xml $(PYTEST_EXTRA) .PHONY: test-regression test-regression: ## Run Python regression tests (functional microservice baseline tests) diff --git a/e2e/conftest.py b/e2e/conftest.py index 39995f4611..e2ebf2d6e3 100644 --- a/e2e/conftest.py +++ b/e2e/conftest.py @@ -19,6 +19,7 @@ import logging import os +import shutil import socket import subprocess import sys @@ -73,7 +74,9 @@ def _services() -> Iterator[str]: port = _find_free_port() url = f"http://127.0.0.1:{port}" - nemo_bin = str(Path(sys.executable).parent / "nemo") + # Prefer `nemo` on PATH (e.g. installed from a wheel via `uv tool install`) + # over the one next to sys.executable (the dev venv). + nemo_bin = shutil.which("nemo") or str(Path(sys.executable).parent / "nemo") args = [ nemo_bin, "services", From c1f8c67d10c54ae551627fab9000fe7165a8500d Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:40:56 -0700 Subject: [PATCH 10/19] run latest version Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 030fe2101e..9d08bdd3e8 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -320,12 +320,12 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 with: - python-version: "3.11" + python-version: "3.13" enable-cache: true - name: Download wheel uses: actions/download-artifact@v8 with: - name: nemo-platform-wheel-py3.11 + name: nemo-platform-wheel-py3.13 path: ${{ runner.temp }}/wheelcheck - name: Install nemo-platform from wheel shell: bash @@ -334,7 +334,7 @@ jobs: run: | set -euo pipefail WHEEL="$(ls ${RUNNER_TEMP}/wheelcheck/*.whl)" - uv tool install --force --python 3.11 "${WHEEL}[services]" + uv tool install --force --python 3.13 "${WHEEL}[services]" - name: Run e2e tests run: | uv run --frozen pytest e2e -v --run-e2e --junitxml=report.xml From ed36b2d29d412bde2d53cce54e654bc29a4208fb Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:45:14 -0700 Subject: [PATCH 11/19] bring into PRs Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 9d08bdd3e8..008c304711 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -311,7 +311,6 @@ jobs: python-e2e-test: name: Python e2e tests needs: [wheel-build] - if: github.event_name != 'pull_request' runs-on: ubuntu-latest timeout-minutes: 15 steps: From 6bc35979c62e5ba75330d3de8506b8e1fa7a1b8d Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:46:32 -0700 Subject: [PATCH 12/19] comments Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 008c304711..8b024496d1 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -244,7 +244,7 @@ jobs: if-no-files-found: error # Smoke-test each wheel: install from the artifact and run basic CLI / - # import checks. This replaces the old combined wheel-test job. + # import checks. wheel-smoke-test: name: ${{ matrix.package }} wheel smoke test (py${{ matrix.python-version }}) needs: [wheel-build] @@ -306,8 +306,7 @@ jobs: " # E2E tests: install nemo-platform from the wheel, start services, and - # run the e2e suite against the real process. Only runs on push to main, - # merge queue, and manual dispatch (not on PRs). + # run the e2e suite against the real process. python-e2e-test: name: Python e2e tests needs: [wheel-build] From 87eea1fdf701afc66dc36c131801d41333a02ba8 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 11:48:34 -0700 Subject: [PATCH 13/19] fixes Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 6 ++++++ TESTING.md | 4 ++-- e2e/test_smoke.py | 17 ++++++++--------- 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 8b024496d1..bc16d3c925 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -233,6 +233,8 @@ jobs: package: ${{ matrix.package }} out-dir: ${{ github.workspace }}/dist cadence: nightly + # Sentinel epoch — the wheel version is meaningless for a CI test + # build, but stamp_sdk_version.py requires \d{14}. nightly-timestamp: "19700101000000" python-version: ${{ matrix.python-version }} - name: Upload wheel @@ -298,6 +300,10 @@ jobs: uv venv .venv --python "${PYTHON_VERSION}" uv pip install --python .venv/bin/python "${WHEEL}" unset PYTHONPATH VIRTUAL_ENV + # Import the package + a couple of representative submodules + # (cli, commands). These are the surfaces a plugin author would + # touch first; if any of them fail to import, the wheel is + # broken in a way that surfaces immediately on day one. .venv/bin/python -c " import nemo_platform_plugin import nemo_platform_plugin.cli diff --git a/TESTING.md b/TESTING.md index 8cfdde69fa..124c57b04c 100644 --- a/TESTING.md +++ b/TESTING.md @@ -117,7 +117,7 @@ def test_create_and_fetch_entity(client, db_session): **Characteristics**: -- Start the platform via `nemo services start` (real process, real ports) +- Start the platform via `nemo services run` (real process, real ports) - Hit services with an external HTTP client (the NeMoPlatform SDK) - Test startup machinery, port binding, config resolution, and cross-service workflows - Slower than integration tests (tens of seconds for startup) but faster than Docker/K8s e2e @@ -135,7 +135,7 @@ uv run --frozen pytest e2e -v --run-e2e NMP_BASE_URL=http://localhost:8080 uv run --frozen pytest e2e -v --run-e2e ``` -**Prerequisites**: `make bootstrap` must have been run. The harness runs `nemo services start` +**Prerequisites**: `make bootstrap` must have been run. The harness runs `nemo services run` on a free port with a unique `--instance` name, so it won't conflict with your dev instance. **When to Write E2E Tests**: diff --git a/e2e/test_smoke.py b/e2e/test_smoke.py index 46d3db4645..e7cc47b6d3 100644 --- a/e2e/test_smoke.py +++ b/e2e/test_smoke.py @@ -4,6 +4,8 @@ that services are up. Add more substantive tests in separate files. """ +import uuid + from nemo_platform import NeMoPlatform @@ -19,15 +21,12 @@ def test_health_live(sdk: NeMoPlatform): assert resp.status_code == 200 -def test_create_and_delete_workspace(workspace: str, sdk: NeMoPlatform): - """Workspace CRUD round-trips through the platform. - - Uses the ``workspace`` fixture which creates a unique workspace - and deletes it on teardown. - """ - page = sdk.workspaces.list() - names = [w.name for w in page.data] - assert workspace in names +def test_create_and_delete_workspace(sdk: NeMoPlatform): + """Workspace create and delete round-trips through the platform.""" + name = f"e2e-smoke-{uuid.uuid4().hex[:8]}" + ws = sdk.workspaces.create(name=name) + assert ws.name == name + sdk.workspaces.delete(name) def test_list_workspaces(sdk: NeMoPlatform, workspace: str): From 77656215fddfc853a6a6664d720f92f9f032206b Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 12:24:56 -0700 Subject: [PATCH 14/19] fixes Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 29 ++++++++++------------------- e2e/conftest.py | 5 +---- 2 files changed, 11 insertions(+), 23 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index bc16d3c925..3843e8250c 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -311,37 +311,28 @@ jobs: print('nemo_platform_plugin', getattr(nemo_platform_plugin, '__version__', '')) " - # E2E tests: install nemo-platform from the wheel, start services, and - # run the e2e suite against the real process. + # E2E tests: start services from the workspace venv and run the e2e + # suite against the real process. python-e2e-test: name: Python e2e tests - needs: [wheel-build] + needs: [policy-wasm] runs-on: ubuntu-latest timeout-minutes: 15 steps: - name: Checkout code uses: actions/checkout@v6 + - name: Download policy WASM + uses: actions/download-artifact@v8 + with: + name: policy-wasm + path: services/core/auth/src/nmp/core/auth/assets - name: Install uv uses: astral-sh/setup-uv@v7 with: - python-version: "3.13" + python-version: "3.11" enable-cache: true - - name: Download wheel - uses: actions/download-artifact@v8 - with: - name: nemo-platform-wheel-py3.13 - path: ${{ runner.temp }}/wheelcheck - - name: Install nemo-platform from wheel - shell: bash - env: - NMP_DATA_DIR: ${{ runner.temp }}/nemo-data - run: | - set -euo pipefail - WHEEL="$(ls ${RUNNER_TEMP}/wheelcheck/*.whl)" - uv tool install --force --python 3.13 "${WHEEL}[services]" - name: Run e2e tests - run: | - uv run --frozen pytest e2e -v --run-e2e --junitxml=report.xml + run: make test-e2e env: _TYPER_FORCE_DISABLE_TERMINAL: "1" - name: Upload test artifacts diff --git a/e2e/conftest.py b/e2e/conftest.py index e2ebf2d6e3..39995f4611 100644 --- a/e2e/conftest.py +++ b/e2e/conftest.py @@ -19,7 +19,6 @@ import logging import os -import shutil import socket import subprocess import sys @@ -74,9 +73,7 @@ def _services() -> Iterator[str]: port = _find_free_port() url = f"http://127.0.0.1:{port}" - # Prefer `nemo` on PATH (e.g. installed from a wheel via `uv tool install`) - # over the one next to sys.executable (the dev venv). - nemo_bin = shutil.which("nemo") or str(Path(sys.executable).parent / "nemo") + nemo_bin = str(Path(sys.executable).parent / "nemo") args = [ nemo_bin, "services", From bd5012c3bbf2f98566f604a3f81de683d98cc918 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 12:26:07 -0700 Subject: [PATCH 15/19] add e2e test Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 3843e8250c..9587a16f56 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -329,7 +329,7 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 with: - python-version: "3.11" + python-version: "3.13" enable-cache: true - name: Run e2e tests run: make test-e2e From 539cf942e507fbda478f50cfa6116cc7afb966ba Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 13:25:05 -0700 Subject: [PATCH 16/19] fixes Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 8 +++++ e2e/conftest.py | 71 ++++++++++++++++++++------------------- e2e/test_smoke.py | 6 ++-- 3 files changed, 48 insertions(+), 37 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 9587a16f56..74daa7753e 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -335,6 +335,13 @@ jobs: run: make test-e2e env: _TYPER_FORCE_DISABLE_TERMINAL: "1" + E2E_SERVICES_LOG: ${{ runner.temp }}/services.log + - name: Dump server logs + if: always() + run: | + echo "::group::Server log" + cat "${{ runner.temp }}/services.log" 2>/dev/null || echo "No server log found" + echo "::endgroup::" - name: Upload test artifacts if: always() uses: actions/upload-artifact@v6 @@ -343,6 +350,7 @@ jobs: retention-days: 30 path: | report.xml + ${{ runner.temp }}/services.log # Required-check pin: branch protection should reference this aggregator # rather than the per-row matrix jobs, so the matrix can grow or shrink diff --git a/e2e/conftest.py b/e2e/conftest.py index 39995f4611..810a299825 100644 --- a/e2e/conftest.py +++ b/e2e/conftest.py @@ -17,15 +17,18 @@ terminates the process after the session. """ +import contextlib import logging import os import socket import subprocess import sys +import tempfile import time import uuid from collections.abc import Iterator from pathlib import Path +from typing import IO, Any import httpx import pytest @@ -35,6 +38,7 @@ _HEALTH_TIMEOUT = 60 _HEALTH_POLL_INTERVAL = 1.0 +_SERVICES_LOG = Path(os.environ.get("E2E_SERVICES_LOG", os.path.join(tempfile.gettempdir(), "services.log"))) def _find_free_port() -> int: @@ -53,17 +57,37 @@ def _wait_for_healthy(url: str, timeout: float = _HEALTH_TIMEOUT) -> bool: if resp.status_code == 200: return True except httpx.RequestError: - pass + pass # Server not up yet, keep polling time.sleep(_HEALTH_POLL_INTERVAL) return False +@contextlib.contextmanager +def background_process(args: list[str], stdout: IO[Any] | None = None) -> Iterator[subprocess.Popen]: + """Run a subprocess, yield the ``Popen``, and terminate on exit. + + Unlike ``Popen``'s built-in context manager (which only waits for the + process), this sends SIGTERM/SIGKILL so long-running servers are + cleaned up. + """ + proc = subprocess.Popen(args, stdout=stdout, stderr=subprocess.STDOUT) + try: + yield proc + finally: + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + logger.warning("Process %d did not exit after SIGTERM, sending SIGKILL", proc.pid) + proc.kill() + proc.wait(timeout=5) + + @pytest.fixture(scope="session") def _services() -> Iterator[str]: """Spawn ``nemo services run`` and yield the base URL. Skipped when ``NMP_BASE_URL`` is already set (external services). - The process is our direct child — on teardown we just terminate it. """ external_url = os.environ.get("NMP_BASE_URL") if external_url: @@ -74,43 +98,20 @@ def _services() -> Iterator[str]: url = f"http://127.0.0.1:{port}" nemo_bin = str(Path(sys.executable).parent / "nemo") - args = [ - nemo_bin, - "services", - "run", - "--service-group", - "all", - "--port", - str(port), - ] + args = [nemo_bin, "services", "run", "--service-group", "all", "--port", str(port)] logger.info("Starting nemo services on port %d", port) - proc = subprocess.Popen( - args, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - ) - - if not _wait_for_healthy(url): - # Grab whatever output we have for diagnostics - proc.terminate() - stdout, _ = proc.communicate(timeout=10) - pytest.fail(f"nemo services run did not become healthy within {_HEALTH_TIMEOUT}s.\nstdout:\n{stdout}") - - logger.info("Platform services ready on port %d (pid %d)", port, proc.pid) - - yield url + log_path = _SERVICES_LOG + with open(log_path, "w") as log_file, background_process(args, stdout=log_file) as proc: + if not _wait_for_healthy(url): + pytest.fail( + f"nemo services run did not become healthy within {_HEALTH_TIMEOUT}s.\nlog:\n{log_path.read_text()}" + ) - logger.info("Terminating nemo services (pid %d)", proc.pid) - proc.terminate() - try: - proc.wait(timeout=10) - except subprocess.TimeoutExpired: - logger.warning("Process did not exit after SIGTERM, sending SIGKILL") - proc.kill() - proc.wait(timeout=5) + logger.info("Platform services ready on port %d (pid %d)", port, proc.pid) + yield url + logger.info("Terminating nemo services (pid %d)", proc.pid) @pytest.fixture(scope="session") diff --git a/e2e/test_smoke.py b/e2e/test_smoke.py index e7cc47b6d3..322bd1efd3 100644 --- a/e2e/test_smoke.py +++ b/e2e/test_smoke.py @@ -25,8 +25,10 @@ def test_create_and_delete_workspace(sdk: NeMoPlatform): """Workspace create and delete round-trips through the platform.""" name = f"e2e-smoke-{uuid.uuid4().hex[:8]}" ws = sdk.workspaces.create(name=name) - assert ws.name == name - sdk.workspaces.delete(name) + try: + assert ws.name == name + finally: + sdk.workspaces.delete(name) def test_list_workspaces(sdk: NeMoPlatform, workspace: str): From 14c80052630777ceaa1d903f2f46d9986ed560fb Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 13:33:20 -0700 Subject: [PATCH 17/19] add testing.md Signed-off-by: Matthew Grossman --- TESTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TESTING.md b/TESTING.md index 124c57b04c..34fcd2150c 100644 --- a/TESTING.md +++ b/TESTING.md @@ -135,8 +135,8 @@ uv run --frozen pytest e2e -v --run-e2e NMP_BASE_URL=http://localhost:8080 uv run --frozen pytest e2e -v --run-e2e ``` -**Prerequisites**: `make bootstrap` must have been run. The harness runs `nemo services run` -on a free port with a unique `--instance` name, so it won't conflict with your dev instance. +**Prerequisites**: `make bootstrap` must have been run. The harness spawns `nemo services run` +on a free port, so it won't conflict with your dev instance. **When to Write E2E Tests**: From 4e24680337947e543522e238169e44fd001d3283 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 13:36:56 -0700 Subject: [PATCH 18/19] fix name Signed-off-by: Matthew Grossman --- .github/workflows/ci.yaml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 74daa7753e..0973fc23e9 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -203,7 +203,7 @@ jobs: coverage.json # Build wheels for all packages × python versions. Downstream jobs - # (wheel-smoke-test, python-e2e-test) download these artifacts instead + # (wheel-test, python-e2e-test) download these artifacts instead # of rebuilding. wheel-build: name: ${{ matrix.package }} wheel build (py${{ matrix.python-version }}) @@ -245,10 +245,10 @@ jobs: retention-days: 7 if-no-files-found: error - # Smoke-test each wheel: install from the artifact and run basic CLI / + # Test each wheel: install from the artifact and run basic CLI / # import checks. - wheel-smoke-test: - name: ${{ matrix.package }} wheel smoke test (py${{ matrix.python-version }}) + wheel-test: + name: ${{ matrix.package }} wheel build + test (py${{ matrix.python-version }}) needs: [wheel-build] runs-on: ubuntu-latest timeout-minutes: 15 @@ -357,7 +357,7 @@ jobs: # without admin intervention. Skipped counts as pass. wheel-test-aggregate: name: Wheel build + test - needs: [wheel-smoke-test] + needs: [wheel-test] if: always() runs-on: ubuntu-latest permissions: @@ -366,16 +366,16 @@ jobs: - name: Check matrix result shell: bash env: - MATRIX_RESULT: ${{ needs.wheel-smoke-test.result }} + MATRIX_RESULT: ${{ needs.wheel-test.result }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | set -euo pipefail case "${MATRIX_RESULT}" in success|skipped) - echo "wheel-smoke-test matrix: ${MATRIX_RESULT}" + echo "wheel-test matrix: ${MATRIX_RESULT}" ;; *) - echo "::error::wheel-smoke-test matrix concluded ${MATRIX_RESULT}" + echo "::error::wheel-test matrix concluded ${MATRIX_RESULT}" echo "::error::see per-row results: ${RUN_URL}" exit 1 ;; From e60a061cfa2a459b5223f424bf5083e7c7ac4356 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Mon, 1 Jun 2026 14:29:33 -0700 Subject: [PATCH 19/19] add comment Signed-off-by: Matthew Grossman --- e2e/conftest.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/e2e/conftest.py b/e2e/conftest.py index 810a299825..49fbffb06c 100644 --- a/e2e/conftest.py +++ b/e2e/conftest.py @@ -88,6 +88,12 @@ def _services() -> Iterator[str]: """Spawn ``nemo services run`` and yield the base URL. Skipped when ``NMP_BASE_URL`` is already set (external services). + + This is the "subprocess" backend. When we add Docker and Kubernetes + backends, this fixture should be replaced by a backend-selection layer + (e.g. ``--docker`` / ``--kubernetes`` CLI flags) that dispatches to the + appropriate setup while yielding the same base URL interface. Tests + should remain agnostic to the backend. """ external_url = os.environ.get("NMP_BASE_URL") if external_url: