Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
name: nemo-agent-local
name: nemo-studio-copilot
created_timestamp: 2026-07-28T20:09:49Z
author: Danielle Ali and Codex
---

# Agent Spec: nemo-agent-local
# Agent Spec: nemo-studio-copilot

> This file is the durable contract for the local NeMo Platform assistant.
> Keep it aligned with the implementation under `agents/nemo-agent-local/`.
> Keep it aligned with the implementation under `agents/nemo-studio-copilot/`.

## Role

Expand Down Expand Up @@ -81,15 +81,15 @@ The mission is grounded in the current deployment proof of concept and its imple

## Evaluation Setup

Unit coverage lives in `agents/nemo-agent-local/tests/test_nemo_agent.py`. Run it with:
Unit coverage lives in `agents/nemo-studio-copilot/tests/test_nemo_studio_copilot.py`. Run it with:

```bash
uv run --frozen pytest agents/nemo-agent-local/tests/test_nemo_agent.py -v
uv run --frozen pytest agents/nemo-studio-copilot/tests/test_nemo_studio_copilot.py -v
```

The NAT evaluation workflow is `agents/nemo-agent-local/src/nemo_agent/nemo-eval.yml`, with inputs in `nemo-eval-data.json`. It currently evaluates workspace creation, workspace listing, and model/provider listing with a tunable RAG judge weighted for coverage (0.5), correctness (0.3), and relevance (0.2). No formal aggregate pass threshold is defined.
The NAT evaluation workflow is `agents/nemo-studio-copilot/src/nemo_studio_copilot/nemo-studio-copilot-eval.yml`, with inputs in `nemo-studio-copilot-eval-data.json`. It currently evaluates workspace creation, workspace listing, and model/provider listing with a tunable RAG judge weighted for coverage (0.5), correctness (0.3), and relevance (0.2). No formal aggregate pass threshold is defined.

Manual Studio validation is documented in `agents/nemo-agent-local/tests/smoke_test.md`. Current coverage gaps include fileset listing, missing-workspace clarification, fast-path failure containment, destructive-action ambiguity, iteration limits, retry limits, cancellation, and end-to-end latency thresholds.
Manual Studio validation is documented in `agents/nemo-studio-copilot/tests/smoke_test.md`. Current coverage gaps include fileset listing, missing-workspace clarification, fast-path failure containment, destructive-action ambiguity, iteration limits, retry limits, cancellation, and end-to-end latency thresholds.

## Change Scope

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ build/
.venv/
node_modules/
tests/
src/nemo_agent/nemo-eval.yml
src/nemo_agent/nemo-eval-data.json
src/nemo_studio_copilot/nemo-studio-copilot-eval.yml
src/nemo_studio_copilot/nemo-studio-copilot-eval-data.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,19 +41,19 @@ RUN --mount=type=cache,id=uv_cache,target=/root/.cache/uv,sharing=locked \
uv pip install . && \
chmod -R a+rX /opt/uv /workspace/.venv

LABEL org.opencontainers.image.title="builtin-nemo-agent" \
LABEL org.opencontainers.image.title="nemo-studio-copilot" \
org.opencontainers.image.version="0.1.0" \
org.opencontainers.image.authors="NVIDIA" \
org.opencontainers.image.created="2026-07-22T20:51:42.179931+00:00" \
org.opencontainers.image.description="Built-in NeMo Platform agent — a LangChain Deep Agent that uses the NeMo SDK to perform platform tasks." \
org.opencontainers.image.description="NeMo Copilot — a LangChain Deep Agent that uses the NeMo SDK to perform platform tasks." \
org.opencontainers.image.revision="" \
org.opencontainers.image.source="" \
com.nemo.agent.id="790bd61b98d0" \
com.nemo.agent.framework="nemo_agent_toolkit" \
com.nemo.agent.nat-version="1.8.0" \
com.nemo.agent.contract-version="0.0.0"

ENV NAT_CONFIG_FILE=/workspace/src/nemo_agent/nemo-agent.yml
ENV NAT_CONFIG_FILE=/workspace/src/nemo_studio_copilot/nemo-studio-copilot.yml

ENV PATH="/workspace/.venv/bin:$PATH"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "builtin-nemo-agent"
name = "nemo-studio-copilot"
version = "0.1.0"
description = "Built-in NeMo Platform agent — a LangChain Deep Agent that uses the NeMo SDK to perform platform tasks."
description = "NeMo Copilot — a LangChain Deep Agent that uses the NeMo SDK to perform platform tasks."
requires-python = ">=3.11,<3.14"
dependencies = [
"nvidia-nat-core>=1.8.0,<1.9",
Expand All @@ -21,13 +21,13 @@ dependencies = [
]

[project.entry-points."nat.components"]
nemo_agents_nemo_agent = "nemo_agent.register"
# Custom NAT workflow type that wraps the nemo-agent graph with permissive
# input/output schemas. See src/nemo_agent/wrapper.py for the full rationale —
nemo_studio_copilot = "nemo_studio_copilot.register"
# Custom NAT workflow type that wraps the nemo-studio-copilot graph with permissive
# input/output schemas. See src/nemo_studio_copilot/wrapper.py for the full rationale —
# in short, NAT 1.6.0's stock `langgraph_wrapper` doesn't accept the
# `{"input_message": ...}` shape that `nvidia-nat-eval` posts and crashes on
# deep-agent state deltas that don't carry `messages`.
nemo_agents_nemo_agent_wrapper = "nemo_agent.wrapper"
nemo_studio_copilot_wrapper = "nemo_studio_copilot.wrapper"

[tool.hatch.build.targets.wheel]
packages = ["src/nemo_agent"]
packages = ["src/nemo_studio_copilot"]
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Eval config for the built-in nemo-agent.
# Eval config for the built-in nemo-studio-copilot.
#
# Uses the custom `nemo_agent_wrapper` workflow type (defined in
# src/nemo_agent/wrapper.py) so the same config works whether the agent runs
# Uses the custom `nemo_studio_copilot_wrapper` workflow type (defined in
# src/nemo_studio_copilot/wrapper.py) so the same config works whether the agent runs
# inline (no --agent) or against the deployed agent's `/generate/full` endpoint
# (--agent nemo-agent). The wrapper accepts both the `{"messages": [...]}`
# (--agent nemo-studio-copilot). The wrapper accepts both the `{"messages": [...]}`
# shape used by chat clients and the `{"input_message": "..."}` shape that
# `nvidia-nat-eval`'s remote workflow client posts to `/generate/full`.
#
Expand All @@ -13,13 +13,13 @@
#
# Run inline (in-process, no deployed agent needed):
# nemo agents evaluate run \
# --eval-config plugins/nemo-agents/builtin_agents/nemo-agent/src/nemo_agent/nemo-eval.yml \
# --eval-config agents/nemo-studio-copilot/src/nemo_studio_copilot/nemo-studio-copilot-eval.yml \
# --output ./eval-out
#
# Run against the deployed agent (requires `nemo agents deploy nemo-agent`):
# Run against the deployed agent (requires `nemo agents deploy nemo-studio-copilot`):
# nemo agents evaluate run \
# --eval-config plugins/nemo-agents/builtin_agents/nemo-agent/src/nemo_agent/nemo-eval.yml \
# --agent nemo-agent \
# --eval-config agents/nemo-studio-copilot/src/nemo_studio_copilot/nemo-studio-copilot-eval.yml \
# --agent nemo-studio-copilot \
# --output ./eval-out

llms:
Expand Down Expand Up @@ -47,7 +47,7 @@ llms:
# only meaningful when something resolves it back to the upstream model.
#
# Caveat: the agent's tools (`nemo_api`, `check_status`) still call platform
# APIs via the Python SDK. The current `nemo-eval-data.json` exercises
# APIs via the Python SDK. The current `nemo-studio-copilot-eval-data.json` exercises
# platform resources, including workspace and model/provider operations, so
# bypassing the gateway is not an LLM-only smoke test. A fully self-contained
# bypass run requires a separate pure-LLM dataset that does not invoke
Expand All @@ -67,16 +67,16 @@ llms:
# max_tokens: 1024

workflow:
_type: nemo_agent_wrapper
_type: nemo_studio_copilot_wrapper
description: NeMo Platform assistant using LangChain Deep Agents

eval:
general:
max_concurrency: 1
output_dir: eval/nemo-agent
output_dir: eval/nemo-studio-copilot
dataset:
_type: json
file_path: nemo-eval-data.json
file_path: nemo-studio-copilot-eval-data.json
structure:
question_key: input_message
answer_key: expected_output
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ llms:
request_timeout: 120

workflow:
_type: nemo_agent_wrapper
_type: nemo_studio_copilot_wrapper
description: NeMo Platform assistant using LangChain Deep Agents
Original file line number Diff line number Diff line change
@@ -1,18 +1,17 @@
# NeMo Platform Agent — NAT workflow configuration
# NeMo Copilot — NAT workflow configuration
#
# A LangChain Deep Agent that uses the NeMo SDK to perform platform tasks.
# Deliberately configured as the "expensive baseline" for the optimization
# pipeline to compress.
#
# Deploy:
# uv pip install -e plugins/nemo-agents
# uv pip install -e plugins/nemo-agents/builtin_agents/nemo-agent
# nemo agents create --name nemo-agent \
# --agent-config plugins/nemo-agents/builtin_agents/nemo-agent/src/nemo_agent/nemo-agent.yml
# nemo agents deploy --agent nemo-agent
# nemo agents create --name nemo-studio-copilot \
# --agent-config agents/nemo-studio-copilot/src/nemo_studio_copilot/nemo-studio-copilot.yml
# nemo agents deploy --agent nemo-studio-copilot
#
# Invoke:
# nemo agents invoke --agent nemo-agent --input "Create a workspace called demo"
# nemo agents invoke --agent nemo-studio-copilot --input "Create a workspace called demo"

llms:
agent:
Expand Down Expand Up @@ -73,9 +72,9 @@ llms:
# max_tokens: 49152

workflow:
# Custom workflow type registered by `nemo-agents-nemo-agent`. Wraps the same
# Custom workflow type registered by `nemo_studio_copilot_wrapper`. Wraps the same
# deep-agent graph as `langgraph_wrapper` would, but with input/output schemas
# that survive both the eval client's `{"input_message": ...}` payload and the
# deep-agent's wider state shape. See src/nemo_agent/wrapper.py.
_type: nemo_agent_wrapper
# deep-agent's wider state shape. See src/nemo_studio_copilot/wrapper.py.
_type: nemo_studio_copilot_wrapper
description: NeMo Platform assistant using LangChain Deep Agents
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
SKILLS_DIR = Path(__file__).parent / "skills"
DEFAULT_WORKSPACE = "default"
DEFAULT_RECURSION_LIMIT = 24
STUDIO_CALLBACK_PATH = "/studio/api/coding-agents/mcp/{session_id}"
STUDIO_CALLBACK_PATH = "/studio/api/copilot/mcp/{session_id}"
STUDIO_CALLBACK_TIMEOUT_SECONDS = 3600.0
_READ_ONLY_SDK_ACTIONS = frozenset(
{
Expand Down Expand Up @@ -80,14 +80,38 @@
)
_FAST_PATH_ANALYSIS = re.compile(r"\b(analy[sz]e|audit|compare|explain|investigate|why)\b", re.IGNORECASE)
_DIRECT_LIST_RESOURCES = (
(re.compile(r"\bworkspaces?\b", re.IGNORECASE), "workspaces"),
(re.compile(r"\bmodels?\b", re.IGNORECASE), "models"),
# Match the requested resource before the contextual workspace qualifier.
# Provider prompts commonly include both "model" and "workspace", so the
# most specific resource must come first and workspace must come last.
(re.compile(r"\bproviders?\b", re.IGNORECASE), "inference.providers"),
(re.compile(r"\bfilesets?\b", re.IGNORECASE), "files.filesets"),
(re.compile(r"\bdatasets?\b", re.IGNORECASE), "datasets"),
(re.compile(r"\bbenchmarks?\b", re.IGNORECASE), "evaluation.benchmarks"),
(re.compile(r"\bmetrics?\b", re.IGNORECASE), "evaluation.metrics"),
(re.compile(r"\bmodels?\b", re.IGNORECASE), "models"),
(re.compile(r"\bworkspaces?\b", re.IGNORECASE), "workspaces"),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
)
_DIRECT_LIST_CONTEXT_TARGET = re.compile(
r"^\s*(?:list|show|what|which)\b.*?\b(?:"
r"(?P<providers>(?:model\s+)?providers?)|"
r"(?P<filesets>filesets?)|"
r"(?P<datasets>datasets?)|"
r"(?P<benchmarks>benchmarks?)|"
r"(?P<metrics>metrics?)|"
r"(?P<models>models?)|"
r"(?P<workspaces>workspaces?)"
r")\b",
re.IGNORECASE,
)
_DIRECT_LIST_CONTEXT_RESOURCES = {
"providers": "inference.providers",
"filesets": "files.filesets",
"datasets": "datasets",
"benchmarks": "evaluation.benchmarks",
"metrics": "evaluation.metrics",
"models": "models",
"workspaces": "workspaces",
}

_client: NeMoPlatform | None = None

Expand All @@ -108,6 +132,9 @@ def _direct_list_resource(messages: list[Any]) -> str | None:
return None
if _FAST_PATH_MUTATION.search(latest_user_text) or _FAST_PATH_ANALYSIS.search(latest_user_text):
return None
context_target = _DIRECT_LIST_CONTEXT_TARGET.search(latest_user_text)
if context_target and context_target.lastgroup:
return _DIRECT_LIST_CONTEXT_RESOURCES[context_target.lastgroup]
for pattern, resource_path in _DIRECT_LIST_RESOURCES:
if pattern.search(latest_user_text):
return resource_path
Expand Down Expand Up @@ -147,7 +174,7 @@ def _list_resource_names(resource_path: str) -> str:
resource = _resolve_resource(_get_client(), resource_path)
serialized = _serialize(_call_sdk_method(resource, "list"))
except Exception as exc:
logger.exception("nemo-agent: direct list fast path failed for resource=%s", resource_path)
logger.exception("nemo-studio-copilot: direct list fast path failed for resource=%s", resource_path)
return _direct_list_error(resource_path, exc)

if isinstance(serialized, dict):
Expand All @@ -157,7 +184,7 @@ def _list_resource_names(resource_path: str) -> str:
names = [item.get("name") for item in serialized if isinstance(item, dict) and isinstance(item.get("name"), str)]
if not names:
return f"No {resource_path.rsplit('.', maxsplit=1)[-1]} found in workspace '{_active_workspace()}'."
logger.info("nemo-agent: direct list fast path resource=%s count=%d", resource_path, len(names))
logger.info("nemo-studio-copilot: direct list fast path resource=%s count=%d", resource_path, len(names))
return "\n".join(names)


Expand All @@ -169,7 +196,7 @@ def _delete_fileset(name: str) -> str:
_call_sdk_method(resource, "delete", {"name": name})
serialized = _serialize(_call_sdk_method(resource, "list"))
except Exception as exc:
logger.exception("nemo-agent: direct fileset delete failed for name=%s", name)
logger.exception("nemo-studio-copilot: direct fileset delete failed for name=%s", name)
if isinstance(exc, ValueError) and "workspace" in str(exc).lower():
return f"Which workspace should I use to delete fileset '{name}'?"
return f"I couldn't delete fileset '{name}' in workspace '{workspace}': {type(exc).__name__}: {exc}"
Expand All @@ -187,7 +214,7 @@ def _delete_fileset(name: str) -> str:
if name in remaining_names:
return f"I couldn't verify deletion: fileset '{name}' still exists in workspace '{workspace}'."

logger.info("nemo-agent: direct fileset delete name=%s workspace=%s", name, workspace)
logger.info("nemo-studio-copilot: direct fileset delete name=%s workspace=%s", name, workspace)
return f"Deleted fileset '{name}' from workspace '{workspace}'."


Expand Down Expand Up @@ -663,7 +690,7 @@ def check_status(service: str, job_name: str) -> str:
class _StreamSafeGraph:
"""Wraps a CompiledStateGraph with NAT-compatible incremental streaming.

The custom ``nemo_agent_wrapper`` accepts Deep Agent state deltas that do
The custom ``nemo_studio_copilot_wrapper`` accepts Deep Agent state deltas that do
not contain ``messages``. Delegate to the graph's real ``astream`` so NAT
can forward chunks as each graph step completes instead of buffering the
entire run behind ``ainvoke``.
Expand Down Expand Up @@ -778,7 +805,9 @@ def _disable_nat_method_retries(model: Any) -> Any:
object.__setattr__(model, name, types.MethodType(wrapped, model))
disabled.append(name)
if disabled:
logger.info("nemo-agent: disabled NAT automatic retry wrappers for model methods: %s", ", ".join(disabled))
logger.info(
"nemo-studio-copilot: disabled NAT automatic retry wrappers for model methods: %s", ", ".join(disabled)
)
return model


Expand Down Expand Up @@ -829,7 +858,7 @@ def _build_backend() -> CompositeBackend:
return CompositeBackend(default=StateBackend(), routes=routes)


def create_nemo_agent(config=None):
def create_nemo_studio_copilot(config=None):
"""Create the NeMo Platform Deep Agent.

Args:
Expand All @@ -845,14 +874,14 @@ def create_nemo_agent(config=None):
skills = [str(SKILLS_DIR)] if discovered else None
if SKILLS_DIR.is_dir() and not discovered:
logger.warning(
"nemo-agent: SKILLS_DIR=%s is configured but no spec-compliant "
"nemo-studio-copilot: SKILLS_DIR=%s is configured but no spec-compliant "
"skills were discovered (each skill must be a subdirectory with "
"a SKILL.md starting with '---' YAML frontmatter). Agent will "
"run without playbook scaffolding.",
SKILLS_DIR,
)
elif discovered:
logger.info("nemo-agent: loaded %d skills: %s", len(discovered), ", ".join(discovered))
logger.info("nemo-studio-copilot: loaded %d skills: %s", len(discovered), ", ".join(discovered))
backend = _build_backend()
graph = create_deep_agent(
model=model,
Expand All @@ -864,10 +893,10 @@ def create_nemo_agent(config=None):
return _StreamSafeGraph(graph)


# Module-level graph factory. The active workflow type is `nemo_agent_wrapper`
# (see ./wrapper.py), which calls ``create_nemo_agent`` directly. This alias is
# Module-level graph factory. The active workflow type is `nemo_studio_copilot_wrapper`
# (see ./wrapper.py), which calls ``create_nemo_studio_copilot`` directly. This alias is
# kept so the graph can also be loaded via ``_type: langgraph_wrapper`` with
# ``graph: .../register.py:agent`` for ad-hoc debugging — note that path will
# hit the NAT 1.6.0 input/output schema bugs that ``nemo_agent_wrapper``
# hit the NAT 1.6.0 input/output schema bugs that ``nemo_studio_copilot_wrapper``
# exists to work around.
agent = create_nemo_agent
agent = create_nemo_studio_copilot
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: "Benchmark task execution contract: complete every numbered require
---
# Benchmark execution contract

This skill defines the execution requirements that every nemo-agent run under
This skill defines the execution requirements that every nemo-studio-copilot run under
`tests/agentic-use/` must satisfy so the canonical gate
(`tests/agentic-use/passrate_token_policy_gate.py`) can score the run on
verifier pass-rate and token totals. See
Expand Down
Loading
Loading