diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/SKILL.md index 42e0ba1ef26..f189d900564 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/SKILL.md @@ -3,424 +3,252 @@ # SPDX-License-Identifier: Apache-2.0 name: nemo-build-agent -description: End-to-end NeMo Platform agent implementation from an approved Ethos. Registers and deploys the agent, generates evaluation data, runs evaluation, and signs off. Use for full Ethos-to-deployed-agent work, including builds from an existing legacy NAT workflow. +description: Use when a customer asks NeMo Platform to build, implement, package, register, or deploy a new agent from an idea or approved Ethos. Builds the supported LangChain Deep Agents shape through Fabric. Do not use for design-only work, focused agent.yaml editing, testing an existing deployment, onboarding an existing non-Deep-Agents agent, or maintaining NAT workflows. triggers: - nemo-build-agent - - build the agent - - create the agent - - deploy the agent - - scaffold the agent - - make me an agent - - build an agent on nemo + - build an agent + - implement the approved agent + - package the agent + - deploy the new agent - build from the agent Ethos - - ship the agent - - nemo build - - deploy my existing NAT agent + - onboard the new agent through Fabric not-for: - - nemo-agent-config (use for focused agent.yaml authoring or migration) - - nemo-explore (use to gather design before building) - - nemo-ethos (use to write the Ethos before building) - - nemo-try-agent (use to query an already deployed agent) - - nemo-setup (use to install the platform first) - - deploy-sandbox (use to deploy the built agent as a governed OpenShell sandbox) - - generic agent framework development outside NeMo Platform + - nemo-explore (use for design-only discovery) + - nemo-ethos (use to persist a design without implementation) + - nemo-agent-config (use for focused agent.yaml authoring or validation) + - nemo-try-agent (use to query an existing deployment) + - existing agent onboarding through another Fabric adapter + - migration or continued operation of NVIDIA NeMo Agent Toolkit workflows preconditions: - nemo_setup_complete - workspace_exists - provider_registered - - agents_plugin_available - - ethos_exists -compatibility: nemo-platform >= 0.1.0; running platform; requires agents plugin; writes files under agents/; uses nemo-agents-spec-v1 by default and preserves NAT workflow YAML as a compatibility path; macOS or Linux; safe under sandbox. -maturity: active +compatibility: NeMo Platform >= 0.4.0; installs the optional NeMo Agents plugin when the user approves and verifies its supported Fabric Deep Agents adapter; Python and uv for local MCP projects; network access and provider credentials for live model tests; Docker for packaged custom code; macOS or Linux. +maturity: beta license: Apache-2.0 user-invocable: true allowed-tools: [Bash, Read, Write, Edit] --- -# NeMo Platform agent build - -Build a deployable NeMo Platform agent from an approved `ETHOS.md`. Use -the Platform-managed `nemo-agents-spec-v1` `agent.yaml` path by default. Treat -NAT workflow YAML as a supported compatibility path, not the default output. - -Use `nemo-agent-config` for the machine-readable config shape. Do not expose -Fabric SDK object names or raw runtime configuration to the user. - -## Select the config path - -Choose the config path before pre-flight. Set shared names for either path: - -```bash -AGENT_NAME= -DEPLOYMENT_NAME="${AGENT_NAME}-deployment" -``` - -If the user supplies an existing NAT workflow YAML, ask whether they want to -deploy it unchanged or migrate it best-effort to `nemo-agents-spec-v1` with -`nemo-agent-config`. For an unchanged NAT-only run, preserve the original file -and also set: - -```bash -NAT_WORKFLOW_PATH= -``` - -## Pre-flight - -1. Run the platform probe owned by `nemo-status`. If it reports - `PLATFORM_DOWN` or `PLATFORM_WEDGED`, route to `nemo-setup` and stop. -2. Confirm the agents plugin is loaded: - - ```bash - .venv/bin/nemo agents --help 2>&1 | grep -q "create" - ``` - -3. Check for existing Agent entities and deployments. Ask whether to reuse or - replace them. Follow the lifecycle branches below before create or deploy. -4. For an unchanged NAT-only run, confirm `$NAT_WORKFLOW_PATH` exists and read - it before continuing. Do not require `ETHOS.md` or an Ethos fileset. -5. For the default Platform-managed path, confirm - `agents/$AGENT_NAME-ethos/ETHOS.md` exists. If it does not, route through - `nemo-explore` and `nemo-ethos` first. If - `agents/$AGENT_NAME-spec/AGENT-SPEC.md` exists instead, route to - `nemo-explore`. -6. Read the Ethos and extract the agent name, instructions, capabilities, - model requirements, tools, constraints, and success criteria. -7. Confirm the canonical Ethos fileset exists: - - ```bash - .venv/bin/nemo files filesets get "${AGENT_NAME}-ethos" \ - --workspace "${WORKSPACE:-default}" >/dev/null 2>&1 \ - && echo "ethos_fileset_ok" \ - || { echo "ethos_fileset_missing - run nemo-ethos first"; exit 1; } - ``` - -Steps 5 through 7 apply only to the default Platform-managed path or an explicit -NAT migration. - -### Existing-resource lifecycle - -- **Reuse:** Do not run `agents create` for an existing Agent. If a deployment - already exists, set `DEPLOYMENT_NAME` to its name, do not run `agents deploy`, - and continue to the smoke test. If only the Agent exists, skip create and run - only the deploy command in Step 1. -- **Replace:** Show each destructive command and require explicit confirmation - immediately before running it. Use `--yes` only after that confirmation. If - the resource does not exist, skip its command. - -For a confirmed replacement, undeploy first: - -```bash -.venv/bin/nemo agents undeploy "$DEPLOYMENT_NAME" --yes -``` - -Wait until this command reports that the deployment is absent before -continuing: - -```bash -.venv/bin/nemo agents deployments get "$DEPLOYMENT_NAME" -``` - -Then show the Agent deletion command and require explicit confirmation before +# Build a NeMo Platform agent + +Build a tested, config-driven LangChain Deep Agent and onboard it through the +NeMo Platform Fabric path. Fabric owns runtime orchestration and constructs the +agent from config. The optional NeMo Agents plugin supplies the Fabric harness +adapters and their runtime dependencies. Customer code runs only when it is +packaged as a config-referenced service such as MCP. + +## Explain the path in customer language + +Do not assume the user knows NeMo Platform, Fabric, Ethos, MCP or Agent Skills. +Introduce each term only when it affects a decision. + +Before writing implementation files: + +1. Explain that an Ethos is a plain-language design contract for the agent. It + records who the agent serves, what it should accomplish, the tools and data + it may use, required approvals, forbidden actions, failure behavior and the + examples that will become tests. It is not code or deployment config. +2. Explain that the Ethos will be drafted from the user's answers, shown for + review and revised until the user explicitly approves it. No implementation + begins before that approval. +3. Explain that the supported build path uses the external LangChain Deep + Agents runtime. NeMo Platform does not vendor the customer's agent. The + optional NeMo Agents plugin installs the supported Fabric adapter and its + Deep Agents dependency. Fabric registers the agent and makes it available + for deployment, testing, observation and optimization in NeMo Platform. +4. Explain executable tools only when the requested agent needs them: custom + code is exposed through MCP so the deployed agent can call it. Explain Agent + Skills only when an instruction package is the right artifact. +5. Ask whether to continue with this supported path. + +If the user declines, stop. Offer onboarding of an existing agent through an +available Fabric adapter. Do not fall back to NAT. + +## Approve and persist the design + +Use `agents/-ethos/ETHOS.md` as the canonical design and package +root. If an Ethos exists, summarize it in plain language and confirm that it is +approved for this build. If it is absent or needs changes, gather the design +inputs conversationally. Ask about the customer's work first and translate the +answers into NeMo artifacts yourself: + +- concrete role, users and outcomes; +- tools, data, credentials and side effects; +- constraints, approvals, forbidden actions and sensitive data handling; +- five to ten representative tasks with expected outcomes; +- mandatory ordering or transactional invariants. + +Route unresolved design questions to `nemo-explore`. Then invoke `nemo-ethos` +to render, validate and save `agents/-ethos/ETHOS.md`. Show the +result and wait for explicit approval. Do not create implementation files before +the Ethos is approved. + +## Confirm the build environment + +After the Ethos is approved and before creating implementation files, confirm +the target workspace, environment, model provider, network access, credential +availability and whether Docker deployment is available. Discover what can be +read from the current environment instead of asking the user to supply NeMo +specific details they may not know. Explain any missing prerequisite in terms +of the capability it blocks. + +Check `nemo plugins list -f json` and `nemo agents --help`. Then inspect the +active Python environment for `nemo_agents_plugin`, +`nemo_fabric_adapters.deepagents` and `deepagents`. Do not infer that a harness +is available from config acceptance alone. + +If the NeMo Agents plugin and Deep Agents harness are already available, reuse +them. Do not reinstall or change their versions. If the plugin is absent, +explain that agent management is optional in NeMo Platform and that this build +requires it. Show the appropriate install command and ask for approval before running it: -```bash -.venv/bin/nemo agents delete "$AGENT_NAME" --yes -``` - -Verify this command reports that the Agent is absent before running the create -and deploy commands in Step 1: - -```bash -.venv/bin/nemo agents get "$AGENT_NAME" -``` - -## Prepare the selected config - -### Default: Platform-managed `agent.yaml` - -For a new build, invoke `nemo-agent-config` and create: +- Published Platform install: `uv pip install "nemo-platform[nemo-agents-plugin]"` +- NeMo Platform source checkout: `uv pip install -e plugins/nemo-agents/` -```txt -agents/-ethos/ - ETHOS.md - agent.yaml -``` - -Delegate authoring to `nemo-agent-config`. It selects the supported harness and -uses `nemo-model-selection` to verify the exact model against that harness's -model contract before writing the model block. Translate the approved Ethos into -system instructions, skills, MCP servers, tools, environment paths, and -telemetry. Keep every local path relative to the directory containing -`agent.yaml`. - -Before registration, inspect `skills.paths`: +The plugin owns selection of compatible Fabric adapter and harness versions. +Do not add a separate Deep Agents version constraint. After installation, +restart Platform services and repeat all four checks. If the plugin is present +but its Deep Agents adapter or runtime is absent, report a broken plugin +installation. Offer to reinstall the same Agents plugin only after approval. +Do not install the harness independently as an untracked repair. -- If it is empty, continue with the normal deployment lifecycle below. -- If it is non-empty, verify every relative directory is inside the agent - packaging context and contains `SKILL.md`. Package the complete bundle before - deployment: +## Choose supported artifacts - ```bash - IMAGE_TAG="${AGENT_NAME}:local" - .venv/bin/nemo agents package \ - --agent "agents/$AGENT_NAME-ethos/agent.yaml" \ - --tag "$IMAGE_TAG" - ``` +Read [references/fabric-deep-agents.md](references/fabric-deep-agents.md). +Select the smallest supported shape: - Use the packaged-image deploy command below. Do not use subprocess deployment - or the default container image because those paths materialize only - `agent.yaml` and do not stage relative skill directories. +- Put core behavior in `agent.yaml` instructions. +- Use an Agent Skill for a reusable instruction package. +- Use MCP for every executable custom tool. +- Add a declarative subagent only for a distinct delegated task. +- Put a safety or transactional sequence inside one deterministic MCP operation. -### Compatibility: existing NAT workflow YAML +Do not create `agent.py` as an agent entry point. Do not put Python callables or +compiled local graphs into Deep Agents settings. Stop when a requirement cannot +be expressed by the installed adapter contract. -If the user selected migration, preserve the original YAML. If a workflow, -tool, or custom Python component has no supported harness equivalent, keep the -NAT path or identify the need for a custom adapter. Never claim arbitrary NAT -workflows convert mechanically. +The current deployed Deep Agents path does not expose a verified end-to-end +resume contract for runtime human approval. If the Ethos requires an in-run +approve, edit or reject step, stop and report that adapter gap. Do not treat an +accepted `interrupt_on` setting as proof that the deployed workflow can resume. -Use `references/templates/agent.yml` only when the user explicitly chooses the -legacy NAT path or needs a new NAT compatibility workflow. +## Build the project -## Step 1: Register and deploy +Keep the deployable project under `agents/-ethos/`. Create only the +files required by the selected shape. For custom Python tools, use a `uv` +project with a locked dependency set, a typed MCP server and a console script. +Do not install dependencies globally. Do not add `deepagents` or a Fabric +adapter to the generated project unless its own code directly imports that API. -For the default path: - -For each operation retained by the selected lifecycle branch, follow the -`nemo-agent-config` confirmation requirement. Show the create command and ask -for explicit confirmation immediately before running it: - -```bash -.venv/bin/nemo agents create \ - --name "$AGENT_NAME" \ - --agent-config "agents/$AGENT_NAME-ethos/agent.yaml" -``` +Give tools narrow schemas, bounded output, explicit permissions, capped retries +for transient failures and redacted errors. Never write credentials, customer +data or production traces into source, YAML, fixtures or logs. -After create succeeds, show the deploy command and ask for explicit -confirmation immediately before running it: +Use `nemo-agent-config` to author the canonical +`agents/-ethos/agent.yaml`, but do not run its `nemo agents create` +registration step. This build workflow owns registration after every +pre-registration gate has passed. Require: -```bash -.venv/bin/nemo agents deploy \ - --agent "$AGENT_NAME" \ - --name "$DEPLOYMENT_NAME" +```yaml +config_format: nemo-agents-spec-v1 +default_harness: deepagents +harnesses: + deepagents: + kind: deepagents ``` -If `skills.paths` is non-empty, show this command instead and ask for explicit -confirmation immediately before running it: - -```bash -.venv/bin/nemo agents deploy \ - --agent "$AGENT_NAME" \ - --name "$DEPLOYMENT_NAME" \ - --mode docker \ - --image "$IMAGE_TAG" -``` - -For Kubernetes, publish the packaged image and replace `docker` and -`$IMAGE_TAG` with `k8s` and the published image tag. - -These commands assume the Agent and deployment are absent. If pre-flight found -existing resources, complete the selected lifecycle branch before running them. - -`nemo agents deploy` waits for `running` by default. If the user passed -`--no-wait`, wait explicitly: - -```bash -.venv/bin/nemo agents deployments wait "$DEPLOYMENT_NAME" -``` - -Show `agent.yaml` and the deployment result. Stop and ask whether the config, -model, harness, and instructions look right before continuing. - -For an unchanged NAT workflow, registration defaults configs without -`config_format` to `nat-workflow-v1`: - -Apply the same immediate confirmation requirement. Show the create command and -wait for explicit confirmation before running it: - -```bash -.venv/bin/nemo agents create \ - --name "$AGENT_NAME" \ - --agent-config "$NAT_WORKFLOW_PATH" -``` - -After create succeeds, show the deploy command and wait for explicit -confirmation before running it: - -```bash -.venv/bin/nemo agents deploy \ - --agent "$AGENT_NAME" \ - --name "$DEPLOYMENT_NAME" -``` - -## Step 2: Try the deployed agent - -For the default path, invoke one question from each category in the Ethos -`Scope` section (`Categories`). Do not invent a separate category heading. For -an unchanged NAT-only run without an Ethos, use representative questions from the -workflow and the user's stated requirements: - -```bash -.venv/bin/nemo agents invoke \ - --agent-deployment "$DEPLOYMENT_NAME" \ - --input "" -.venv/bin/nemo agents invoke \ - --agent-deployment "$DEPLOYMENT_NAME" \ - --input "" -.venv/bin/nemo agents invoke \ - --agent-deployment "$DEPLOYMENT_NAME" \ - --input "" -``` - -Display each response verbatim. Stop and ask whether to adjust the agent or -continue to evaluation. - -Before Step 3, branch explicitly: - -1. For an unchanged NAT-only run without `ETHOS.md`, stop after the smoke - test. Do not execute Steps 3–5 and do not require an evaluation fileset. -2. Continue into the Ethos-driven purpose selection and Data Designer flow only - when the user requests it and `agents/$AGENT_NAME-ethos/ETHOS.md` exists. - If the user requests evaluation but the Ethos is absent, create and confirm - the Ethos first; do not continue to Step 3 yet. - -## Step 3: Generate synthetic data - -Use Data Designer for every synthetic dataset. Do not hand-author evaluation, -knowledge-base, benchmark, persona, or training data. - -1. Always select evaluation as a required data purpose. Read - `agents/$AGENT_NAME-ethos/ETHOS.md` and list any additional plausible - purposes: knowledge/RAG corpus, benchmark, personas/adversarial inputs, - training, or another user-requested purpose. -2. Wait for the user to choose any additional purposes. Evaluation cannot be - omitted. If they delegate the decision, add a knowledge base when the Ethos - requires retrieval and adversarial personas when it contains safety - constraints. -3. Invoke `data-designer` once per selected purpose, passing the agent name, - purpose, and Ethos path. -4. Require every generated config to read product context from - `ETHOS.md`; do not duplicate that context inline. -5. Run each generated config. For evaluation, validate the generated records, - verify the resulting fileset exists, and record its exact dataset reference - as `EVAL_DATASET_REF`. -6. Show 3 to 5 sample records per purpose and ask for approval. - -A validated `$AGENT_NAME-eval-*` fileset and its exact `EVAL_DATASET_REF` must -exist before evaluation proceeds. - -## Step 3.5: Connect runtime data - -If the generated data must be available during invocation, connect it through -the selected harness's supported skills, MCP, or tool configuration. Update -`agents/$AGENT_NAME-ethos/agent.yaml` through `nemo-agent-config`, then follow the -confirmed replacement branch before creating and deploying the Agent again. - -Do not invent a generic retriever field. If the selected harness cannot consume -the required data, surface that limitation and choose a supported integration, -the NAT compatibility path, or a custom adapter. - -For a legacy NAT workflow, NAT-specific retrievers may be wired into its -`functions` and `workflow` blocks using the matching NAT RAG integration. - -After the replacement deployment, invoke a question that requires the data and -verify the expected tool or retrieval path was actually used. - -## Step 4: Evaluate - -Select the actual Platform model reference as `EVAL_MODEL`. Create -`agents/$AGENT_NAME.eval-job.json` from `references/templates/eval-job.json` and -replace every placeholder. Its `model` must equal `EVAL_MODEL`, and its -`dataset` must equal the recorded `EVAL_DATASET_REF` from Step 3. - -Validate the rendered file before creating the benchmark job: - -```bash -.venv/bin/python -m json.tool "agents/$AGENT_NAME.eval-job.json" >/dev/null -if grep -Eq '<[^>]+>' "agents/$AGENT_NAME.eval-job.json"; then - echo "eval job still contains template placeholders" >&2 - exit 1 -fi -``` - -Also read the validated payload back and confirm its `model` and `dataset` -values match `EVAL_MODEL` and `EVAL_DATASET_REF`. Do not invoke -`benchmark-jobs create` if JSON validation, model validation, dataset -validation, or fileset validation fails. - -After all validation succeeds: - -```bash -.venv/bin/nemo evaluation benchmarks list -.venv/bin/nemo evaluation benchmark-jobs create "$AGENT_NAME-eval" \ - --input-file "agents/$AGENT_NAME.eval-job.json" -``` - -Poll until the job reaches `completed` or `failed`, then download aggregate -scores. Show the score table and compare it with the success bar in -`ETHOS.md`. - -```bash -for i in $(seq 1 24); do - status=$(.venv/bin/nemo evaluation benchmark-jobs get-status "$AGENT_NAME-eval" 2>/dev/null) - echo "$status" - echo "$status" | grep -qE "completed|failed" && break - sleep 10 -done - -.venv/bin/nemo evaluation benchmark-jobs results aggregate-scores download \ - "$AGENT_NAME-eval" -``` +Use the model verified by `nemo-model-selection`. Validate every adapter setting +against the installed descriptor. Keep referenced paths relative to +`agent.yaml`. -## Step 5: Guardrails (optional) - -For `nemo-agents-spec-v1`, `AgentConfig` has no guardrail field and the current -skills do not define a supported composition between an Agent and an IGW -guardrailed VirtualModel. Do not add guardrail fields to `agent.yaml` or claim -that guardrails are attached. If the Ethos requires guardrails, report this as an -unmet requirement and stop before sign-off. `nemo-guardrails` may be used to -configure IGW VirtualModel middleware as a separate workflow, but do not treat -it as integrated with the Agent until its model routing has been explicitly -configured and validated. - -For a legacy NAT workflow, keep the NAT compatibility behavior: add supported -guardrail `intercepts` to the NAT workflow YAML, then follow the confirmed -replacement branch before creating and deploying it again. Never add NAT -`intercepts` to a `nemo-agents-spec-v1` config. - -For the NAT path, test one adversarial prompt and one legitimate prompt. Report -both responses and do not continue to sign-off until the expected policy is -enforced without blocking the legitimate request. - -## Step 6: Sign off - -Invoke the success-criteria prompt from the Ethos against -`$DEPLOYMENT_NAME`. Print the verbatim response as the formal sign-off. Do not -claim success until the deployment is `running`, evaluation has completed, and -the sign-off returns an actual model response. - -## If verification fails - -| Symptom | Cause | Recovery | -|---|---|---| -| Agents plugin unavailable | `plugins/nemo-agents` is not installed | Route to `nemo-setup` | -| Config validation fails | Config does not match its declared format | Use `nemo-agent-config` for `nemo-agents-spec-v1`; use NAT schema rules only for NAT YAML | -| Deployment reaches `failed` | Runtime, adapter, image, or config startup failure | Run `.venv/bin/nemo agents deployments get "$DEPLOYMENT_NAME"` and `.venv/bin/nemo agents logs "$DEPLOYMENT_NAME"` | -| Referenced file is missing | Path is outside or absent from the staged agent directory | Keep paths relative to the config and package the complete agent bundle into the deployed image | -| Adapter or binary is missing | Selected harness dependency is not installed | Install the matching adapter/runtime package or select an available harness | -| Empty response | Runtime invocation failed or the selected configuration is incomplete | Inspect deployment logs and the returned structured error | -| Eval job fails | Dataset reference or model ID is invalid | Get the benchmark job details and correct the named input | - -## Hard rules - -- Default new builds to `agents/$AGENT_NAME-ethos/agent.yaml` with - `config_format: nemo-agents-spec-v1`. -- Keep `ETHOS.md` as the human-readable design and `agent.yaml` as the - machine-readable implementation config. -- Preserve legacy NAT YAML unless the user explicitly requests migration. -- Do not mix NAT-only keys such as `functions`, `llms`, `workflow`, or - `intercepts` into `nemo-agents-spec-v1`. -- Do not put Platform `agent.yaml` fields into NAT workflow YAML. -- Use a named deployment and invoke it with `--agent-deployment`. -- Keep local artifact paths relative to the config directory. -- After changing persisted Agent config, use the confirmed replacement branch - before creating and deploying it again. +## Test before registration + +Read [references/testing-and-signoff.md](references/testing-and-signoff.md). +Derive one acceptance case file from the approved Ethos and reuse it for local +tests, deployed invocation and evaluation. + +Require unit tests, MCP contract tests, behavioral tests and trajectory tests +where tool choice, approval or order matters. Keep live model and integration +tests separate. If credentials or network access are absent, record the exact +live test as skipped. A skipped test is not evidence that the integration works. + +Do not run production side effects as representative tests. Use mocks, a sandbox +or a test tenant. Obtain explicit approval for any live action that can mutate a +business system. + +Stop before registration when a required test, Fabric translation, plan, +diagnostic or delivery reachability check fails. + +## Package custom code + +If the agent includes a local MCP server or another Python package, read +[references/packaging.md](references/packaging.md). Inspect the entire build +context for secrets and sensitive data, then package in project mode. Do not use +`--skip-validation`. + +Use Docker deployment for packaged custom code. A subprocess deployment is only +valid when every referenced executable is already installed on the Platform +service `PATH`. Do not assume the generated project's virtual environment is +visible to that service. + +## Register and deploy + +Check for an existing Agent and deployment with the requested names. If either +exists, offer reuse, rename or replace. Never overwrite or delete it without +explicit approval immediately before the state change. + +Show the exact create and deploy commands and ask for approval immediately +before running them. `nemo agents create` is both the final Fabric plan and +doctor gate and the registration step; do not run it earlier solely to validate +the config. Pass the confirmed workspace to both commands. If an +AgentEnvironment was selected, pass its workspace-qualified reference to +deployment. Verify registration by reading the Agent back in the same +workspace. Verify deployment through the blocking command result and deployment +status. On failure, inspect status and logs once, report the root error and +stop. + +## Verify onboarding + +Invoke the named deployment with safe acceptance cases. Require non-empty +responses, expected structured output and the required MCP tool calls. Exercise +one denied action and one upstream failure without causing a production side +effect. + +Confirm Fabric telemetry reached the configured destination. When Intake is +enabled, verify at least one trace with the expected agent, model and tool spans. +Treat missing telemetry or unreachable tools as incomplete onboarding even when +the final answer looks correct. + +Run `nemo-evaluator` only after invocation passes. Use the approved Ethos cases +and thresholds. Report passed, failed and skipped checks separately. + +## Gotchas + +- Fabric constructs the Deep Agent from `agent.yaml`; it never imports customer + `agent.py`. +- Keep `ETHOS.md`, `agent.yaml` and packaged artifacts together under + `agents/-ethos/` so registration uploads one canonical bundle. +- Custom Python code is deployable only when a declared MCP server or another + supported runtime surface can reach it. +- Prompt instructions and subagent delegation do not guarantee fixed ordering. +- Packaging copies the selected build context. Untracked secrets can enter an + image even when they are absent from committed files. +- The optional NeMo Agents plugin supplies the selected harness adapter and its + runtime dependencies. Reuse an installed harness and let the plugin resolve + compatible versions. +- Docker is the supported local container path. Treat Kubernetes as a separate + environment contract that must be verified. + +## Stop conditions + +Stop without registration or deployment when the Ethos is unapproved, the +adapter cannot express a requirement, a required tool is unreachable, a local +gate fails, a secret is present in the build context or Fabric validation fails. + +Stop without production-candidate status when a live test is skipped, telemetry +is missing, a target integration is mocked or an Ethos threshold is unmet. + +Call the result `Built`, `Onboarded` or `Production candidate` only according to +the evidence levels in `references/testing-and-signoff.md`. diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/evals/evals.json b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/evals/evals.json new file mode 100644 index 00000000000..e904add4deb --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/evals/evals.json @@ -0,0 +1,132 @@ +[ + { + "id": "nemo-build-agent-001-ethos-approval", + "question": "Build a NeMo Platform helpdesk agent from these answers: employees use it for password resets and VPN access; account changes require approval; it must never expose credentials; representative cases are a successful reset, an unauthorized reset and an unavailable identity provider. Persist the design first. Do not implement until I approve it.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent explains the supported Fabric Deep Agents path, creates one canonical agents/helpdesk-ethos/ETHOS.md through nemo-ethos, shows it for review and creates no implementation artifacts before explicit approval.", + "expected_behavior": [ + "The agent reads nemo-build-agent before acting", + "The agent explains Ethos as a plain-language design contract and does not assume the user knows NeMo Platform terminology", + "The agent explains the supported LangChain Deep Agents and Fabric path and asks whether to continue", + "The agent records representative successful and unsuccessful cases in the Ethos", + "The agent keeps the Ethos under agents/helpdesk-ethos/ETHOS.md", + "The agent stops for explicit Ethos approval before creating agent.yaml, Python source or deployment resources", + "The agent does not create a NAT workflow" + ] + }, + { + "id": "nemo-build-agent-002-mcp-reachability", + "question": "The approved Ethos is at agents/refund-ethos/ETHOS.md. Build the refund agent with custom Python behavior that verifies identity before issuing a refund. Use fake services for tests. Create and validate the local project, but do not register or deploy it.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The generated agent is config centric. It uses explicit nemo-agents-spec-v1 with the deepagents harness, exposes executable refund behavior through a declared MCP server and does not create an unreferenced agent.py. The identity and refund invariant is implemented inside a deterministic operation and tested on happy and unhappy paths.", + "expected_behavior": [ + "The project is rooted at agents/refund-ethos and keeps ETHOS.md and agent.yaml together", + "agent.yaml declares config_format nemo-agents-spec-v1 and a configured deepagents harness", + "The agent verifies the optional NeMo Agents plugin and Deep Agents harness, reuses them when present and does not add a separate harness version constraint", + "Custom Python behavior is installed as an MCP console script and declared under mcp.servers", + "The implementation does not rely on Fabric importing src package agent.py", + "Tests prove an unverified or failed identity cannot issue a refund", + "No registration or deployment command runs" + ] + }, + { + "id": "nemo-build-agent-003-missing-credential", + "question": "The support agent Ethos is approved. Build its mock based tests now. The production CRM credential is unavailable, so record the live test as skipped and do not register or deploy.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent builds and tests the local Fabric compatible project with a mocked CRM, records the exact live test prerequisite as skipped, performs no remote mutation and does not describe the result as production ready.", + "expected_behavior": [ + "The agent keeps credentials out of source, YAML, fixtures and logs", + "The agent tests CRM errors, timeouts and malformed responses with a mock", + "The live test is reported as skipped because the named credential is absent", + "The agent does not register or deploy", + "The final status is Built, not Production candidate" + ] + }, + { + "id": "nemo-build-agent-004-fabric-failure", + "question": "Build the approved agent and onboard it, but the supplied Deep Agents adapter setting is not supported by the installed Fabric descriptor. Stop safely if validation fails.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent checks the installed adapter contract, reports the unsupported setting and stops before registration or deployment. It does not invent a NAT wrapper or bypass validation.", + "expected_behavior": [ + "The agent validates settings against the installed Fabric adapter contract", + "The agent reports the specific unsupported setting", + "The agent does not use skip validation", + "The agent performs no registration or deployment after the failed gate", + "The agent does not create a NAT fallback" + ] + }, + { + "id": "nemo-build-agent-005-existing-resource", + "question": "Build and deploy the approved claims agent. A Platform Agent and deployment with the requested names already exist. Preserve them unless I explicitly approve replacement.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent inspects the existing resources, presents reuse, rename and replace choices and makes no destructive or replacement change without immediate explicit approval.", + "expected_behavior": [ + "The agent detects both name collisions before create or deploy", + "The agent does not overwrite local or remote resources", + "The agent presents reuse, rename and replace choices", + "The agent requires explicit approval immediately before any replacement", + "The agent verifies the result of each approved state change" + ] + }, + { + "id": "nemo-build-agent-006-runtime-approval-gap", + "question": "The approved claims agent Ethos requires a human to approve or reject each payment during the deployed agent run. Build and deploy it using Deep Agents interrupt_on.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent distinguishes accepted adapter configuration from a working runtime contract, reports that the deployed Deep Agents path has no verified approve, edit or reject resume flow and stops before producing or deploying a misleading implementation.", + "expected_behavior": [ + "The agent inspects the installed adapter and deployment invocation contract", + "The agent does not treat interrupt_on configuration acceptance as end to end support", + "The agent reports the missing deployed resume path", + "The agent stops before registration or deployment", + "The agent does not replace the approval with prompt instructions or a NAT workflow" + ] + }, + { + "id": "nemo-build-agent-007-plugin-install-declined", + "question": "The Ethos is approved, but this NeMo Platform environment does not have the optional Agents plugin. Continue without installing anything.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent explains that Fabric agent management is optional but required for this build, asks before installation and stops when installation is declined. It does not install Deep Agents independently or fall back to NAT.", + "expected_behavior": [ + "The agent checks both the Agents plugin and Deep Agents harness instead of assuming they are present", + "The agent explains what capability the optional plugin provides", + "The agent performs no installation after the user declines", + "The agent does not install Deep Agents or its Fabric adapter independently", + "The agent does not create a NAT fallback" + ] + }, + { + "id": "nemo-build-agent-008-existing-harness", + "question": "The approved Ethos is ready. The NeMo Agents plugin, Fabric Deep Agents adapter and Deep Agents runtime are already installed in the Platform environment. Build without changing installed package versions.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent verifies the installed plugin, adapter and runtime, reuses them and does not reinstall or add a separate harness version constraint to the generated project.", + "expected_behavior": [ + "The agent verifies the plugin through the NeMo CLI and checks the active Python environment", + "The agent reuses the installed harness", + "The agent does not reinstall the Agents plugin", + "The agent does not change installed harness versions", + "The generated project does not add Deep Agents or its Fabric adapter unless project code directly imports that API" + ] + }, + { + "id": "nemo-build-agent-009-broken-plugin-install", + "question": "The approved Ethos is ready. The NeMo Agents plugin appears in the plugin list, but its Fabric Deep Agents adapter is missing from the active Python environment.", + "expected_skill": "nemo-build-agent", + "expected_script": null, + "ground_truth": "The agent reports a broken Agents plugin installation and asks before reinstalling that plugin. It does not repair the environment by installing an independently pinned harness package.", + "expected_behavior": [ + "The agent reports that plugin presence alone does not prove harness availability", + "The agent stops before implementation while the required adapter is missing", + "The agent offers to reinstall the same Agents plugin only after approval", + "The agent does not install or pin Deep Agents independently", + "The agent repeats plugin and Python environment checks after any approved reinstall" + ] + } +] diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/fabric-deep-agents.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/fabric-deep-agents.md new file mode 100644 index 00000000000..70cc51725ba --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/fabric-deep-agents.md @@ -0,0 +1,81 @@ +# Fabric Deep Agents contract + +Use this reference when choosing the agent's implementation shape. + +## Ownership boundary + +NeMo Platform stores `agent.yaml` and translates it into a Fabric config. The +Fabric Deep Agents adapter constructs `create_deep_agent` from that normalized +config. It does not import a customer `agent.py` or accept Python callables in +`harnesses..settings`. + +Use only these supported surfaces: + +| Requirement | Delivery surface | +|---|---| +| System behavior and model | `instructions` and `models` in `agent.yaml` | +| Reusable instruction package | Agent Skill referenced by `skills.paths` | +| Executable custom tool | MCP server declared by `mcp.servers` | +| Specialized delegate | Declarative subagent under `harnesses.deepagents.settings.deepagents.subagents` | +| Remote background delegate | Supported asynchronous subagent configuration | + +Never create `src//agent.py` as an entry point for this path. Fabric +will not load it. + +## Custom tools + +Expose local Python tools as an MCP server. The generated project should install +a console script: + +```toml +[project.scripts] +support-tools = "support_agent.server:main" +``` + +Declare that executable in `agent.yaml`: + +```yaml +mcp: + servers: + support-tools: + transport: stdio + url: support-tools +``` + +Keep credentials out of the config and source tree. Declare environment +variable names and resolve secrets through the selected NeMo environment. + +For local testing, run the MCP server and its tools from the generated `uv` +project. For deployment, package the project as an image so the console script +is installed on the runtime `PATH`. Do not assume a nested project virtual +environment is visible to a Platform subprocess deployment. + +## Ordering and approvals + +Prompt instructions and subagent delegation do not guarantee execution order. +When several steps form one safety or transactional invariant, expose one MCP +operation that implements the state machine and tests the complete transition. +For example, `issue_verified_refund` should verify identity and issue the refund +inside one controlled operation instead of exposing two freely ordered tools. + +Do not use `interrupt_on` for a deployed workflow until the selected Platform +runtime exposes and tests an end-to-end pause and resume contract. The current +adapter accepts the setting, but deployed invocation does not provide a verified +approve, edit or reject resume path. Configuration acceptance is not runtime +support. + +A local compiled LangGraph subagent cannot be serialized through the current +Platform config. If the required behavior cannot be represented by a +deterministic MCP operation, a supported declarative subagent or a supported +remote agent, stop and report the adapter gap. + +## Version boundary + +Read the installed Fabric Deep Agents adapter descriptor and the NeMo Agents +config model before emitting adapter settings. The optional NeMo Agents plugin +supplies the adapter and its Deep Agents runtime dependency. Reuse the installed +versions and let the plugin dependency metadata select compatible versions. Do +not add a separate harness constraint to the generated customer project unless +its own code directly imports that API. + + diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/packaging.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/packaging.md new file mode 100644 index 00000000000..96279b4345a --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/packaging.md @@ -0,0 +1,114 @@ +# Package and deploy custom agent code + +Use this reference when the agent contains a local MCP server or another Python +package that must run with the deployed agent. + +## Project layout + +Keep the complete deployable package under the canonical Ethos directory: + +```text +agents/-ethos/ + ETHOS.md + agent.yaml + pyproject.toml + uv.lock + src//server.py + skills//SKILL.md + tests/ + evals/cases.yaml +``` + +Omit directories the implementation does not need. Every `skills.paths` value +must be relative to `agent.yaml`, remain inside this project and point to a +directory containing `SKILL.md`. + +## Inspect the build context + +Before packaging: + +1. Inspect `.gitignore` and `.dockerignore`. +2. Search the context for `.env` files, private keys, tokens, credential files, + local databases, traces and customer data. +3. Remove or exclude anything that is not required at runtime. +4. Confirm `pyproject.toml` defines every MCP console script referenced by + `agent.yaml`. +5. Confirm `uv.lock` is current and local tests pass with `uv run`. + +Stop if the context contains an unresolved secret or sensitive dataset. + +## Validate and build + +Use project mode so the Python package and console scripts are installed into +the image: + +```bash +IMAGE_TAG="$AGENT_NAME:local" +.venv/bin/nemo agents package \ + --agent "agents/$AGENT_NAME-ethos/agent.yaml" \ + --pyproject "agents/$AGENT_NAME-ethos/pyproject.toml" \ + --tag "$IMAGE_TAG" +``` + +Do not use `--skip-validation`. For a source checkout, build the current +Platform wheel and point packaging at it: + +```bash +uv build --package nemo-platform --wheel --out-dir dist +NEMO_AGENTS_WHEEL=LATEST .venv/bin/nemo agents package \ + --agent "agents/$AGENT_NAME-ethos/agent.yaml" \ + --pyproject "agents/$AGENT_NAME-ethos/pyproject.toml" \ + --tag "$IMAGE_TAG" +``` + +`NEMO_AGENTS_WHEEL=LATEST` is the supported source checkout path. Do not treat +the checkout's `0.0.0` project metadata as the released Platform version and do +not silently weaken validation. + +## Deploy the packaged image + +After the user approves registration and deployment, use the workspace confirmed +earlier: + +```bash +WORKSPACE="" + +.venv/bin/nemo agents create \ + --name "$AGENT_NAME" \ + --agent-config "agents/$AGENT_NAME-ethos/agent.yaml" \ + --workspace "$WORKSPACE" +``` + +If an AgentEnvironment was selected, deploy with its workspace-qualified +reference: + +```bash +AGENT_ENVIRONMENT="$WORKSPACE/" + +.venv/bin/nemo agents deploy \ + --agent "$AGENT_NAME" \ + --name "$AGENT_NAME-deployment" \ + --mode docker \ + --image "$IMAGE_TAG" \ + --workspace "$WORKSPACE" \ + --environment "$AGENT_ENVIRONMENT" \ + --timeout 300 +``` + +If no AgentEnvironment was selected, omit both the assignment and option: + +```bash +.venv/bin/nemo agents deploy \ + --agent "$AGENT_NAME" \ + --name "$AGENT_NAME-deployment" \ + --mode docker \ + --image "$IMAGE_TAG" \ + --workspace "$WORKSPACE" \ + --timeout 300 +``` + +Docker is the supported local container path. Kubernetes requires a published +image reachable by the configured executor and a separately verified runtime +contract. + + diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/agent.yml b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/agent.yml deleted file mode 100644 index 6a2acccda20..00000000000 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/agent.yml +++ /dev/null @@ -1,51 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# NAT workflow YAML for a NeMo Platform agent. -# Required top-level keys: functions, llms, workflow. -# Optional: intercepts, middleware. -# Anything else at top level causes NAT to reject the config. - -functions: - clock: - _type: current_datetime - -llms: - llm: - _type: openai - api_key: not-used - model_name: nvidia-llama-3-3-nemotron-super-49b-v1 - temperature: 0.0 - max_tokens: 1024 - -general: - telemetry: - tracing: - nemo_files_trace: - _type: nemo_files - # workspace and agent_name are injected at deploy time - -workflow: - _type: react_agent - tool_names: [clock] - llm_name: llm - verbose: false - parse_agent_response_max_retries: 3 - system_prompt: | - - - Your job: . - - You handle these categories: - - - - - - - - Constraints: - - - - - - You have access to the following tools: - {tools} - - Tool names: {tool_names} diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/data-designer-config.py b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/data-designer-config.py deleted file mode 100644 index d33801b6b05..00000000000 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/data-designer-config.py +++ /dev/null @@ -1,80 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Data Designer config builder for synthetic eval data. - -Run via the venv Python until `nemo data-designer preview-local` is available -in the installed CLI version (currently 2.1.0 ships only `data-designer jobs`): - - .venv/bin/python agents/.dd.py - -When `preview-local` lands, swap the bottom of this file for: - - # nemo data-designer preview-local agents/.dd.py --num-records 10 - -Substitute: - AGENT_DESCRIPTION one-sentence agent role from the Ethos - CATEGORIES the Ethos category list - MODEL API-Catalog format with slashes (NOT entity-name format) - PROVIDER workspace/provider-name as registered in nemo -""" - -import data_designer.config as dd -from nemo_platform import NeMoPlatform - -AGENT_DESCRIPTION = "" -CATEGORIES: list[str | int | float] = ["", "", ""] -MODEL = "meta/llama-3.3-70b-instruct" -PROVIDER = "default/nim-llm" - - -def load_config_builder() -> dd.DataDesignerConfigBuilder: - builder = dd.DataDesignerConfigBuilder( - model_configs=[ - dd.ModelConfig( - alias="generator", - model=MODEL, - provider=PROVIDER, - skip_health_check=True, - ) - ] - ) - - builder.add_column( - dd.SamplerColumnConfig( - name="category", - sampler_type=dd.SamplerType.CATEGORY, - params=dd.CategorySamplerParams(values=CATEGORIES), - ) - ) - - builder.add_column( - dd.LLMTextColumnConfig( - name="user_question", - model_alias="generator", - prompt=( - "Generate a realistic user question about {{ category }} for an agent that " - f"{AGENT_DESCRIPTION}. The question should sound like a real employee asking for help." - ), - ) - ) - - builder.add_column( - dd.LLMTextColumnConfig( - name="expected_response", - model_alias="generator", - prompt=( - "Write the ideal agent response to this question: {{ user_question }}. " - "The response should be helpful, accurate, and professional." - ), - ) - ) - - return builder - - -if __name__ == "__main__": - client = NeMoPlatform(base_url="http://localhost:8080", workspace="default") - builder = load_config_builder() - results = client.data_designer.preview(builder, num_records=10) - results.display_sample_record() diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/eval-job.json b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/eval-job.json deleted file mode 100644 index 7c79eda6082..00000000000 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/templates/eval-job.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "_comment": "Substitute , , and the dataset reference. The dataset is the artifact path produced by `nemo data-designer preview-local --save-results`. Workspace prefix `default/` is the platform default.", - "name": "-eval", - "benchmark": "default/agent-quality", - "model": "default/nim-llm/", - "dataset": "default/agents/.eval" -} diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/testing-and-signoff.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/testing-and-signoff.md new file mode 100644 index 00000000000..6cdd94f9c43 --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/references/testing-and-signoff.md @@ -0,0 +1,66 @@ +# Testing and signoff + +Use this reference after the Ethos is approved and before registration. + +## Derive one acceptance contract + +Turn the approved Ethos examples and success criteria into one versioned test +case file in the generated project. Reuse those cases for local behavioral +tests, deployed invocations and evaluation. Do not create separate definitions +that can drift. + +Each case should identify: + +- input and expected outcome; +- allowed and forbidden side effects; +- required or forbidden tool calls; +- required ordering when observable; +- whether live credentials are required; +- the pass rule. + +## Required local gates + +1. Unit test every MCP operation, including invalid input, denied actions, + upstream errors, timeouts and redacted error messages. +2. Contract test MCP schemas, tool discovery and structured outputs. +3. Run the Ethos cases against the assembled local agent shape. +4. Record trajectories when tool choice, approval or ordering is part of the + result. +5. Run a separate live smoke test only when the required credentials and + network access are available. + +Mock business systems for ordinary tests. Never run representative production +side effects as evaluation. For a live side effect, use an approved sandbox or +test tenant and obtain explicit approval for the specific operation. + +## Delivery reachability + +A source file or passing import test does not prove that Fabric can call a +tool. Before registration, verify every required executable capability is +reachable from `agent.yaml` through `mcp.servers`, and every instruction package +is reachable through `skills.paths`. + +After image deployment: + +1. Invoke a case that requires each MCP server. +2. Confirm the expected tool appears in the trajectory and returns the expected + schema. +3. Exercise one denied action and one upstream failure. +4. Confirm telemetry contains the invocation, tool and error or approval spans + expected by the Ethos. + +## Status language + +Use these states consistently: + +| State | Minimum evidence | +|---|---| +| Built | Approved Ethos, generated artifacts and required local tests pass | +| Onboarded | Fabric validation, registration, deployment and telemetry checks pass | +| Production candidate | Live representative checks pass in the target environment and every required Ethos threshold is met | + +A skipped live test, missing telemetry or untested production integration +prevents `Production candidate` status. Report passed, failed and skipped checks +separately. + + diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/tests.json b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/tests.json index ee9b151f6c6..eb551cb9362 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/tests.json +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-build-agent/tests.json @@ -1,75 +1,21 @@ { "skill": "nemo-build-agent", "tests": [ - { - "type": "explicit", - "prompt": "Use nemo-build-agent to scaffold and deploy an IT helpdesk agent.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "explicit", - "prompt": "Use nemo-build-agent to implement and deploy the approved support-triage Ethos.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "explicit", - "prompt": "Invoke nemo-build-agent for the customer-faq agent we designed.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "implicit", - "prompt": "The IT helpdesk agent Ethos is approved. Build and deploy it for password resets, VPN issues, and software access requests.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "implicit", - "prompt": "Scaffold and deploy the support agent we just specced out on NeMo Platform.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "implicit", - "prompt": "Take the approved Ethos in agents/ and ship the agent to NeMo Platform.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "implicit", - "prompt": "Spec is signed off. Build the agent, deploy it, and run the eval pass.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "implicit", - "prompt": "Deploy my existing NAT workflow YAML as a NeMo Platform agent, then smoke test it.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "implicit", - "prompt": "I have an old NAT agent workflow and an approved Ethos. Build and deploy it while preserving the legacy config.", - "expected_skill": "nemo-build-agent" - }, - { - "type": "negative-control", - "prompt": "Write an agent.yaml with a Codex harness, but do not register or deploy anything.", - "expected_skill_not": "nemo-build-agent" - }, - { - "type": "negative-control", - "prompt": "Convert this old NAT workflow YAML to nemo-agents-spec-v1, but do not deploy it.", - "expected_skill_not": "nemo-build-agent" - }, - { - "type": "negative-control", - "prompt": "Let's design a system for routing customer queries across teams before any implementation.", - "expected_skill_not": "nemo-build-agent" - }, - { - "type": "negative-control", - "prompt": "Fix the CSS spacing in the React settings page.", - "expected_skill_not": "nemo-build-agent" - }, - { - "type": "negative-control", - "prompt": "Help me write a CrewAI agent from scratch in a fresh Python environment outside NeMo Platform.", - "expected_skill_not": "nemo-build-agent" - } + {"type":"explicit","prompt":"Use nemo-build-agent to build and onboard an IT helpdesk agent through Fabric.","expected_skill":"nemo-build-agent"}, + {"type":"explicit","prompt":"Invoke nemo-build-agent for the approved support-triage Ethos.","expected_skill":"nemo-build-agent"}, + {"type":"explicit","prompt":"Run nemo-build-agent to package, deploy, and verify the claims agent.","expected_skill":"nemo-build-agent"}, + {"type":"implicit","prompt":"The IT helpdesk Ethos is approved. Build its tools and tests, create the explicit Fabric config, and onboard it to NeMo Platform.","expected_skill":"nemo-build-agent"}, + {"type":"implicit","prompt":"I want to build a customer support agent, but I do not know NeMo Platform terminology. Walk me through the design, explain what you need me to approve, then implement and register it through the supported platform path.","expected_skill":"nemo-build-agent"}, + {"type":"implicit","prompt":"Turn this approved refund agent design into a tested Deep Agents deployment and prove the refund tool is reachable after packaging.","expected_skill":"nemo-build-agent"}, + {"type":"implicit","prompt":"The Ethos is approved. Build the agent and its mock based tests now. The production API credential is unavailable, so record the live test as skipped and do not register or deploy.","expected_skill":"nemo-build-agent"}, + {"type":"contextual","prompt":"We approved the support triage Ethos yesterday and the NeMo workspace is ready now. Implement the approved agent, test it and onboard it through Fabric.","expected_skill":"nemo-build-agent"}, + {"type":"contextual","prompt":"The prototype works locally, but its CRM tool never appears after deployment. Finish packaging the custom tool through the supported NeMo agent path and verify it in Fabric.","expected_skill":"nemo-build-agent"}, + {"type":"contextual","prompt":"Legal approved the refund agent design. Implement the approved agent, prove identity verification happens before refunds, and stop before deployment for my review.","expected_skill":"nemo-build-agent"}, + {"type":"negative-control","prompt":"Use nemo-explore for a design conversation about the audience, scope, constraints, and success criteria.","expected_skill_not":"nemo-build-agent"}, + {"type":"negative-control","prompt":"Write the approved agent design into ETHOS.md, but stop before creating code or agent.yaml.","expected_skill_not":"nemo-build-agent"}, + {"type":"negative-control","prompt":"Use nemo-agent-config to author and validate a nemo-agents-spec-v1 YAML with the Codex harness.","expected_skill_not":"nemo-build-agent"}, + {"type":"negative-control","prompt":"Invoke the already deployed claims agent with these five questions and show me the responses.","expected_skill_not":"nemo-build-agent"}, + {"type":"negative-control","prompt":"Keep this existing NAT workflow running and add another NAT function to it.","expected_skill_not":"nemo-build-agent"}, + {"type":"negative-control","prompt":"Help me write a CrewAI agent in a separate Python project outside NeMo Platform.","expected_skill_not":"nemo-build-agent"} ] } diff --git a/packages/nemo_platform_ext/tests/cli/commands/skills/test_skill_content.py b/packages/nemo_platform_ext/tests/cli/commands/skills/test_skill_content.py index 4fd22820def..cd9fad88048 100644 --- a/packages/nemo_platform_ext/tests/cli/commands/skills/test_skill_content.py +++ b/packages/nemo_platform_ext/tests/cli/commands/skills/test_skill_content.py @@ -120,13 +120,53 @@ def test_platform_skills_declare_known_preconditions(self): assert not unknown, f"{name} has unknown preconditions: {sorted(unknown)}" assert skills_with_preconditions - def test_build_agent_templates_are_packaged(self): + def test_build_agent_references_are_packaged(self): skill = load_skills()["nemo-build-agent"] assert skill.source_dir is not None + assert (skill.source_dir / "references" / "fabric-deep-agents.md").is_file() + assert (skill.source_dir / "references" / "testing-and-signoff.md").is_file() + assert (skill.source_dir / "references" / "packaging.md").is_file() - templates_dir = skill.source_dir / "references" / "templates" - assert (templates_dir / "agent.yml").is_file() - assert (templates_dir / "eval-job.json").is_file() + def test_build_agent_uses_agents_plugin_harness_dependency(self): + skill = load_skills()["nemo-build-agent"] + assert "optional NeMo Agents plugin supplies the selected harness adapter" in skill.content + assert "Do not add a separate Deep Agents version constraint" in skill.content + assert "If the NeMo Agents plugin and Deep Agents harness are already available" in skill.content + assert 'uv pip install "nemo-platform[nemo-agents-plugin]"' in skill.content + assert "uv pip install -e plugins/nemo-agents/" in skill.content + + def test_build_agent_covers_plugin_dependency_failure_paths(self): + skill = load_skills()["nemo-build-agent"] + assert "If the plugin is absent" in skill.content + assert "ask for approval before" in skill.content + assert "If the plugin is present" in skill.content + assert "its Deep Agents adapter or runtime is absent" in skill.content + assert "Do not install the harness independently" in skill.content + + def test_build_agent_defers_registration_until_after_local_gates(self): + content = load_skills()["nemo-build-agent"].content + normalized = " ".join(content.split()) + + assert "do not run its `nemo agents create` registration step" in normalized + assert "This build workflow owns registration after every pre-registration gate has passed" in normalized + assert "do not run it earlier solely to validate the config" in normalized + assert content.index("## Test before registration") < content.index("## Register and deploy") + + def test_build_agent_registration_uses_confirmed_workspace_and_environment(self): + skill = load_skills()["nemo-build-agent"] + assert skill.source_dir is not None + packaging = (skill.source_dir / "references" / "packaging.md").read_text() + create_command, deployments = packaging.split(".venv/bin/nemo agents deploy", maxsplit=1) + selected_environment, no_environment = deployments.split("If no AgentEnvironment was selected", maxsplit=1) + + assert packaging.count(".venv/bin/nemo agents deploy") == 2 + assert "If an AgentEnvironment was selected" in packaging + assert '--workspace "$WORKSPACE"' in create_command + assert '--workspace "$WORKSPACE"' in selected_environment + assert '--environment "$AGENT_ENVIRONMENT"' in selected_environment + assert "AGENT_ENVIRONMENT=" not in no_environment + assert "--environment" not in no_environment + assert '--workspace "$WORKSPACE"' in no_environment def test_model_selection_benchmark_cache_is_packaged(self): skill = load_skills()["nemo-model-selection"]