test(nemo-agents): add e2e for Fabric backed agents

[mmogallapalli]

Summary

Add deterministic Fabric/DeepAgents E2E deployment coverage alongside the existing NAT scenarios for subprocess, Docker, and Kubernetes. The tests exercise agent registration, backend-specific readiness, invocation through the standard agents gateway, deterministic mock-provider output, and the existing cleanup paths without installing dependencies dynamically.

Related Issue

Closes AIRCORE-960.

Changes

• Generalize the shared agent deployment E2E helper to support NAT and nemo-agents-spec-v1 configs across subprocess and container backends.
• Add NAT and Fabric/DeepAgents subprocess deployment tests using the standard subprocess E2E harness.
• Split the Docker scenario into explicit NAT and Fabric/DeepAgents cases while preserving its image, marker, endpoint, and cleanup conventions.
• Split the Kubernetes scenario into explicit NAT and Fabric/DeepAgents cases while preserving its Kind image, marker, timeout, and cleanup conventions.
• Use the Platform mock inference provider for deterministic responses without external credentials or model calls.
• Handle the nemo_platform_plugin client 404 raised for an absent Fabric spec fileset so inline agent.yaml staging falls back as intended.
• Add a regression unit test for the plugin-client missing-fileset response.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Documentation updated for user-visible behavior
• Documentation not applicable — justification: this adds internal E2E coverage and corrects an existing inline-config fallback; there is no user-facing API or workflow change.

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run -a passes, or any blocked checks are identified below
• Targeted tests pass, or tests are marked not applicable above
• No secrets, API keys, or credentials are included

Targeted validation:

• uv run --frozen pytest e2e/test_nemo_agents_subprocess.py -v --run-e2e --no-cov — 2 passed.
• Docker NAT case passed locally using the prebuilt nmp-local/nmp-api:e2e image and temporary Docker Desktop harness overrides; those local overrides are not part of this branch.
• NMP_E2E_IMAGE_REGISTRY=nmp-local NMP_E2E_IMAGE_TAG=e2e uv run --frozen pytest e2e/test_nemo_agents_docker.py::test_fabric_docker_agent_deploys_and_invokes_through_gateway -v --run-e2e --no-cov — 1 passed.
• NMP_BASE_URL=http://127.0.0.1:8080 NMP_E2E_IMAGE_REGISTRY=nmp-local NMP_E2E_IMAGE_TAG=e2e uv run --frozen pytest e2e/test_nemo_agents_k8s.py -v --run-e2e --no-cov against a local Kind deployment — 2 passed.
• .venv/bin/pytest plugins/nemo-agents/tests/unit/test_fabric_artifact_staging.py -v --no-cov — 9 passed.
• Ruff format and lint checks passed for the changed E2E and Fabric artifact-staging files.

Summary by CodeRabbit

• New Features

  • Added support for deploying and invoking NAT and Fabric agents through subprocess, Docker, and Kubernetes workflows.
  • Added configuration format selection for NAT and Fabric agents.
  • Improved endpoint handling across subprocess and container deployment modes.

• Bug Fixes

  • Improved Fabric artifact staging when files are unavailable from the plugin service.

• Tests

  • Added end-to-end coverage for NAT and Fabric agent deployment and gateway invocation across all supported deployment modes.

[coderabbitai]

📝 Walkthrough

Walkthrough

The shared deployment helper now supports subprocess, Docker, and Kubernetes deployments with NAT and Fabric configuration formats. It adds mode-specific endpoint validation and preserves the container wrapper. E2E coverage now includes both formats across all three deployment modes. Fabric staging handles plugin-client not-found errors.

Changes

Agent deployment and invocation

Layer / File(s)	Summary
Configuration format generation e2e/agents_deploy_helpers.py	The helper defines NAT and NeMo Agents specification formats. It generates the selected configuration and rejects unsupported formats.
Generalized deployment runner e2e/agents_deploy_helpers.py	run_agent_deploy_and_invoke supports subprocess, Docker, and Kubernetes modes. Endpoint checks vary by deployment mode. The container helper delegates to the generalized runner.
Subprocess deployment tests e2e/test_nemo_agents_subprocess.py	The tests configure the local subprocess harness and invoke NAT and Fabric agents through the gateway.
Docker deployment tests e2e/test_nemo_agents_docker.py	The tests separately deploy and invoke NAT and Fabric agents with their corresponding configuration formats.
Kubernetes deployment tests e2e/test_nemo_agents_k8s.py	The tests separately deploy and invoke NAT and Fabric agents in Kubernetes with their corresponding configuration formats.

Fabric artifact staging

Layer / File(s)	Summary
Fileset not-found fallback plugins/nemo-agents/src/nemo_agents_plugin/runner/fabric_artifact_staging.py, plugins/nemo-agents/tests/unit/test_fabric_artifact_staging.py	Fabric staging catches platform and plugin-client not-found errors. Unit coverage verifies fallback to inline agent.yaml.

Sequence Diagram(s)

sequenceDiagram
  participant E2ETest
  participant DeploymentHelper
  participant DeploymentMode
  participant Gateway
  participant AgentRuntime
  E2ETest->>DeploymentHelper: select NAT or Fabric configuration
  DeploymentHelper->>DeploymentMode: start subprocess, Docker, or Kubernetes deployment
  DeploymentMode-->>DeploymentHelper: return endpoint and process state
  DeploymentHelper->>Gateway: invoke deployed agent
  Gateway->>AgentRuntime: route agent request
  AgentRuntime-->>Gateway: return response
  Gateway-->>E2ETest: return invocation result

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#715: Introduced Docker agent deployment coverage extended by this PR.
• NVIDIA-NeMo/nemo-platform#838: Added Fabric configuration and deployment paths exercised by these tests.
• NVIDIA-NeMo/nemo-platform#1007: Updated Fabric artifact staging and fallback handling.

Suggested reviewers: tylersbray, mikeknep, mckornfield

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding end-to-end tests for Fabric-backed NeMo Agents.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch mmogallapall/aircore-960-add-e2e-tests-for-fabric-backed-agent-serving-across

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	31464/40086	78.5%	63.0%
Integration Tests	18326/38038	48.2%	20.8%