chore(platform-runner): stop registering legacy evaluator service by SandyChapman · Pull Request #228 · NVIDIA-NeMo/nemo-platform

SandyChapman · 2026-06-09T14:50:15Z

Summary

remove the legacy evaluator service from built-in platform runner registration
remove legacy evaluation from default API and OpenAPI service groups
add focused coverage that evaluator plugin services are still API-discovered while legacy evaluation is absent by default

Validation

uv run --frozen pytest packages/nmp_platform_runner/tests/test_registry.py packages/nmp_platform_runner/tests/test_server.py -q
uv run --frozen ruff check packages/nmp_platform_runner/src/nmp/platform_runner/registry.py packages/nmp_platform_runner/tests/test_registry.py packages/nmp_platform_runner/tests/test_server.py
uv run --frozen ruff format --check packages/nmp_platform_runner/src/nmp/platform_runner/registry.py packages/nmp_platform_runner/tests/test_registry.py packages/nmp_platform_runner/tests/test_server.py
uv run --frozen ty check packages/nmp_platform_runner/src/nmp/platform_runner/registry.py packages/nmp_platform_runner/tests/test_registry.py packages/nmp_platform_runner/tests/test_server.py

Summary by CodeRabbit

Breaking Changes
- The legacy "evaluation" service was removed from the platform registry and will no longer appear in runner, API, or OpenAPI service listings.
Documentation
- Evaluation API permissions and per-API scopes (evaluation:read/write) were removed; CLI scope examples and scope guidance were updated.
Tests
- Service registry tests updated to reflect the removal and verify current service groupings.
- E2E and unit test helpers for evaluation flows were removed.

coderabbitai · 2026-06-09T14:58:01Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Legacy evaluation service entry removed from the platform runner registry and OpenAPI lists. Tests add an EvaluatorService double and assert evaluation is absent while evaluator is present. Documentation removes evaluation scopes and the Evaluation API permissions section.

Changes

Evaluation Service Removal

Layer / File(s)	Summary
Remove evaluation service from registry `packages/nmp_platform_runner/src/nmp/platform_runner/registry.py`	`evaluation` is removed from `AVAILABLE_SERVICES`, `API_SERVICES`, and `OPENAPI_SERVICES`, deregistering it from discovery and OpenAPI filtering.
Update tests for evaluator inclusion and evaluation removal `packages/nmp_platform_runner/tests/test_registry.py`	Adds `EvaluatorService` test double, changes `AgentsDeploymentController.reconcile_one` param, adds tests asserting `evaluator` appears in `api`/`all` and `evaluation` is absent, and updates OpenAPI expectations.
Remove evaluation scopes and permissions from docs `docs/auth/authorization/api-scopes.md`, `docs/auth/authorization/permissions-reference.md`	Removes the Evaluation per-API scopes row, updates the CLI scopes example and "When to Restrict Scopes" bullets, and deletes the Evaluation API permissions section.

Sequence Diagram(s)

sequenceDiagram
  participant ServiceRegistry
  participant GroupBuilder
  participant OpenAPIGenerator
  ServiceRegistry->>GroupBuilder: build groups using API_SERVICES
  ServiceRegistry->>OpenAPIGenerator: filter by OPENAPI_SERVICES for OpenAPI generation

Possibly related PRs

chore(platform-seed): remove evaluator seed dependency #227: Related removal of evaluator/evaluation functionality (removes evaluator seeding and dependency).

Suggested Reviewers

arpitsardhana
tylersbray

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Title accurately describes the main change: removing the legacy evaluator service from platform runner registration, which is the primary purpose of the PR across all modified files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch remove-evaluator-runner-registration/schapman

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-09T15:03:22Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	18982/25048	75.8%	62.3%
Integration Tests	12008/23736	50.6%	25.8%

github-actions · 2026-06-09T16:19:31Z

Documentation preview is ready

Preview: https://nvidia-nemo.github.io/nemo-platform/pr-preview/pr-228/pr-228/

Built from 98b13a3 in workflow run.

This preview is deployed from this PR branch, updates when docs changes are pushed, and will be removed when the PR closes.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/auth/authorization/permissions-reference.md`:
- Around line 22-25: The Viewer/Editor/Admin columns for permissions like
evaluation.benchmarks.(read | list | create | delete), evaluation.jobs.(read |
list | create | delete | cancel), evaluation.live.exec, and
evaluation.metrics.(read | list | create | delete) are blank and unclear; add a
short legend/note under the permissions table clarifying that an empty cell
means “not assigned in predefined roles; requires explicit role binding or
another role type” (or equivalent phrasing), and reference those specific
permission patterns in the note so readers know the meaning applies to
evaluation.* entries.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dfc7e8a3-9cb2-4278-a12e-109c74951ebe

📥 Commits

Reviewing files that changed from the base of the PR and between e7f92b8 and 06ce64d.

⛔ Files ignored due to path filters (267)

sdk/python/nemo-platform/.nmpcontext/openapi.yaml is excluded by !sdk/**
sdk/python/nemo-platform/.nmpcontext/stainless.yaml is excluded by !sdk/**
sdk/python/nemo-platform/api.md is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/_client.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/_decoders/jsonl.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/api.md is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/benchmark_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmarks.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/evaluation.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/metric_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metrics.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/files/filesets.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/aggregate_range_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/aggregate_rubric_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/aggregated_metric_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_create_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_jobs_list_filter_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_jobs_page.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_jobs_sort_field.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_get_logs_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_result_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_result_retrieve_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_results_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/benchmark_evaluation_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/benchmark_metric_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/row_score_download_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_offline_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_offline_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_agent_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_agent_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_retrieve_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_retrieve_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmarks_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/built_in_dataset.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/dataset_rows.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/dataset_rows_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/evaluate_dataset_rows_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/extended_benchmark.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/field_mapping.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/field_mapping_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/fileset.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/fileset_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/fileset_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/histogram.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/histogram_bin.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/inference_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/inference_params_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/json_score_parser.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/json_score_parser_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_create_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluate_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_jobs_list_filter_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_jobs_page.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_jobs_sort_field.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_row_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_get_logs_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_result_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_result_retrieve_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_results_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_jobs/results/row_score_download_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_offline_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_offline_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_agent_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_agent_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_retrieve_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_retriever_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_retriever_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_type.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metrics_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/model.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/model_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/model_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/parameter.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/parameter_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/percentiles.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/range_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/range_score_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/reasoning_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/reasoning_params_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/regex_score_parser.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/regex_score_parser_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_score_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/retriever_pipeline.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/retriever_pipeline_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/row_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_score_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_score_stat.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online_model.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online_model_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_offline_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_offline_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_online_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_online_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_metadata_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_metadata_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_update_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/error_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/field_error.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/fileset_metadata.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/test_aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/test_artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/test_row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/test_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/test_aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/test_artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/test_row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/test_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_benchmark_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_benchmark_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_benchmarks.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_metric_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_metric_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_metrics.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/test_models.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/decoders/test_jsonl.py is excluded by !sdk/**
sdk/stainless.yaml is excluded by !sdk/**

📒 Files selected for processing (7)

docs/auth/authorization/permissions-reference.md
openapi/ga/individual/platform.openapi.yaml
openapi/ga/openapi.yaml
openapi/openapi.yaml
packages/nmp_platform_runner/src/nmp/platform_runner/registry.py
packages/nmp_platform_runner/tests/test_registry.py
services/core/auth/src/nmp/core/auth/assets/static-authz.yaml

💤 Files with no reviewable changes (1)

packages/nmp_platform_runner/src/nmp/platform_runner/registry.py

🚧 Files skipped from review as they are similar to previous changes (1)

packages/nmp_platform_runner/tests/test_registry.py

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/auth/authorization/api-scopes.md (1)
116-121: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Rename end section to Next Steps.

Guideline requires a Next Steps section at page end; this page uses Related. Rename the heading and keep these cross-links under it.

As per coding guidelines, "Include 'Next Steps' section at the end with cross-links to related documentation content."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/auth/authorization/api-scopes.md` around lines 116 - 121, Rename the
final section heading from "Related" to "Next Steps" and leave the existing
cross-links unchanged; locate the "Related" heading and update its heading text
to "Next Steps" in the docs/auth/authorization/api-scopes.md content (the list
of links to Roles & Permissions, Using Authentication, OIDC Setup, Security
Model) so the page ends with the required Next Steps section per guidelines.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/auth/authorization/api-scopes.md`:
- Around line 116-121: Rename the final section heading from "Related" to "Next
Steps" and leave the existing cross-links unchanged; locate the "Related"
heading and update its heading text to "Next Steps" in the
docs/auth/authorization/api-scopes.md content (the list of links to Roles &
Permissions, Using Authentication, OIDC Setup, Security Model) so the page ends
with the required Next Steps section per guidelines.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5941cae7-1408-4875-b253-b2c9d6f33485

📥 Commits

Reviewing files that changed from the base of the PR and between 06ce64d and 28ddc42.

⛔ Files ignored due to path filters (267)

sdk/python/nemo-platform/.nmpcontext/openapi.yaml is excluded by !sdk/**
sdk/python/nemo-platform/.nmpcontext/stainless.yaml is excluded by !sdk/**
sdk/python/nemo-platform/api.md is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/_client.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/_decoders/jsonl.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/api.md is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/benchmark_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmark_jobs/results/row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/benchmarks.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/evaluation.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/metric_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/results.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metric_jobs/results/row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/evaluation/metrics.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/resources/files/filesets.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_goal_accuracy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/agent_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/aggregate_range_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/aggregate_rubric_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/aggregated_metric_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/answer_accuracy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_create_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_jobs_list_filter_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_jobs_page.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_evaluation_jobs_sort_field.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_get_logs_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_result_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_result_retrieve_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_job_results_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/benchmark_evaluation_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/benchmark_metric_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_jobs/results/row_score_download_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_offline_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_offline_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_agent_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_agent_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_online_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_retrieve_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmark_retrieve_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/benchmarks_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/bleu_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/built_in_dataset.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_entity_recall_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_precision_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_recall_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/context_relevance_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/dataset_rows.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/dataset_rows_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/evaluate_dataset_rows_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/exact_match_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/extended_benchmark.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/f1_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/faithfulness_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/field_mapping.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/field_mapping_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/fileset.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/fileset_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/fileset_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/histogram.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/histogram_bin.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/inference_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/inference_params_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/json_score_parser.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/json_score_parser_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/llm_judge_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_create_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluate_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_jobs_list_filter_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_jobs_page.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_jobs_sort_field.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_evaluation_row_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_get_logs_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_result.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_result_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_result_retrieve_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_job_results_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_jobs/results/row_score_download_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_list_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_offline_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_offline_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_agent_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_agent_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_online_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_retrieve_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_retriever_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_retriever_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metric_type.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/metrics_list_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/model.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/model_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/model_ref.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/nemo_agent_toolkit_remote_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/noise_sensitivity_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/number_check_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/parameter.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/parameter_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/percentiles.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/range_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/range_score_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/reasoning_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/reasoning_params_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/regex_score_parser.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/regex_score_parser_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/remote_score_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_groundedness_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/response_relevancy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/retriever_pipeline.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/retriever_pipeline_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rouge_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/row_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_score.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_score_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/rubric_score_stat.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online_model.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online_model_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_online_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/run_config_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/string_check_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_offline_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_offline_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_online_job.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_benchmark_online_job_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/system_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_call_accuracy_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/tool_calling_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/evaluation/topic_adherence_metric_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_create_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_metadata_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_metadata_param_param.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/files/fileset_update_params.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/error_response.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/field_error.py is excluded by !sdk/**
sdk/python/nemo-platform/src/nemo_platform/types/shared/fileset_metadata.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/test_aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/test_artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/results/test_row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/benchmark_jobs/test_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/__init__.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/test_aggregate_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/test_artifacts.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/results/test_row_scores.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/metric_jobs/test_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_benchmark_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_benchmark_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_benchmarks.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_metric_job_results.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_metric_jobs.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/evaluation/test_metrics.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/api_resources/test_models.py is excluded by !sdk/**
sdk/python/nemo-platform/tests/decoders/test_jsonl.py is excluded by !sdk/**
sdk/stainless.yaml is excluded by !sdk/**

📒 Files selected for processing (13)

docs/auth/authorization/api-scopes.md
docs/auth/authorization/permissions-reference.md
openapi/ga/individual/platform.openapi.yaml
openapi/ga/openapi.yaml
openapi/openapi.yaml
packages/nmp_platform_runner/src/nmp/platform_runner/registry.py
packages/nmp_platform_runner/tests/test_registry.py
services/core/auth/scripts/auth-tools.py
services/core/auth/src/nmp/core/auth/app/policy_tests/evaluation_namespace_check_test.rego
services/core/auth/src/nmp/core/auth/app/policy_tests/helpers_test.rego
services/core/auth/src/nmp/core/auth/app/policy_tests/models_namespace_check_test.rego
services/core/auth/src/nmp/core/auth/assets/static-authz.yaml
services/core/auth/tests/test_embedded_pdp.py

💤 Files with no reviewable changes (2)

docs/auth/authorization/permissions-reference.md
packages/nmp_platform_runner/src/nmp/platform_runner/registry.py

🚧 Files skipped from review as they are similar to previous changes (1)

packages/nmp_platform_runner/tests/test_registry.py

Signed-off-by: Sandy Chapman <schapman@nvidia.com>

PR #228 (stop registering legacy evaluator service) deleted the entire evaluation resource block from sdk/stainless.yaml, including the mapping for InferenceParams. That schema still exists in the OpenAPI spec because the shared PromptData schema references it (PromptData.inference_params), and PromptData is reached via the models service endpoints. With no explicit mapping, sync-models re-homes InferenceParams under the models resource (the common location of the endpoints that reference it). That generates types/models/inference_params.py and makes shared/prompt_data.py import from ..models.inference_params, which creates a circular import at SDK load time: shared/prompt_data -> models/inference_params -> models/__init__ -> model_entity -> shared/prompt_data breaking 'make stainless' / 'make update-sdk'. InferenceParams is a cross-cutting type: at the source level it lives in nmp_common and is imported by both the models service and the evaluator service. Its SDK home should not be any single feature resource. Pin it in $shared (alongside PromptData, which already lives there) so the generated import is intra-package (shared -> shared) and acyclic. sync-models now makes no changes and InferenceParams resolves to types/shared/inference_params.py. Signed-off-by: Ben McCown <bmccown@nvidia.com>

make update-sdk / make stainless fail on main because the regenerated SDK contains circular imports. Two shared schemas reference types that the mapper homes under feature resources, creating shared -> <resource> -> shared import cycles at SDK load time: 1) InferenceParams: shared/prompt_data references it. PR #228 removed the evaluation resource block from stainless.yaml (including this mapping), so sync-models re-homed it under models: shared/prompt_data -> models/inference_params -> models/__init__ -> model_entity -> shared/prompt_data 2) DatasetMetadataContent: shared/fileset_metadata references it while it was mapped under files: shared/fileset_metadata -> files/dataset_metadata_content -> files/__init__ -> files/fileset -> shared/fileset_metadata The committed SDK only avoided this via a TYPE_CHECKING guard, which the Stainless regen drops, turning it into a real runtime import cycle. Both are genuinely cross-cutting types. Pin them in $shared (alongside PromptData / FilesetMetadata, which already live there) so the generated imports are intra-package (shared -> shared) and acyclic. The *_param variants follow automatically into shared_params. sync-models is clean and makes no further changes. Signed-off-by: Ben McCown <bmccown@nvidia.com>

make update-sdk / make stainless fail on main because the regenerated SDK contains circular imports. Two shared schemas reference types that the mapper homes under feature resources, creating shared -> <resource> -> shared import cycles at SDK load time: 1) InferenceParams: shared/prompt_data references it. PR #228 removed the evaluation resource block from stainless.yaml (including this mapping), so sync-models re-homed it under models: shared/prompt_data -> models/inference_params -> models/__init__ -> model_entity -> shared/prompt_data 2) DatasetMetadataContent: shared/fileset_metadata references it while it was mapped under files: shared/fileset_metadata -> files/dataset_metadata_content -> files/__init__ -> files/fileset -> shared/fileset_metadata The committed SDK only avoided this via a TYPE_CHECKING guard, which the Stainless regen drops, turning it into a real runtime import cycle. Both are genuinely cross-cutting types. Pin them in $shared (alongside PromptData / FilesetMetadata, which already live there) so the generated imports are intra-package (shared -> shared) and acyclic. The *_param variants follow automatically into shared_params. Includes the regenerated SDK from make update-sdk. Verified: - make update-sdk completes end-to-end (stainless + web + cli) - nemo-platform-sdk-tools is-up-to-date passes - the SDK imports with no circular-import error Signed-off-by: Ben McCown <bmccown@nvidia.com>

…teJob The legacy platform evaluator service was removed in #228, deleting all metric/benchmark entity management APIs (MetricEvaluationJob, useEvaluationListMetricJobs, useEvaluationListBenchmarks, etc.) from the platform SDK. Studio's evaluation UI referenced these exclusively, breaking studio-ci web-typecheck on main. Migrated to the new nemo-evaluator plugin SDK (EvaluateJob-based API): - Replaced MetricEvaluationJob/BenchmarkEvaluationJob with EvaluateJob - Replaced useEvaluationList/Get/DeleteJob with useEvaluatorList/Get/DeleteEvaluateJob - Removed metric entity management routes (no equivalent in new API): EvaluationBenchmarksRoute, EvaluationMetricsRoute, EvaluationMetricCreate/DetailsRoute - Removed benchmark/metric entity data views and side panels - Kept and migrated: EvaluationResultsRoute, EvaluationResultDetailsRoute, ActionMenu, JobBulkDeleteModal, ComparisonPanel, DetailsPanel, evaluationJob selectors, util/evaluations Signed-off-by: mschwab <mschwab@nvidia.com>

SandyChapman marked this pull request as ready for review June 9, 2026 14:54

SandyChapman requested review from a team as code owners June 9, 2026 14:54

arpitsardhana approved these changes Jun 9, 2026

View reviewed changes

SandyChapman force-pushed the remove-evaluator-runner-registration/schapman branch 2 times, most recently from 49941f7 to 06ce64d Compare June 9, 2026 16:18

coderabbitai Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread docs/auth/authorization/permissions-reference.md Outdated

SandyChapman force-pushed the remove-evaluator-runner-registration/schapman branch from 06ce64d to 28ddc42 Compare June 9, 2026 16:29

coderabbitai Bot reviewed Jun 9, 2026

View reviewed changes

SandyChapman force-pushed the remove-evaluator-runner-registration/schapman branch 3 times, most recently from 8890835 to 849cfa8 Compare June 9, 2026 18:53

chore(platform-runner): stop registering legacy evaluator service

98b13a3

Signed-off-by: Sandy Chapman <schapman@nvidia.com>

SandyChapman force-pushed the remove-evaluator-runner-registration/schapman branch from 849cfa8 to 98b13a3 Compare June 9, 2026 18:55

SandyChapman added this pull request to the merge queue Jun 9, 2026

Merged via the queue into main with commit a94d9db Jun 9, 2026
36 checks passed

coderabbitai Bot mentioned this pull request Jun 9, 2026

chore(packaging): remove legacy evaluator wiring #230

Merged

benmccown mentioned this pull request Jun 9, 2026

fix(sdk): pin InferenceParams to $shared in stainless config #246

Merged

marcusds mentioned this pull request Jun 9, 2026

fix(studio): wire evaluator plugin SDK + migrate eval UI to EvaluateJob API #251

Merged

4 tasks

Navigation Menu

Uh oh!

chore(platform-runner): stop registering legacy evaluator service - #228

chore(platform-runner): stop registering legacy evaluator service#228
SandyChapman merged 1 commit into
mainfrom
remove-evaluator-runner-registration/schapman

SandyChapman commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Possibly related PRs

Suggested Reviewers

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

SandyChapman commented Jun 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Possibly related PRs

Suggested Reviewers

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation preview is ready

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SandyChapman commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 9, 2026 •

edited

Loading