refactor(evaluator)!: drop the plugin's local execution path

docs/evaluator/agent-eval/index.mdx

@@ -17,8 +17,8 @@ got there*. Each task carries its own metrics, so a single suite can grade heter
 - Use `await AgentEvaluator().run(tasks=..., target=...)` for local task-driven SDK runs
 that do not require running nemo-platform.
 - Use the Evaluator plugin's `uv run nemo evaluator agent-evaluate submit` job for durable runs with inline
-tasks or stored tasksets. The high-level `client.evaluator.run/submit` interfaces
-above remain dataset-driven only.
+tasks or stored tasksets. The high-level `client.evaluator.submit` interface
+above remains dataset-driven only.
 
 </Note>
 

docs/evaluator/index.mdx

@@ -85,14 +85,13 @@ config = RunConfig(limit_samples=100, parallelism=8)
 
 ### 2. Run it — three dataset-driven modes
 
-The same `metric`, `dataset`, and `config` run in three places. What changes is the **caller** — a
-bare SDK evaluator for local iteration, or the platform's `client.evaluator` resource for
-plugin-backed and durable execution.
+The same `metric`, `dataset`, and `config` run in two places. What changes is the **caller** — a
+bare SDK evaluator for local iteration, or the platform's `client.evaluator` resource for durable
+execution.
 
 | Mode | Caller | Call | Best for |
 |------|--------|------|----------|
-| **Local SDK** | `Evaluator()` (`nemo_evaluator_sdk`) | `await evaluator.run(metrics=metric, dataset=dataset, config=config)` | Fast in-process iteration with no platform services. |
-| **Local plugin** | `client.evaluator` | `evaluator.run(metric=metric, dataset=dataset, config=config)` | Local runs through the platform runtime and Inference Gateway. |
+| **Local SDK** | `Evaluator()` (`nemo_evaluator_sdk`) | `await evaluator.run(metrics=[metric], dataset=dataset, config=config)` | Fast in-process iteration with no platform services. |
 | **Remote job** | `client.evaluator` | `evaluator.submit(metric=metric, dataset=dataset, config=config)` | Durable, monitored platform jobs for production and regressions. |
 
 The platform caller is mounted on a `NeMoPlatform` client:
@@ -109,9 +108,6 @@ client = NeMoPlatform(
 )
 evaluator: Evaluator = client.evaluator
 
-# Fast local iteration through the plugin runtime.
-local_result = evaluator.run(metric=metric, dataset=dataset, config=config)
-
 # Production evaluation as a durable platform job.
 job = evaluator.submit(metric=metric, dataset=dataset, config=config)
 job.wait_until_done()
@@ -124,8 +120,8 @@ result = job.get_result()
 - Use `await AgentEvaluator().run(tasks=..., target=...)` for local task-driven SDK runs
 that do not require running nemo-platform.
 - Use the Evaluator plugin's `uv run nemo evaluator agent-evaluate submit` job for durable runs with inline
-tasks or stored tasksets. The high-level `client.evaluator.run/submit` interfaces
-above remain dataset-driven only.
+tasks or stored tasksets. The high-level `client.evaluator.submit` interface
+above remains dataset-driven only.
 
 </Note>
 
@@ -136,7 +132,7 @@ the same and execution gains platform capabilities:
 
 | Capability | Local SDK | Platform (`client.evaluator`) |
 |------------|-----------|-------------------------------|
-| **Execution** | Local in-process run | Local plugin runs, plus durable platform jobs |
+| **Execution** | Local in-process run | Durable platform jobs |
 | **Inference** | Direct model or agent endpoint calls | The same, and can route through the NeMo Platform [Inference Gateway](/documentation/models-and-inference) and platform-managed endpoints |
 | **Datasets** | Inline rows and local files | Inline rows, local paths resolved at submission time, and NeMo Platform [Filesets](/documentation/get-started/core-concepts/manage-files) |
 | **Results** | Returned in memory | Platform artifact storage with typed result download |

docs/evaluator/metrics/agent-configuration.mdx

@@ -37,7 +37,7 @@ evaluator: Evaluator = client.evaluator  # this object is an Evaluator resource
 
 If your agent endpoint requires authentication, configure `api_key_secret` on the `Agent`.
 
-For local `evaluator.run(...)` calls, `api_key_secret` must name an environment variable available to the local Python process. For remote `evaluator.submit(...)` jobs, it must name a NeMo platform secret in the target workspace. See [Model API Authentication](/documentation/evaluate-models/metrics/model-configuration#model-api-authentication) for the local-versus-remote behavior.
+For `evaluator.submit(...)` jobs, `api_key_secret` must name a NeMo platform secret in the target workspace. See [Model API Authentication](/documentation/evaluate-models/metrics/model-configuration#model-api-authentication).
 
 For remote `evaluator.submit(...)` jobs, create the secret in the platform workspace before submitting the job:
 
@@ -89,7 +89,7 @@ agent = Agent(
     trajectory_path="$.reasoning_steps",
 )
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {"question": "What is the capital of France?", "expected_answer": "Paris"},
@@ -98,6 +98,8 @@ result = evaluator.run(
     target=agent,
     prompt_template="Question: {{item.question}}\nAnswer:",
 )
+job.wait_until_done()
+result = job.get_result()
 for score in result.aggregate_scores.scores:
     print(f"{score.name}: mean={score.mean}")
 ```

docs/evaluator/metrics/agentic.mdx

@@ -73,7 +73,7 @@ Agentic metrics evaluate different aspects of agent behavior:
 
 <Note>
 
-Use `evaluator.run(...)` for local in-process evaluation and `evaluator.submit(...)` for durable remote platform jobs. The examples below use inline dataset rows through `dataset=[...]`, but you can use a file Path or a FilesetRef instead.
+Use `evaluator.submit(...)` for durable platform jobs. For local in-process evaluation without the platform, use `nemo_evaluator_sdk.Evaluator` directly. The examples below use inline dataset rows through `dataset=[...]`, but you can use a file Path or a FilesetRef instead.
 
 </Note>
 ## Prerequisites
@@ -135,7 +135,7 @@ from nemo_evaluator_sdk import (
 )
 ```
 
-Use `dataset=[...]` for inline rows. For offline scoring options, use `config=RunConfig(parallelism=...)`. Whenever outputs must be generated before scoring, pass `target=Model(...)` or `target=Agent(...)` plus the corresponding online parameters. Use the same `dataset`, `config`, and `target` arguments for both `evaluator.run(...)` and `evaluator.submit(...)`; durable jobs follow the identical pattern as local runs and only differ in waiting for and fetching results.
+Use `dataset=[...]` for inline rows. For offline scoring options, use `config=RunConfig(parallelism=...)`. Whenever outputs must be generated before scoring, pass `target=Model(...)` or `target=Agent(...)` plus the corresponding online parameters. Pass `dataset`, `config`, and `target` to `evaluator.submit(...)`, then wait for the job and fetch its result.
 
 ---
 
@@ -193,13 +193,13 @@ Evaluates whether the agent invoked the correct tools with the correct arguments
 
 <Tabs>
 
-<Tab title="Run Locally">
+<Tab title="Offline Scoring">
 
 ```python
 from nemo_evaluator_sdk.metrics.ragas import ToolCallAccuracyMetric
 metric = ToolCallAccuracyMetric()
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -219,6 +219,8 @@ result = evaluator.run(
         }
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 print(result.aggregate_scores)
 ```
 
@@ -366,15 +368,15 @@ Data must use OpenAI-compliant tool calling format:
 </Note>
 <Tabs>
 
-<Tab title="Run Locally">
+<Tab title="Offline Scoring">
 
 ```python
 
 
 from nemo_evaluator_sdk import ToolCallingMetric
 metric = ToolCallingMetric(reference="{{item.tool_calls}}")
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -420,6 +422,8 @@ result = evaluator.run(
         }
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 print(result.aggregate_scores)
 ```
 
@@ -531,13 +535,13 @@ Measures how well the agent maintained focus on assigned topics throughout a con
 
 <Note>
 
-Topic Adherence is a multi-turn metric: it scores a complete conversation supplied as a `user_input` message list. Online target generation produces a single response from a single prompt and cannot construct a multi-turn conversation, so it is not supported for this metric. Use the offline **Run Locally** or **Submit Job** modes with a pre-built multi-turn conversation.
+Topic Adherence is a multi-turn metric: it scores a complete conversation supplied as a `user_input` message list. Online target generation produces a single response from a single prompt and cannot construct a multi-turn conversation, so it is not supported for this metric. Use the offline **Offline Scoring** or **Submit Job** modes with a pre-built multi-turn conversation.
 
 </Note>
 
 <Tabs>
 
-<Tab title="Run Locally">
+<Tab title="Offline Scoring">
 
 ```python
 from nemo_evaluator_sdk import InferenceParams, Model, SecretRef
@@ -557,7 +561,7 @@ metric = TopicAdherenceMetric(
     inference=InferenceParams.model_validate({"temperature": 0, "response_format": {"type": "json_object"}}),
 )
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -572,6 +576,8 @@ result = evaluator.run(
         }
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 print(result.aggregate_scores)
 ```
 
@@ -701,7 +707,7 @@ Compare the agent's outcome against a known reference:
 
 <Tabs>
 
-<Tab title="Run Locally">
+<Tab title="Offline Scoring">
 
 ```python
 from nemo_evaluator_sdk import Model, SecretRef
@@ -714,7 +720,7 @@ judge_model = Model(
 )
 metric = AgentGoalAccuracyMetric(use_reference=True, judge_model=judge_model)
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -735,6 +741,8 @@ result = evaluator.run(
         }
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 print(result.aggregate_scores)
 ```
 
@@ -848,7 +856,7 @@ The judge LLM infers the goal from the conversation context:
 
 <Tabs>
 
-<Tab title="Run Locally">
+<Tab title="Offline Scoring">
 
 ```python
 from nemo_evaluator_sdk import Model, SecretRef
@@ -861,7 +869,7 @@ judge_model = Model(
 )
 metric = AgentGoalAccuracyMetric(use_reference=False, judge_model=judge_model)
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -890,6 +898,8 @@ result = evaluator.run(
         }
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 print(result.aggregate_scores)
 ```
 
@@ -963,7 +973,7 @@ Evaluates the factual correctness of an agent's answer by comparing it against a
 
 <Tabs>
 
-<Tab title="Run Locally">
+<Tab title="Offline Scoring">
 
 ```python
 from nemo_evaluator_sdk import Model, SecretRef
@@ -976,7 +986,7 @@ judge_model = Model(
 )
 metric = AnswerAccuracyMetric(judge_model=judge_model)
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -986,6 +996,8 @@ result = evaluator.run(
         }
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 print(result.aggregate_scores)
 ```
 

docs/evaluator/metrics/index.mdx

@@ -8,7 +8,7 @@ Metrics define how to score the outputs of your models, agents, or pipelines.
 
 ## What is a metric?
 
-A metric is a scoring definition that evaluates model or agent outputs. In the Evaluator plugin SDK, metrics are inline Python objects passed directly to `evaluator.run(...)` or `evaluator.submit(...)`.
+A metric is a scoring definition that evaluates model or agent outputs. In the Evaluator plugin SDK, metrics are inline Python objects passed directly to `evaluator.submit(...)`.
 
 - **Inputs**: For custom metrics, inputs define scoring logic composed of dataset fields and model outputs; for judge-based custom metrics, this also includes judge-model inputs (for example, judge prompts/rubrics and configuration).
 - **Outputs**: Row-level scores and aggregate statistics.
@@ -57,13 +57,15 @@ evaluator: Evaluator = client.evaluator
 
 metric = ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}")
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {"expected": "Paris", "output": "Paris"},
         {"expected": "Berlin", "output": "Munich"},
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 
 print(result.aggregate_scores)
 ```

docs/evaluator/metrics/llm-as-a-judge.mdx

@@ -18,7 +18,6 @@ NeMo Evaluator supports two execution modes through the Evaluator plugin SDK:
 
 | Mode | Use Case | SDK Call |
 |------|----------|----------|
-| **Local execution** | Rapid prototyping, metric development, and synchronous workflows | `evaluator.run(metric=metric, dataset=dataset)` |
 | **Durable remote job** | Production workloads that should run as platform jobs | `evaluator.submit(metric=metric, dataset=dataset)` |
 
 ## Prerequisites
@@ -114,7 +113,7 @@ metric = LLMJudgeMetric(
 )
 
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -127,6 +126,8 @@ result = evaluator.run(
         },
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 
 for score in result.aggregate_scores.scores:
     print(f"{score.name}: mean={score.mean:.2f}, count={score.count}")
@@ -222,7 +223,7 @@ metric = LLMJudgeMetric(
     },
 )
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {
@@ -231,8 +232,9 @@ result = evaluator.run(
         },
         {"input": "Explain quantum physics", "output": "I don't know."},
     ],
-    aggregate_fields=("rubric_distribution", "mode_category"),
 )
+job.wait_until_done()
+result = job.get_result(aggregate_fields=("rubric_distribution", "mode_category"))
 
 print(result.aggregate_scores.model_dump(exclude_none=True))
 ```
@@ -244,14 +246,15 @@ By default, aggregate scores include `count`, `mean`, `min`, and `max`. Request
 
 
 ```python
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {"input": "What is the capital of France?", "output": "Paris."},
         {"input": "What is 2 + 2?", "output": "4."},
     ],
-    aggregate_fields=("std_dev", "variance"),
 )
+job.wait_until_done()
+result = job.get_result(aggregate_fields=("std_dev", "variance"))
 
 for score in result.aggregate_scores.scores:
     print(f"{score.name}:")

docs/evaluator/metrics/manage-metrics.mdx

@@ -4,7 +4,7 @@ description: ""
 ---
 <a id="eval-metrics-manage-metrics"></a>
 
-Instantiate the metric class you want to run and pass it with `dataset` and optional configuration to `evaluator.run(...)` or `evaluator.submit(...)`.
+Instantiate the metric class you want to run and pass it with `dataset` and optional configuration to `evaluator.submit(...)`.
 
 ## Initialize the SDK
 
@@ -35,13 +35,15 @@ metric = ExactMatchMetric(
     candidate="{{item.output}}",
 )
 
-result = evaluator.run(
+job = evaluator.submit(
     metric=metric,
     dataset=[
         {"expected": "Paris", "output": "Paris"},
         {"expected": "Berlin", "output": "Munich"},
     ],
 )
+job.wait_until_done()
+result = job.get_result()
 
 for score in result.aggregate_scores.scores:
     print(f"{score.name}: mean={score.mean}")