Skip to content

chore: remove run (local execution) CLI verbs across all services #1228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ironcommit wants to merge 1 commit into from
Draft

chore: remove run (local execution) CLI verbs across all services #1228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/agents/optimization.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ from nemo_agents_plugin.jobs.optimize_skills import OptimizeSkillsJob
from nemo_platform_plugin.scheduler import NemoJobScheduler

spec = yaml.safe_load(Path(".agent-improver.yml").read_text())
NemoJobScheduler().run_local(
NemoJobScheduler().submit_remote(
OptimizeSkillsJob,
spec,
workspace="default",
Expand Down Expand Up @@ -342,7 +342,7 @@ client = NeMoPlatform(
workspace=WORKSPACE,
)

result = NemoJobScheduler().run_local(
result = NemoJobScheduler().submit_remote(

@mckornfield mckornfield Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 / blocking — R8

run_local → submit_remote left sdk=client (not a submit_remote parameter → TypeError) and does not pass the required base_url (raises ValueError when missing).

Same pattern in the other submit_remote examples in this file, plugins/nemo-optimization/examples/hermes-optimize/README.md, and the minimal ARCHITECTURE.md example.

OptimizeJob,
{
"optimize_config": str(optimize_config),
Expand Down Expand Up @@ -414,7 +414,7 @@ client = NeMoPlatform(
workspace=WORKSPACE,
)

result = NemoJobScheduler().run_local(
result = NemoJobScheduler().submit_remote(
OptimizeJob,
{
"optimize_config": str(optimize_config),
Expand Down
46 changes: 20 additions & 26 deletions docs/anonymizer/cli.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ This reference covers the `nemo anonymizer` commands exposed by the Anonymizer p
| Command | Source | Description |
|----------------------------------|---------------------------------|--------------------------------------------------------------|
| `nemo anonymizer validate` | Manual Typer command | Validate an `AnonymizerConfig` (and optional `model_configs`).|
| `nemo anonymizer preview run` | Generated from `NemoFunction` | Local streaming preview. |
| `nemo anonymizer preview submit` | Generated from `NemoFunction` | Local streaming preview. |
| `nemo anonymizer preview submit` | Generated from `NemoFunction` | Remote streaming preview against the plugin service. |
| `nemo anonymizer run run` | Generated from `NemoJob` | Local job execution in the CLI process. |
| `nemo anonymizer run submit` | Generated from `NemoJob` | Local job execution in the CLI process. |
| `nemo anonymizer run submit` | Generated from `NemoJob` | Submit an `anonymizer.run` job to the NeMo Platform Jobs worker. |
| `nemo anonymizer run explain` | Generated from `NemoJob` | Print the job key, submit endpoint, and JSON schemas. |

Expand All @@ -39,10 +39,10 @@ The command does not accept `data.source`. Input-source validation happens durin

## `nemo anonymizer preview`

Both `preview run` and `preview submit` take a spec file matching `PreviewRequest`.
Both `preview submit` and `preview submit` take a spec file matching `PreviewRequest`.

```bash
nemo anonymizer preview run \
nemo anonymizer preview submit \
--spec-file /tmp/anonymizer-preview.yaml \
--workspace "${NMP_WORKSPACE:-default}"

Expand All @@ -60,7 +60,7 @@ nemo anonymizer preview submit \

### Preview source kinds

| Form | `preview run` | `preview submit` |
| Form | `preview submit` | `preview submit` |
|---------------------------------------|---------------|------------------|
| Local path (`/tmp/input.csv`) | yes | no |
| HTTP(S) URL (`https://.../input.csv`) | yes | yes |
Expand All @@ -71,7 +71,7 @@ nemo anonymizer preview submit \
`preview` streams newline-delimited JSON frames to stdout. Filter with `jq`:

```bash
nemo anonymizer preview run --spec-file /tmp/anonymizer-preview.yaml > /tmp/preview.ndjson
nemo anonymizer preview submit --spec-file /tmp/anonymizer-preview.yaml > /tmp/preview.ndjson

jq -R 'fromjson? | select(.kind == "preview_dataset") | .records' /tmp/preview.ndjson
```
Expand All @@ -81,7 +81,7 @@ Frame kinds: `log`, `preview_dataset`, `trace_dataset`, `failed_records`, `heart
## `nemo anonymizer run`

```bash
nemo anonymizer run run --spec-file /tmp/anonymizer-run.yaml
nemo anonymizer run submit --spec-file /tmp/anonymizer-run.yaml

nemo anonymizer run submit \
--spec-file /tmp/anonymizer-run.yaml \
Expand All @@ -98,21 +98,23 @@ nemo anonymizer run explain

### Run source kinds

| Form | `run run` | `run submit` |
|---------------------------------------|-----------|--------------|
| Local path (`/tmp/input.csv`) | yes | no |
| HTTP(S) URL (`https://.../input.csv`) | yes | yes |
| Fileset reference (`fs#path`) | yes | yes |
| Form | `run submit` |
|---------------------------------------|--------------|
| Local path (`/tmp/input.csv`) | no |
| HTTP(S) URL (`https://.../input.csv`) | yes |
| Fileset reference (`fs#path`) | yes |

### Run output

`run run` prints `{"exit_code": 0}` on success. The local job results manager logs the artifact directory to stderr:
`run submit` submits an `anonymizer.run` job to the Jobs service and prints the assigned job name and submit endpoint:

```text
Saved result 'artifacts' to file:///.../persistent/results/artifacts
|-- job name: anonymizer-run-2026-05-12-abc123
|-- submit endpoint: /apis/anonymizer/v2/workspaces/default/jobs/run
{"name": "anonymizer-run-2026-05-12-abc123", ...}
```

The artifact directory contains:
The downloaded artifact directory contains:

| File | Description |
|-----------------------|--------------------------------------------------------------|
Expand All @@ -121,14 +123,6 @@ The artifact directory contains:
| `metadata.json` | Run metadata (includes original text column). |
| `failed_records.json` | Per-record failures. Only written when records failed. |

`run submit` submits an `anonymizer.run` job to the Jobs service and prints the assigned job name and submit endpoint:

```text
|-- job name: anonymizer-run-2026-05-12-abc123
|-- submit endpoint: /apis/anonymizer/v2/workspaces/default/jobs/run
{"name": "anonymizer-run-2026-05-12-abc123", ...}
```

Track and pull artifacts using either the standard `nemo jobs ...` commands or the Python SDK:

```bash
Expand All @@ -145,7 +139,7 @@ dataset = results.load_dataset()

See [SDK Resources](/documentation/anonymize-data/sdk-resources) for the full `AnonymizerJobResource` / `AnonymizerJobResults` surface.

Compared to `run run`, `run submit` rejects local file paths in `data.source` (use a fileset reference or `http(s)` URL) and requires explicit `model_configs` because the job runs outside the CLI process.
`run submit` rejects local file paths in `data.source` (use a fileset reference or `http(s)` URL) and requires explicit `model_configs` because the job runs on the platform worker.

## Spec File Reference

Expand All @@ -154,11 +148,11 @@ Both preview and run specs use the shared `AnonymizerRequest` shape:
| Field | Type | Required | Notes |
|-------------------|-------------------------------------------------|----------|------------------------------------------------------------------------|
| `config` | `AnonymizerConfig` | yes | Library config. See the [library docs](https://github.com/NVIDIA-NeMo/Anonymizer/tree/main/docs). |
| `data.source` | string | yes | Local path, `http(s)` URL, or fileset reference. |
| `data.source` | string | yes | `http(s)` URL or fileset reference. |
| `data.text_column`| string | no | Defaults to `text`. |
| `data.id_column` | string | no | Optional record identifier column. |
| `data.data_summary` | string | no | Optional short description of the data. |
| `model_configs` | list of Data Designer `ModelConfig` | depends | Required for `preview submit` and `run submit`; optional for `preview run` and `run run`. |
| `model_configs` | list of Data Designer `ModelConfig` | depends | Required for `preview submit` and `run submit`; optional for `preview submit` and `run submit`. |
| `selected_models` | object with `detection` / `replace` / `rewrite` | no | Role overrides on top of bundled defaults. Requires `model_configs`. |

Preview-only:
Expand Down
13 changes: 6 additions & 7 deletions docs/anonymizer/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,10 @@ preview_result.trace_dataset # detection trace
preview_result.display_record(0) # render a record with entity highlights
```

For a full anonymization run, execute the job locally or submit it to the Jobs worker:
For a full anonymization run, submit the job to the Jobs worker:

```bash
nemo anonymizer run run --spec-file /path/to/run-spec.yaml # in-process
nemo anonymizer run submit --spec-file /path/to/run-spec.yaml # NeMo Services job
nemo anonymizer run submit --spec-file /path/to/run-spec.yaml
```

The SDK equivalent of `run submit` is `sdk.anonymizer.run(request)`, which returns an `AnonymizerJobResource` you can poll with `wait_until_done()` and pull artifacts from with `download_artifacts()`.
Expand All @@ -80,9 +79,9 @@ When using Anonymizer as a NeMo Platform service:
| Feature | Standalone Library | NeMo Platform Service |
|-------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| **Inference** | Direct calls to NVIDIA Build defaults | Routes through the Inference Gateway via `model_configs` |
| **Execution** | Local Python process | Streaming preview runs in the plugin service; full runs execute either in the local CLI (`run run`) or on the Jobs worker (`run submit`) |
| **Input sources** | Local file, `http(s)` URL | Local file (`run run` only), `http(s)` URL, or NeMo Platform Fileset |
| **Artifacts** | Local filesystem | Local artifact directory (`persistent/results/artifacts`) for `run run`; NeMo Platform job artifact storage for `run submit` |
| **Execution** | Local Python process | Streaming preview submits in the plugin service; full runs execute on the Jobs worker |
| **Input sources** | Local file, `http(s)` URL | `http(s)` URL or NeMo Platform Fileset |
| **Artifacts** | Local filesystem | NeMo Platform job artifact storage for `run submit` |
| **Authentication**| Direct API keys | NeMo Platform Secrets service |

## Replacement Strategies
Expand All @@ -106,7 +105,7 @@ This package is a thin wrapper around the [NVIDIA NeMo Anonymizer library](https
- A `nemo anonymizer` CLI with `validate`, `preview`, and `run` command groups.
- An `sdk.anonymizer` SDK accessor (`AnonymizerResource`, `AsyncAnonymizerResource`).
- A streaming `anonymizer.preview` function that emits `preview_dataset`, `trace_dataset`, and `failed_records` frames from the plugin service.
- An `anonymizer.run` job that writes `dataset.parquet`, `trace.parquet`, `metadata.json`, and optional `failed_records.json`. The job can execute in the local CLI process (`nemo anonymizer run run`) or on the NeMo Platform Jobs worker (`nemo anonymizer run submit` / `sdk.anonymizer.run`).
- An `anonymizer.run` job that writes `dataset.parquet`, `trace.parquet`, `metadata.json`, and optional `failed_records.json` on the NeMo Platform Jobs worker (`nemo anonymizer run submit` / `sdk.anonymizer.run`).
- Fileset input handling (`fileset://<workspace>/<fileset>#<path>`).
- Inference Gateway routing for model providers referenced from `model_configs`.

Expand Down
2 changes: 1 addition & 1 deletion docs/anonymizer/sdk-resources.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ The async variant (`AsyncAnonymizerJobResource`) exposes the same surface with `

## AnonymizerJobResults

`download_artifacts` returns an `AnonymizerJobResults` object that loads parquet / JSON artifacts into memory. The same class also works for the local `run run` flow — point it at the artifact directory the local job results manager logs:
`download_artifacts` returns an `AnonymizerJobResults` object that loads parquet / JSON artifacts into memory. You can also point it at an extracted artifact directory:

```python
from pathlib import Path
Expand Down
8 changes: 4 additions & 4 deletions docs/anonymizer/tutorials/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ When using Anonymizer as a NeMo Platform service:
| Feature | Difference | Details |
|----------------|---------------------------------------------------------|----------------------------------------------------------------------------------------|
| **Inference** | Routes through the Inference Gateway | Configure providers once and reference them by name from `model_configs`. |
| **Input data** | Filesets and HTTP(S) URLs (local paths only in local CLI execution) | Use `sdk.files.filesets.create` / `sdk.files.upload`, then reference with `#<path>`. |
| **Artifacts** | Local or platform-managed | `run run` writes to `persistent/results/artifacts` locally; `run submit` stores artifacts in NeMo Platform job storage. |
| **Input data** | Filesets and HTTP(S) URLs | Use `sdk.files.filesets.create` / `sdk.files.upload`, then reference with `#<path>`. |
| **Artifacts** | Platform-managed | `run submit` stores artifacts in NeMo Platform job storage. |

## Prerequisites

Expand Down Expand Up @@ -152,14 +152,14 @@ The tutorials reference this file with `fileset://{WORKSPACE}/anonymizer-inputs#

<Card title="Preview a Config" href="/documentation/anonymize-data/tutorials/preview-a-config">

Stream a small anonymized sample to iterate on `AnonymizerConfig` and `model_configs`. Covers `sdk.anonymizer.preview`, `nemo anonymizer preview run` / `preview submit`, and the NDJSON frame stream.
Stream a small anonymized sample to iterate on `AnonymizerConfig` and `model_configs`. Covers `sdk.anonymizer.preview`, `nemo anonymizer preview submit` / `preview submit`, and the NDJSON frame stream.

<small><span class="md-tag">beginner</span> <span class="md-tag">anonymizer</span></small>

</Card>
<Card title="Run an Anonymizer Job" href="/documentation/anonymize-data/tutorials/run-an-anonymizer-job">

Run the full pipeline locally with `nemo anonymizer run run` or submit it to the Jobs worker with `nemo anonymizer run submit`. Load `dataset.parquet`, `trace.parquet`, and `failed_records.json` artifacts.
Submit the full pipeline with `nemo anonymizer run submit`. Load `dataset.parquet`, `trace.parquet`, and `failed_records.json` artifacts.

<small><span class="md-tag">intermediate</span> <span class="md-tag">anonymizer</span></small>

Expand Down
30 changes: 10 additions & 20 deletions docs/anonymizer/tutorials/preview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ Available preview surfaces:
| Surface | Where it runs | Local paths | `model_configs` required |
|----------------------------------|--------------------------------------|-------------|--------------------------|
| `sdk.anonymizer.preview(...)` | Anonymizer plugin service (remote) | Rejected | Required |
| `nemo anonymizer preview run` | Local CLI process | Allowed | Optional |
| `nemo anonymizer preview submit` | Anonymizer plugin service (remote) | Rejected | Required |

## Step 1: Build a `PreviewRequest`
Expand Down Expand Up @@ -87,7 +86,7 @@ Field reference:
| `data.text_column`| string | Column containing text to anonymize. Defaults to `text`. |
| `data.id_column` | string | Optional record identifier column. |
| `data.data_summary`| string | Optional short description passed to Anonymizer library prompts. |
| `model_configs` | list | Data Designer `ModelConfig` entries. `provider` must reference an Inference Gateway provider name (or `workspace/provider`). Omit to use Anonymizer library defaults (CLI local execution only). |
| `model_configs` | list | Data Designer `ModelConfig` entries. `provider` must reference an Inference Gateway provider name (or `workspace/provider`). Required for SDK and CLI preview. |
| `selected_models` | object | Optional `detection` / `replace` / `rewrite` overrides on top of the bundled defaults. Requires `model_configs`. |
| `num_records` | int (≥ 1) | Number of records to preview. Defaults to 10. |

Expand Down Expand Up @@ -143,7 +142,7 @@ preview.dataset.to_parquet("anonymized-preview.parquet", index=False)
</Accordion>
## Step 4: Run from the CLI (Alternative)

The CLI accepts the same request shape as a YAML spec file. Use `preview run` for local execution (allows local paths, model configs optional) or `preview submit` for the plugin service path (same as `sdk.anonymizer.preview`).
The CLI accepts the same request shape as a YAML spec file. `preview submit` uses the plugin service path, the same as `sdk.anonymizer.preview`.

Write the spec to YAML:

Expand All @@ -155,15 +154,7 @@ spec_path = Path("/tmp/anonymizer-preview.yaml")
spec_path.write_text(yaml.safe_dump(request.model_dump(mode="json", exclude_none=True)))
```

Run preview locally:

```bash
nemo anonymizer preview run \
--spec-file /tmp/anonymizer-preview.yaml \
--workspace "${NMP_WORKSPACE:-default}"
```

Or submit to the plugin service:
Submit to the plugin service:

```bash
nemo anonymizer preview submit \
Expand All @@ -172,10 +163,10 @@ nemo anonymizer preview submit \
--base-url "${NMP_BASE_URL:-http://localhost:8080}"
```

Both commands stream NDJSON frames to stdout. Filter with `jq`:
The CLI command streams NDJSON frames to stdout. Filter with `jq`:

```bash
nemo anonymizer preview run \
nemo anonymizer preview submit \
--spec-file /tmp/anonymizer-preview.yaml \
--workspace "${NMP_WORKSPACE:-default}" \
> /tmp/anonymizer-preview.ndjson
Expand All @@ -188,13 +179,12 @@ If `preview submit` returns 404 against the gateway, the plugin service isn't mo

## Input Source Forms

The plugin accepts three forms for `data.source`:
The plugin accepts two forms for `data.source`:

| Form | `sdk.anonymizer.preview` / `preview submit` | `preview run` |
|---------------------------------------|-----------------------------------|---------------|
| Local path (`/tmp/input.csv`) | No | Yes |
| HTTP(S) URL (`https://.../input.csv`) | Yes | Yes |
| Fileset reference | Yes | Yes |
| Form | Supported |
|---------------------------------------|-----------|
| HTTP(S) URL (`https://.../input.csv`) | Yes |
| Fileset reference | Yes |

Fileset references take any of these forms; the workspace and fileset must already exist:

Expand Down
Loading
Loading