diff --git a/docs/anonymizer/index.md b/docs/anonymizer/index.md index dfbd0c07e5..f61bc01e61 100644 --- a/docs/anonymizer/index.md +++ b/docs/anonymizer/index.md @@ -12,7 +12,7 @@ The service wraps the open-source [NVIDIA NeMo Anonymizer library](https://githu The library defines **what** to anonymize and **how**. The platform decides **where the work runs** and **how models are reached**. !!! note - The code snippets below are for conceptual demonstration purposes only. For runnable examples, see the [quickstart](quickstart.md) and [tutorials](tutorials/index.md). + The code snippets below are for conceptual demonstration purposes only. For runnable examples, see the [tutorials](tutorials/index.md). ### 1. Build a config with the library @@ -106,12 +106,6 @@ This package is a thin wrapper around the [NVIDIA NeMo Anonymizer library](https
-- **[Quick Start](quickstart.md)** - - --- - - Install the plugin, configure inference, and run your first preview and job. - - **[Tutorials](tutorials/index.md)** --- diff --git a/docs/anonymizer/quickstart.md b/docs/anonymizer/quickstart.md deleted file mode 100644 index e98f563585..0000000000 --- a/docs/anonymizer/quickstart.md +++ /dev/null @@ -1,277 +0,0 @@ - -# Quick Start - -This guide walks through previewing and running an Anonymizer job on {{platform_name}}. - -## Prerequisites - -- Access to a {{platform_name}} deployment with the `anonymizer` plugin service enabled. -- An API key for a model provider used by the Anonymizer pipeline. - -## Step 1: Install the Plugin - -Follow the [Setup guide](../get-started/setup.md) to install {{platform_name}} and complete `nemo setup`. From a repo checkout, run `uv sync` at the repo root; the root workspace includes the Anonymizer plugin, so no separate editable plugin install step is needed. `nemo services run` then picks up the plugin automatically and mounts `/apis/anonymizer/...` on the gateway. - -Verify the CLI is registered: - -```bash -nemo anonymizer --help -``` - -You should see `validate`, `preview`, and `run` command groups. - -## Step 2: Initialize the SDK - -```python -import os -from nemo_platform import NeMoPlatform - -base_url = os.environ.get("NMP_BASE_URL", "http://localhost:8080") -WORKSPACE = os.environ.get("NMP_WORKSPACE", "default") -sdk = NeMoPlatform(base_url=base_url, workspace=WORKSPACE) -anonymizer = sdk.anonymizer -``` - -## Step 3: Configure Inference - -Anonymizer routes inference through the [Inference Gateway service](../run-inference/about.md). You need a model provider configured before running anything that uses `model_configs`. - -`nemo setup` walks you through creating a provider secret and registering an Inference Gateway provider as part of the install flow. If you skipped that step or want to add another provider, re-run `nemo setup` — see the [Setup guide](../get-started/setup.md) for details. - ---8<-- "_snippets/nvidia-build-model-provider.md" - -## Step 4: Upload an Input Fileset - -Create a small CSV containing PII and upload it to a fileset: - -```python -import os -import tempfile -from pathlib import Path - -from nemo_platform._exceptions import ConflictError - -WORKSPACE = os.environ.get("NMP_WORKSPACE", "default") -FILESET = "anonymizer-inputs" -INPUT_FILENAME = "anonymizer-input.csv" - -with tempfile.NamedTemporaryFile("w", suffix=".csv", delete=False) as f: - f.write( - "id,biography\n" - "1,Alice Johnson lives in Seattle and works at NVIDIA.\n" - "2,Bob Smith can be reached at bob.smith@example.com.\n" - ) - input_path = Path(f.name) - -try: - sdk.files.filesets.create( - name=FILESET, - workspace=WORKSPACE, - description="Anonymizer input files", - ) -except ConflictError: - pass # already exists - -sdk.files.upload( - local_path=str(input_path), - fileset=FILESET, - workspace=WORKSPACE, - remote_path=INPUT_FILENAME, -) -``` - -The plugin accepts three input source forms: - -- Local path (local execution only): `/tmp/anonymizer-input.csv` -- HTTP(S) URL: `https://.../input.csv` -- Fileset reference: `anonymizer-inputs#anonymizer-input.csv`, `default/anonymizer-inputs#anonymizer-input.csv`, or `fileset://default/anonymizer-inputs#anonymizer-input.csv` - -## Step 5: Preview Anonymization - -Preview streams a small anonymized sample so you can iterate on the config without running a full job. Build a `PreviewRequest` and call `anonymizer.preview`: - -```python -import os -from anonymizer.config.anonymizer_config import AnonymizerConfig -from anonymizer.config.replace_strategies import Redact -from data_designer.config import ModelConfig -from nemo_anonymizer_plugin.app.input import AnonymizerInputSpec -from nemo_anonymizer_plugin.app.task_config import PreviewRequest - -MODEL_PROVIDER = os.environ.get("NMP_ANON_PROVIDER", "nvidia-build") - -config = AnonymizerConfig( - replace=Redact(format_template="[REDACTED_{label}]"), -) - -model_configs = [ - ModelConfig(alias="gliner-pii-detector", provider=MODEL_PROVIDER, model="nvidia/gliner-pii"), - ModelConfig(alias="gpt-oss-120b", provider=MODEL_PROVIDER, model="openai/gpt-oss-120b"), - ModelConfig(alias="nemotron-30b-thinking", provider=MODEL_PROVIDER, model="nvidia/nemotron-3-nano-30b-a3b"), -] - -request = PreviewRequest( - config=config, - data=AnonymizerInputSpec( - source=f"fileset://{WORKSPACE}/{FILESET}#{INPUT_FILENAME}", - text_column="biography", - id_column="id", - ), - model_configs=model_configs, - num_records=2, -) - -preview = anonymizer.preview(request) - -preview.dataset # pandas DataFrame of anonymized records -preview.trace_dataset # detection trace -preview.failed_records # list of per-record failures (usually empty) -preview.display_record(0) # render a record with entity highlights -``` - -`preview.dataset` is a regular pandas DataFrame, so you can persist it with `to_csv` or `to_parquet`. - -??? "Run preview from the CLI instead" - The same flow is available from the CLI. Write the spec to YAML: - - ```python - import yaml - from pathlib import Path - - preview_spec_path = Path("/tmp/anonymizer-preview.yaml") - preview_spec_path.write_text(yaml.safe_dump(request.model_dump(mode="json", exclude_none=True))) - ``` - - Then run either of: - - ```bash - nemo anonymizer preview run \ - --spec-file /tmp/anonymizer-preview.yaml \ - --workspace "${NMP_WORKSPACE:-default}" - - nemo anonymizer preview submit \ - --spec-file /tmp/anonymizer-preview.yaml \ - --workspace "${NMP_WORKSPACE:-default}" \ - --base-url "${NMP_BASE_URL:-http://localhost:8080}" - ``` - - The CLI streams newline-delimited JSON frames (`preview_dataset`, `trace_dataset`, `failed_records`, ...) to stdout. See the [preview tutorial](tutorials/preview.md) for the frame schema and `jq` recipes. - -!!! note - `anonymizer.preview` calls the plugin service, so it rejects local file paths in `data.source` and requires `model_configs`. The fileset reference and `model_configs` in the example above satisfy both constraints. - -## Step 6: Run a Full Job - -When the preview looks correct, run the full pipeline. The `anonymizer.run` job can execute either locally in the CLI process (`run run`) or on the {{platform_name}} Jobs worker (`run submit` / `sdk.anonymizer.run()`). - -Build an `AnonymizerRequest`: - -```python -from nemo_anonymizer_plugin.app.task_config import AnonymizerRequest - -run_request = AnonymizerRequest( - config=config, - data=AnonymizerInputSpec( - source=f"fileset://{WORKSPACE}/{FILESET}#{INPUT_FILENAME}", - text_column="biography", - id_column="id", - ), - model_configs=model_configs, -) -``` - -**Option A — submit to the Jobs worker:** - -```python -job = sdk.anonymizer.run(run_request, wait_until_done=True) -results = job.download_artifacts() - -dataset = results.load_dataset() -print(dataset.head()) -print(f"records={len(dataset)} failures={len(results.load_failed_records())}") -``` - -`sdk.anonymizer.run()` returns an `AnonymizerJobResource`. `wait_until_done=True` blocks until the job reaches a terminal state; `download_artifacts()` fetches the job artifacts and returns an `AnonymizerJobResults` for in-memory access. See [SDK Resources](sdk-resources.md) for the full surface. - -The CLI equivalent submits the same spec. First write it to YAML: - -```python -import yaml -from pathlib import Path - -run_spec_path = Path("/tmp/anonymizer-run.yaml") -run_spec_path.write_text(yaml.safe_dump(run_request.model_dump(mode="json", exclude_none=True))) -``` - -Then submit it: - -```bash -nemo anonymizer run submit \ - --spec-file /tmp/anonymizer-run.yaml \ - --workspace "${NMP_WORKSPACE:-default}" \ - --base-url "${NMP_BASE_URL:-http://localhost:8080}" -``` - -Track the submitted job with `nemo jobs get-status --workspace "${NMP_WORKSPACE:-default}"` and `nemo jobs get-logs --workspace "${NMP_WORKSPACE:-default}"`. - -**Option B — run locally in the CLI process:** - -```python -import yaml -from pathlib import Path - -spec_path = Path("/tmp/anonymizer-run.yaml") -spec_path.write_text(yaml.safe_dump(run_request.model_dump(mode="json", exclude_none=True))) -``` - -```bash -nemo anonymizer run run --spec-file /tmp/anonymizer-run.yaml -``` - -The CLI prints `{"exit_code": 0}` on success and logs the artifact directory (`file://.../persistent/results/artifacts`) to stderr. The directory contains: - -- `dataset.parquet`: anonymized output. -- `trace.parquet`: detection trace. -- `metadata.json`: run metadata. -- `failed_records.json`: per-record failures, only when there were failures. - -!!! note "Differences between `run run` and `run submit`" - `run submit` rejects local file paths in `data.source` (use a fileset reference or `http(s)` URL) and requires explicit `model_configs` referencing Inference Gateway providers. `run run` accepts local paths and can run without `model_configs` when the library defaults suffice. - -## Step 7: Inspect Artifacts - -For Option A (`run submit`), the `AnonymizerJobResults` returned by `download_artifacts()` already loads parquet files lazily — `results.load_dataset()`, `results.load_trace()`, and `results.load_failed_records()` return pandas DataFrames / lists. - -For Option B (`run run`), load the parquet files directly from the local artifact directory: - -```python -from pathlib import Path - -import pandas as pd - -ARTIFACTS_DIR = Path("/path/to/persistent/results/artifacts") # from the stderr log - -dataset = pd.read_parquet(ARTIFACTS_DIR / "dataset.parquet", dtype_backend="pyarrow") -trace = pd.read_parquet(ARTIFACTS_DIR / "trace.parquet", dtype_backend="pyarrow") - -print(dataset.head()) -``` - -The trace dataset (and the dataset itself for `annotate` / `substitute` strategies) contains pyarrow-backed `struct>` columns. Use `pyarrow.parquet.read_table(...).to_pylist()` if you need plain Python `dict`/`list` values for JSON output. - -## Troubleshooting - -| Problem | Cause | Solution | -|----------------------------------------------------|-------------------------------------------------------------|---------------------------------------------------------------------------------------------------------| -| `nemo anonymizer preview submit` returns 404 | The `anonymizer` plugin service isn't mounted on the gateway | Confirm `uv sync` ran successfully at the repo root and re-run `nemo services run` so the plugin is discovered. See [Step 1](#step-1-install-the-plugin). | -| `model_configs are required for remote execution` | `anonymizer.preview` / `preview submit` requires explicit `model_configs` | Add `model_configs` referencing an Inference Gateway provider. | -| `Input source ... is a local path` | Plugin-service execution rejects local paths | Use an `http(s)` URL or a fileset reference. | -| `Fileset input ... must resolve to a .csv or .parquet file` | Fileset path is a directory or wrong extension | Point the `#` fragment at a single `.csv` or `.parquet` file. | -| `provider not found` | Inference provider missing | Inspect or create the provider using the inference/model-provider docs, then reference it in `model_configs`. | - -## Next Steps - -- **Tutorials:** Walk through preview and run flows in detail in the [tutorials](tutorials/index.md). -- **SDK reference:** See [SDK Resources](sdk-resources.md) for the `anonymizer` accessor, preview result, and job result types. -- **CLI reference:** See [CLI Reference](cli.md) for spec-file fields and command flags. -- **Library docs:** Detection, replacement strategy parameters, and rewrite mode are documented in the [open-source library](https://github.com/NVIDIA-NeMo/Anonymizer/tree/main/docs). diff --git a/docs/anonymizer/tutorials/index.md b/docs/anonymizer/tutorials/index.md index 0ba4ffe897..18a7057c06 100644 --- a/docs/anonymizer/tutorials/index.md +++ b/docs/anonymizer/tutorials/index.md @@ -76,11 +76,65 @@ When using Anonymizer as a {{platform_name}} service: ## Prerequisites -Before starting these tutorials, complete the [Quick Start](../quickstart.md) to: +Complete [Setup](../../get-started/setup.md) to install {{platform_name}}, run `nemo services run`, and configure an inference provider. The root workspace includes the Anonymizer plugin, so `nemo services run` discovers it automatically and mounts `/apis/anonymizer/...` on the gateway — no separate plugin install step is needed. Verify the CLI is registered: -- Install the plugin and verify the `nemo anonymizer` CLI. -- Configure an inference provider used in `model_configs`. -- Create a fileset and upload a CSV containing PII. +```bash +nemo anonymizer --help +``` + +You should see `validate`, `preview`, and `run` command groups. + +These tutorials route inference through an [Inference Gateway](../../run-inference/about.md) provider, so a {{platform_name}} cluster must be running before you preview or run a job. The examples reference the default NVIDIA Build provider created during setup. + +--8<-- "_snippets/nvidia-build-model-provider.md" + +### Upload an Input Fileset + +`sdk.anonymizer.preview`, `preview submit`, and `run submit` reject local file paths, so the tutorials read from a fileset. Create a small CSV containing PII and upload it to a fileset named `anonymizer-inputs`: + +```python +import os +import tempfile +from pathlib import Path + +from nemo_platform import NeMoPlatform +from nemo_platform._exceptions import ConflictError + +WORKSPACE = os.environ.get("NMP_WORKSPACE", "default") +FILESET = "anonymizer-inputs" +INPUT_FILENAME = "anonymizer-input.csv" + +sdk = NeMoPlatform( + base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), + workspace=WORKSPACE, +) + +with tempfile.NamedTemporaryFile("w", suffix=".csv", delete=False) as f: + f.write( + "id,biography\n" + "1,Alice Johnson lives in Seattle and works at NVIDIA.\n" + "2,Bob Smith can be reached at bob.smith@example.com.\n" + ) + input_path = Path(f.name) + +try: + sdk.files.filesets.create( + name=FILESET, + workspace=WORKSPACE, + description="Anonymizer input files", + ) +except ConflictError: + pass # already exists + +sdk.files.upload( + local_path=str(input_path), + fileset=FILESET, + workspace=WORKSPACE, + remote_path=INPUT_FILENAME, +) +``` + +The tutorials reference this file with `fileset://{WORKSPACE}/anonymizer-inputs#anonymizer-input.csv`. ## Tutorials diff --git a/docs/anonymizer/tutorials/preview.md b/docs/anonymizer/tutorials/preview.md index d5a9a9d41a..6814c9728e 100644 --- a/docs/anonymizer/tutorials/preview.md +++ b/docs/anonymizer/tutorials/preview.md @@ -7,9 +7,11 @@ For detection and replacement strategy details, see the [open-source library doc ## Prerequisites -- The Anonymizer plugin installed and the `nemo anonymizer` CLI available. See the [Quick Start](../quickstart.md). +Complete the [tutorials prerequisites](index.md#prerequisites), which cover: + +- A running {{platform_name}} cluster with the `nemo anonymizer` CLI available (see [Setup](../../get-started/setup.md)). - An inference provider configured (default examples use `nvidia-build`). -- A fileset named `anonymizer-inputs` with `anonymizer-input.csv` uploaded (created in the Quick Start). +- A fileset named `anonymizer-inputs` with `anonymizer-input.csv` uploaded. ## What `preview` Does @@ -172,7 +174,7 @@ jq -R 'fromjson? | select(.kind == "preview_dataset") | .records' \ /tmp/anonymizer-preview.ndjson ``` -If `preview submit` returns 404 against the gateway, the plugin service isn't mounted. Confirm the plugin is installed and restart `nemo services run`; see [Quick Start — Step 1](../quickstart.md#step-1-install-the-plugin). +If `preview submit` returns 404 against the gateway, the plugin service isn't mounted. Restart `nemo services run` so the plugin is discovered and remounts `/apis/anonymizer/...`; see [Setup](../../get-started/setup.md). ## Input Source Forms diff --git a/docs/anonymizer/tutorials/run.md b/docs/anonymizer/tutorials/run.md index 16f3c52626..0663ffa615 100644 --- a/docs/anonymizer/tutorials/run.md +++ b/docs/anonymizer/tutorials/run.md @@ -7,9 +7,11 @@ For detection, rewrite, and replacement strategy details, see the [open-source l ## Prerequisites -- The Anonymizer plugin installed and the `nemo anonymizer` CLI available. See the [Quick Start](../quickstart.md). +Complete the [tutorials prerequisites](index.md#prerequisites), which cover: + +- A running {{platform_name}} cluster with the `nemo anonymizer` CLI available (see [Setup](../../get-started/setup.md)). - An inference provider configured (default examples use `nvidia-build`). -- A fileset named `anonymizer-inputs` with `anonymizer-input.csv` uploaded (created in the Quick Start). +- A fileset named `anonymizer-inputs` with `anonymizer-input.csv` uploaded. ## What `run` Does diff --git a/mkdocs.yml b/mkdocs.yml index 2f58792dff..fe7465f4fb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -120,6 +120,7 @@ plugins: get-started/quickstart.md: get-started/setup.md get-started/getting-started.md: index.md generate-synthetic-data/index.md: data-designer/index.md + anonymizer/quickstart.md: anonymizer/tutorials/index.md - swagger-ui-tag: docExpansion: list @@ -334,7 +335,6 @@ nav: - SDK Resources: safe-synthesizer/sdk-resources.md - Anonymize Data: - About: anonymizer/index.md - - Quickstart: anonymizer/quickstart.md - Tutorials: - Overview: anonymizer/tutorials/index.md - Preview a Config: anonymizer/tutorials/preview.md diff --git a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md index 019515134f..a8a963a096 100644 --- a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md +++ b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md @@ -8,7 +8,7 @@ argument-hint: [describe the dataset and how PII should be handled] Do not explore the workspace first. The workflow's Learn step gives you everything you need. -**Source of truth.** For anything you're unsure about, prefer the in-tree CLI docs over your memory: `docs/anonymizer/index.md`, `docs/anonymizer/quickstart.md`, `docs/anonymizer/cli.md`, and `docs/anonymizer/tutorials/{preview,run}.md`. The [NVIDIA NeMo Anonymizer library docs](https://github.com/NVIDIA-NeMo/Anonymizer/tree/main/docs) own detection, replacement strategy parameters, and rewrite-mode semantics. +**Source of truth.** For anything you're unsure about, prefer the in-tree CLI docs over your memory: `docs/anonymizer/index.md`, `docs/anonymizer/cli.md`, and `docs/anonymizer/tutorials/{index,preview,run}.md`. The [NVIDIA NeMo Anonymizer library docs](https://github.com/NVIDIA-NeMo/Anonymizer/tree/main/docs) own detection, replacement strategy parameters, and rewrite-mode semantics. # Goal @@ -56,7 +56,7 @@ Read **only** the workflow file that matches the selected mode, then follow it: # Troubleshooting - **`nemo anonymizer` CLI not found:** The plugin isn't installed in this environment. From the repo root, run `uv sync`; the root workspace includes the Anonymizer plugin. Confirm with `nemo anonymizer --help`. Do not install anything without the user's permission. -- **`nemo anonymizer preview submit` returns 404:** The plugin service isn't mounted on the gateway. `nemo setup` does not auto-mount it. Re-run `nemo services run` (no `--services` flag) and verify the routes show up under `/apis/anonymizer/` in the OpenAPI listing. See `docs/anonymizer/quickstart.md` Step 1. +- **`nemo anonymizer preview submit` returns 404:** The plugin service isn't mounted on the gateway. `nemo setup` does not auto-mount it. Re-run `nemo services run` (no `--services` flag) and verify the routes show up under `/apis/anonymizer/` in the OpenAPI listing. See `docs/anonymizer/tutorials/index.md` Prerequisites. - **`model_configs are required for remote execution`:** `preview submit` and `run submit` go through plugin-service / Jobs paths. Add `model_configs` referencing an Inference Gateway provider; use the inference/model-provider docs or skill for provider discovery. - **`Input source ... is a local path`:** Plugin-service execution rejects local paths. Either upload the file to a fileset, use an `http(s)` URL, or switch to `preview run` / `run run` (local execution). - **`Fileset input ... must resolve to a .csv or .parquet file`:** The `#` fragment points at a directory or a non-CSV/Parquet file. Point it at a single file. diff --git a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md index 127cb84409..3ffce5d102 100644 --- a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md +++ b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md @@ -2,7 +2,7 @@ The user has signaled they don't want to answer questions. Make defensible decisions and keep moving. Do **not** run the full `run` job autonomously — finalize with a one-line command the user can launch. -Source of truth for defaults: `docs/anonymizer/quickstart.md`. If anything below conflicts with the docs, the docs win. +Source of truth for defaults: `docs/anonymizer/tutorials/index.md` and `docs/anonymizer/tutorials/{preview,run}.md`. If anything below conflicts with the docs, the docs win. 1. **Resolve CLI command** — Run `command -v nemo 2>/dev/null || (test -x .venv/bin/nemo && realpath .venv/bin/nemo) || echo CLI_NOT_FOUND`. - If the output is `CLI_NOT_FOUND`, STOP and follow the Troubleshooting section in SKILL.md. diff --git a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.md b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.md index f3df74f074..f65bc5f87a 100644 --- a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.md +++ b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.md @@ -2,7 +2,7 @@ This is an interactive, iterative anonymization design process. Do not disengage from the loop unless the user says they are satisfied. -Source of truth for this workflow: `docs/anonymizer/quickstart.md`, `docs/anonymizer/tutorials/preview.md`, and `docs/anonymizer/tutorials/run.md`. Defer to them if the CLI flags or capabilities here look out of date. +Source of truth for this workflow: `docs/anonymizer/tutorials/index.md`, `docs/anonymizer/tutorials/preview.md`, and `docs/anonymizer/tutorials/run.md`. Defer to them if the CLI flags or capabilities here look out of date. 1. **Resolve CLI command** — Run `command -v nemo 2>/dev/null || (test -x .venv/bin/nemo && realpath .venv/bin/nemo) || echo CLI_NOT_FOUND`. - If the output is a path, use ` anonymizer` as the command prefix for all `nemo anonymizer …` invocations in this workflow.