chore: remove run (local execution) CLI verbs across all services - #1228
chore: remove run (local execution) CLI verbs across all services#1228ironcommit wants to merge 3 commits into
run (local execution) CLI verbs across all services#1228Conversation
ironcommit
changed the title
run (local execution) CLI verbs across all services
📝 WalkthroughWalkthroughThe change removes local execution from plugin CLIs, schedulers, SDK resources, and Safe Synthesizer workflows. It standardizes platform submission through ChangesRemote execution transition
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 20
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
docs/evaluator/metrics/model-configuration.mdx (1)
188-224: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winAdd
RunConfigOnlineModelandprompt_templateto the inlineModelexample.A model target requires both values. Without them, the durable job fails validation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/evaluator/metrics/model-configuration.mdx` around lines 188 - 224, Update the inline Model example to include the required RunConfigOnlineModel configuration and prompt_template value, ensuring the submit call passes the online run config and the Model includes its prompt template. Keep the existing endpoint resolution and job flow unchanged.plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md (1)
74-84: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winRemove local paths from the source example.
Line 80 says that
data.sourceaccepts a local path. Line 42 states that submission accepts only HTTP(S) URLs or fileset references. Remove “local path” from the comment.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md` around lines 74 - 84, Update the data.source comment in the anonymizer preview submit example to remove the local-path option, leaving only HTTP(S) URLs and fileset references as supported sources.docs/evaluator/tutorials/run-llm-judge-evaluation.mdx (1)
306-321: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDelete
quick_test_jobduring cleanup.This flow creates
quick_test_job, but Lines 652-656 delete onlyjob_v1andjob_v2. The remaining job can prevent workspace deletion at Lines 669-670.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/evaluator/tutorials/run-llm-judge-evaluation.mdx` around lines 306 - 321, Update the cleanup section to also delete quick_test_job, alongside job_v1 and job_v2, before workspace deletion. Ensure the quick_test_job returned by evaluator.submit is cleaned up after its result is retrieved.plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.md (1)
11-16: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winVerify platform readiness before dependent operations.
Follow
SETUP.md, check an existing instance, and fail on/health/readybefore model-fileset setup or job submission.curl -sdoes not enforce readiness. The diagnose workflow checks availability after setup.
plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.md#L11-L16: add an enforced readiness check beforesetup_model_filesets.py.plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/diagnose.md#L44-L50: move the Files API readiness check before model-fileset setup.As per coding guidelines: “Before operations requiring a running NeMo Platform, follow SETUP.md and verify service readiness.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.md` around lines 11 - 16, Ensure both workflow sites verify NeMo Platform readiness before dependent operations: in plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.md lines 11-16, follow SETUP.md, check an existing instance, and replace the non-enforcing curl health check with an enforced /health/ready check before setup_model_filesets.py; in plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/diagnose.md lines 44-50, move the Files API readiness check before model-fileset setup while preserving failure on an unavailable or unready service.Source: Coding guidelines
🧹 Nitpick comments (2)
plugins/nemo-evaluator/tests/test_inline_bundle_execution.py (1)
55-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract this fixture; it is duplicated across three test modules.
_job_contexthere is byte-identical in behavior to_make_job_contextinplugins/nemo-evaluator/tests/test_evaluate_job.py(Lines 103-112), andtest_agent_evaluate.pydefines a third copy. Move it to a sharedconftest.pyfixture.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/tests/test_inline_bundle_execution.py` around lines 55 - 63, Extract the duplicated _job_context setup into a shared pytest fixture in conftest.py, consolidating it with _make_job_context from test_evaluate_job.py and the equivalent helper in test_agent_evaluate.py. Update all three test modules to use the shared fixture and remove their local helper definitions while preserving the existing JobContext, StoragePaths, directory creation, and LocalJobResults behavior.plugins/nemo-safe-synthesizer/tests/unit/test_cli.py (1)
8-13: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winInvoke
run-localdirectly in this test.A hidden registered command passes this help-only check. Invoke
["run-local"]and assert a nonzero exit code withNo such command.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-safe-synthesizer/tests/unit/test_cli.py` around lines 8 - 13, Update test_cli_exposes_runtime_but_not_removed_local_command to invoke the CLI with ["run-local"] instead of only requesting help, then assert a nonzero exit code and that the output contains "No such command".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/anonymizer/cli.mdx`:
- Around line 14-17: Complete the remote-only documentation migration: in
docs/anonymizer/cli.mdx lines 14-17 keep one row each for preview submit and run
submit, removing local behavior; update lines 39-42 and 60-71 to describe and
source only the remote preview path; update lines 98-114 to keep run submit
remote-only; remove duplicate command requirements in lines 139-152. In
docs/anonymizer/tutorials/preview.mdx lines 179-184, update the preview field
reference to remove local-path support and align with the remote-only execution
path.
In `@docs/data-designer/cli.mdx`:
- Around line 41-42: Remove the obsolete local execution entries for preview
submit and create submit from the CLI documentation, including the corresponding
local sections around the referenced rows. Retain only the rows describing
submission through NeMo Services, so each command has a single service-based
execution model.
In `@docs/data-designer/tutorials/seeding.mdx`:
- Around line 259-265: Update docs/data-designer/tutorials/seeding.mdx at lines
259-265 by deleting the duplicate submit command block, removing “locally” from
the labels at lines 256 and 262, and removing the local-execution claim at line
247. In docs/data-designer/tutorials/basics.mdx at line 205, rename “Local CLI
Execution” to “CLI Submission,” remove the “run locally through the CLI” option
at line 194, and remove “local” from line 285.
In `@docs/evaluator/metrics/llm-as-a-judge.mdx`:
- Line 21: Update the execution-mode table around the “Platform job” row so it
no longer duplicates the durable-job mode on the adjacent row. Either
consolidate the entries into one platform-job mode or restore the intended
standalone SDK mode, while keeping each mode’s description and example accurate.
In `@docs/evaluator/metrics/remote.mdx`:
- Around line 24-25: Remove the duplicate execution-mode row from the execution
modes table, keeping a single entry for durable platform jobs with the existing
evaluator.submit(metric=metric, dataset=dataset) example.
In `@docs/evaluator/tutorials/run-llm-judge-evaluation.mdx`:
- Line 63: Update the tutorial’s remaining local-execution guidance around the
resource description, local execution section, and testing recommendation to
describe submitted evaluator jobs only. Remove references to running metrics
locally and direct users toward the durable submit flow via
evaluator.submit(...), while preserving the surrounding evaluation instructions.
In `@docs/safe-synthesizer/about/host-local-development.mdx`:
- Around line 53-57: Rename the “Related Topics” section to “Next Steps” in the
host-local-development documentation. Replace the source-tree Plugin README
reference with its published canonical Fern documentation URL, or remove that
bullet if no canonical page exists; keep the existing published links unchanged.
- Around line 8-12: Update the host-local development page before the
introductory content and Runtime Commands heading to list prerequisites covering
platform access, plugin installation, and required runtime setup. Keep the
existing explanation and command documentation unchanged after the new
prerequisites section.
In `@docs/safe-synthesizer/getting-started.mdx`:
- Around line 23-48: Update the “Submit Jobs” section to use the documentation’s
paired Python/CLI tab format. Keep the existing Python SDK example, and add a
CLI tab showing the supported runtime setup and inspection flow while clearly
stating that remote-only CLI submission is unavailable; if a CLI submit command
exists, document that command instead.
In `@docs/studio/data-designer-build.mdx`:
- Line 156: Update the “Preview logs” description to state that sampled output
is streamed while the submitted preview executes, rather than implying streaming
ends when submission is accepted.
In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md`:
- Around line 100-105: Remove stale local-execution and generated `run`
documentation across the submission-only flow: in
packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md:100-105
update the CLI surface to document only `submit` and `explain`; in
docs/evaluator/metrics/rag.mdx:61-68 and its referenced areas near lines 82,
124, and 914 remove local `run` references; in
docs/evaluator/metrics/remote.mdx:254-257 and referenced areas near lines 241
and 260 remove local `run` and local secret-resolution wording; in
docs/evaluator/metrics/results.mdx:34-42 rename “Get Results from a Local Run”;
and in packages/nemo_platform_plugin/src/nemo_platform_plugin/function.py:9-11
remove local-CLI references from the `run()` and `run_signature()` docstrings.
In `@packages/nemo_platform_plugin/tests/test_commands.py`:
- Around line 255-260: Update the test setup around the runner invocation to
import os and clear relevant NMP_* environment variables before executing the
command, ensuring _resolve_submit_auth_headers cannot read ambient
authentication configuration and the exact kwargs["headers"] assertion remains
deterministic.
In `@plugins/nemo-anonymizer/openapi/openapi.yaml`:
- Line 1288: Regenerate the remote-only OpenAPI contract and SDK so the source
schema accepts only URLs or filesets, and model_configs is required in
AnonymizerRequest and PreviewRequest. Update the corresponding OpenAPI data
models to match the submitted-execution contract before running the repository’s
specification and SDK generation workflow.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/model-configs.md`:
- Line 52: Remove or revise the “Local default-everything preview” entry in the
model configuration documentation so it no longer claims that preview submit or
run submit work without model_configs. Keep the documented requirement that both
commands require model_configs, and update any remaining wording in that entry
to describe only supported local behavior.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md`:
- Around line 21-22: Update the CLI usage guidance for anonymizer.preview and
anonymizer.run to require following SETUP.md, setting
NMP_BASE_URL=http://localhost:8080, checking for an existing instance, and
verifying /health/ready before submitting either command.
- Line 37: Update the iteration guidance in SKILL.md so it instructs users to
run “nemo anonymizer preview submit” first, followed by the full-job command
“nemo anonymizer run submit”; remove the duplicated preview command.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md`:
- Around line 22-23: Update the Build and Preview workflow steps to require
local platform readiness before submission: set
NMP_BASE_URL=http://localhost:8080, follow SETUP.md, and verify that
/health/ready succeeds before running nemo anonymizer preview submit. Do not
allow preview submission based only on /openapi.json.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.md`:
- Around line 10-11: Update the workflow setup before the plugin-service probe
and every subsequent nemo command: set NMP_BASE_URL to http://localhost:8080 and
verify that /health/ready returns successfully. Stop and instruct the user to
resolve service readiness if the check fails, then retain the existing
openapi.json mounting check and input-source flow.
In `@plugins/nemo-auditor/README.md`:
- Around line 47-54: Align the CLI documentation in the README with the
registered command described in the auditor skill. Verify whether `nemo auditor
audit submit` is supported; if registered, document its usage in the README and
update the statement that no plugin-specific CLI command exists, otherwise
remove the command references from `SKILL.md`.
In
`@plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.py`:
- Around line 396-400: Update main to normalize a None argv to sys.argv[1:]
before the rejection check, then preserve the existing SystemExit behavior for
any supplied arguments and call run_from_env only when the normalized list is
empty.
---
Outside diff comments:
In `@docs/evaluator/metrics/model-configuration.mdx`:
- Around line 188-224: Update the inline Model example to include the required
RunConfigOnlineModel configuration and prompt_template value, ensuring the
submit call passes the online run config and the Model includes its prompt
template. Keep the existing endpoint resolution and job flow unchanged.
In `@docs/evaluator/tutorials/run-llm-judge-evaluation.mdx`:
- Around line 306-321: Update the cleanup section to also delete quick_test_job,
alongside job_v1 and job_v2, before workspace deletion. Ensure the
quick_test_job returned by evaluator.submit is cleaned up after its result is
retrieved.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md`:
- Around line 74-84: Update the data.source comment in the anonymizer preview
submit example to remove the local-path option, leaving only HTTP(S) URLs and
fileset references as supported sources.
In
`@plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.md`:
- Around line 11-16: Ensure both workflow sites verify NeMo Platform readiness
before dependent operations: in
plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.md
lines 11-16, follow SETUP.md, check an existing instance, and replace the
non-enforcing curl health check with an enforced /health/ready check before
setup_model_filesets.py; in
plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/diagnose.md
lines 44-50, move the Files API readiness check before model-fileset setup while
preserving failure on an unavailable or unready service.
---
Nitpick comments:
In `@plugins/nemo-evaluator/tests/test_inline_bundle_execution.py`:
- Around line 55-63: Extract the duplicated _job_context setup into a shared
pytest fixture in conftest.py, consolidating it with _make_job_context from
test_evaluate_job.py and the equivalent helper in test_agent_evaluate.py. Update
all three test modules to use the shared fixture and remove their local helper
definitions while preserving the existing JobContext, StoragePaths, directory
creation, and LocalJobResults behavior.
In `@plugins/nemo-safe-synthesizer/tests/unit/test_cli.py`:
- Around line 8-13: Update
test_cli_exposes_runtime_but_not_removed_local_command to invoke the CLI with
["run-local"] instead of only requesting help, then assert a nonzero exit code
and that the output contains "No such command".
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 05868de5-b0a8-42e8-a4e4-66b17fdb3cdd
⛔ Files ignored due to path filters (1)
sdk/python/nemo-platform/src/nemo_platform/cli/app.pyis excluded by!sdk/**
📒 Files selected for processing (136)
docs/agents/optimization.mdxdocs/anonymizer/cli.mdxdocs/anonymizer/index.mdxdocs/anonymizer/sdk-resources.mdxdocs/anonymizer/tutorials/index.mdxdocs/anonymizer/tutorials/preview.mdxdocs/anonymizer/tutorials/run.mdxdocs/auditor/configs/index.mdxdocs/auditor/index.mdxdocs/auditor/sdk-resources.mdxdocs/auditor/targets/index.mdxdocs/auditor/targets/inference-gateway.mdxdocs/auditor/tutorials/index.mdxdocs/auditor/tutorials/run-audit-locally.mdxdocs/data-designer/cli.mdxdocs/data-designer/execution-modes.mdxdocs/data-designer/index.mdxdocs/data-designer/migration.mdxdocs/data-designer/sdk-resources.mdxdocs/data-designer/tutorials/basics.mdxdocs/data-designer/tutorials/index.mdxdocs/data-designer/tutorials/seeding.mdxdocs/evaluator/agent-eval/index.mdxdocs/evaluator/index.mdxdocs/evaluator/metrics/agent-configuration.mdxdocs/evaluator/metrics/agentic.mdxdocs/evaluator/metrics/index.mdxdocs/evaluator/metrics/llm-as-a-judge.mdxdocs/evaluator/metrics/manage-metrics.mdxdocs/evaluator/metrics/model-configuration.mdxdocs/evaluator/metrics/rag.mdxdocs/evaluator/metrics/remote.mdxdocs/evaluator/metrics/results.mdxdocs/evaluator/metrics/similarity.mdxdocs/evaluator/sdk-resources.mdxdocs/evaluator/test_doc_examples.pydocs/evaluator/tutorials/run-llm-judge-evaluation.mdxdocs/safe-synthesizer/about/host-local-development.mdxdocs/safe-synthesizer/about/index.mdxdocs/safe-synthesizer/about/jobs.mdxdocs/safe-synthesizer/about/reference.mdxdocs/safe-synthesizer/getting-started.mdxdocs/safe-synthesizer/tutorials/safe-synthesizer-101.mdxdocs/studio/data-designer-build.mdxpackages/nemo_platform_ext/src/nemo_platform_ext/cli/app.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/cli.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/cli_renderer.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/cli_state.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/commands.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/discovery.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/docs/JOB.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/function.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/job.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/job_context.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/job_results.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/run_dependencies.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/scheduler.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/tasks/dispatcher.pypackages/nemo_platform_plugin/tests/test_cli_hooks.pypackages/nemo_platform_plugin/tests/test_cli_renderer.pypackages/nemo_platform_plugin/tests/test_commands.pypackages/nemo_platform_plugin/tests/test_dispatcher.pypackages/nemo_platform_plugin/tests/test_job_context.pypackages/nemo_platform_plugin/tests/test_run_dependencies.pypackages/nemo_platform_plugin/tests/test_scheduler.pypackages/nmp_customization_common/src/nmp/customization_common/cli/overrides.pypackages/nmp_customization_common/src/nmp/customization_common/contributor/base.pyplugins/example-plugin/src/nemo_example_plugin/jobs/say_hello.pyplugins/example-plugin/tests/test_say_hello_job.pyplugins/nemo-agents/src/nemo_agents_plugin/jobs/evaluate_agent.pyplugins/nemo-agents/src/nemo_agents_plugin/jobs/fileset_io.pyplugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.mdplugins/nemo-agents/tests/unit/test_utils.pyplugins/nemo-anonymizer/README.mdplugins/nemo-anonymizer/openapi/openapi.yamlplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/cli.pyplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/functions/preview.pyplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/inputs.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/model-configs.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/preview-review.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/replace-strategies.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/rewrite-mode.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.mdplugins/nemo-anonymizer/tests/unit/test_cli.pyplugins/nemo-anonymizer/tests/unit/test_run_job.pyplugins/nemo-auditor/README.mdplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/sdk.pyplugins/nemo-auditor/src/nemo_auditor/skills/auditor/SKILL.mdplugins/nemo-auditor/tests/test_sdk_resources.pyplugins/nemo-automodel/src/nemo_automodel_plugin/cli/inputs.pyplugins/nemo-automodel/tests/test_cli.pyplugins/nemo-customizer/src/nemo_customizer/skills/nemo-customizer/references/troubleshooting.mdplugins/nemo-data-designer/README.mdplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/inputs.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/main.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/renderers.pyplugins/nemo-data-designer/tests/integration/test_preview_local_cli.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/metric_refs.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/fs_utils.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/resources.pyplugins/nemo-evaluator/src/nemo_evaluator/task_refs.pyplugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.pyplugins/nemo-evaluator/tests/integration/test_harbor_plugin_run.pyplugins/nemo-evaluator/tests/test_agent_evaluate.pyplugins/nemo-evaluator/tests/test_evaluate_job.pyplugins/nemo-evaluator/tests/test_inline_bundle_execution.pyplugins/nemo-evaluator/tests/test_sdk.pyplugins/nemo-evaluator/tests/test_skill_examples.pyplugins/nemo-optimization/examples/hermes-optimize/README.mdplugins/nemo-optimization/src/nemo_optimization/agents.pyplugins/nemo-optimization/tests/test_optimize_job.pyplugins/nemo-rl/src/nemo_rl_plugin/cli/inputs.pyplugins/nemo-safe-synthesizer/README.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/cli.pyplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/SKILL.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/config-runs.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/config.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/diagnose.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/results.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/run.mdplugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.pyplugins/nemo-safe-synthesizer/tests/e2e/test_local_synthesis.pyplugins/nemo-safe-synthesizer/tests/unit/test_cli.pyplugins/nemo-safe-synthesizer/tests/unit/test_local_run.pyplugins/nemo-safe-synthesizer/tests/unit/test_runtime.pyplugins/nemo-unsloth/src/nemo_unsloth_plugin/cli/inputs.pyplugins/nemo-unsloth/tests/test_cli.pyplugins/nemo-unsloth/tests/test_contributor.py
💤 Files with no reviewable changes (6)
- plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/config.md
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/fs_utils.py
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/resources.py
- plugins/nemo-safe-synthesizer/tests/unit/test_local_run.py
- plugins/nemo-safe-synthesizer/tests/e2e/test_local_synthesis.py
- plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/skills/safe-synthesizer/workflows/results.md
| | `nemo anonymizer preview submit` | Generated from `NemoFunction` | Local streaming preview. | | ||
| | `nemo anonymizer preview submit` | Generated from `NemoFunction` | Remote streaming preview against the plugin service. | | ||
| | `nemo anonymizer run run` | Generated from `NemoJob` | Local job execution in the CLI process. | | ||
| | `nemo anonymizer run submit` | Generated from `NemoJob` | Local job execution in the CLI process. | | ||
| | `nemo anonymizer run submit` | Generated from `NemoJob` | Submit an `anonymizer.run` job to the NeMo Platform Jobs worker. | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Complete the remote-only documentation migration.
preview submit and run submit each describe both local and remote behavior. The preview tutorial also conflicts with its field reference, which still permits local paths.
docs/anonymizer/cli.mdx#L14-L17: Keep one row for each command. Remove stale local behavior.docs/anonymizer/cli.mdx#L39-L42: Describe onepreview submitexecution path.docs/anonymizer/cli.mdx#L60-L71: Keep the source matrix consistent with that path.docs/anonymizer/cli.mdx#L98-L114: Keeprun submitremote-only.docs/anonymizer/cli.mdx#L139-L152: Remove duplicate command requirements.docs/anonymizer/tutorials/preview.mdx#L179-L184: Update the field reference at line 82 to remove local-path support.
📍 Affects 2 files
docs/anonymizer/cli.mdx#L14-L17(this comment)docs/anonymizer/cli.mdx#L39-L42docs/anonymizer/cli.mdx#L60-L71docs/anonymizer/cli.mdx#L98-L114docs/anonymizer/cli.mdx#L139-L152docs/anonymizer/tutorials/preview.mdx#L179-L184
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/anonymizer/cli.mdx` around lines 14 - 17, Complete the remote-only
documentation migration: in docs/anonymizer/cli.mdx lines 14-17 keep one row
each for preview submit and run submit, removing local behavior; update lines
39-42 and 60-71 to describe and source only the remote preview path; update
lines 98-114 to keep run submit remote-only; remove duplicate command
requirements in lines 139-152. In docs/anonymizer/tutorials/preview.mdx lines
179-184, update the preview field reference to remove local-path support and
align with the remote-only execution path.
| | `preview submit` | Local CLI process | Optional | | ||
| | `create submit` | Local CLI process | Optional | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Remove the obsolete local execution instructions.
preview submit and create submit appear twice with contradictory execution models. The local sections state that submit runs without NeMo Services. The later rows state that the same commands submit work to NeMo Services.
Delete or rewrite the local preview and create sections. Retain only the service submission behavior.
As per PR objectives, local run APIs and commands are removed.
Also applies to: 51-61
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/data-designer/cli.mdx` around lines 41 - 42, Remove the obsolete local
execution entries for preview submit and create submit from the CLI
documentation, including the corresponding local sections around the referenced
rows. Retain only the rows describing submission through NeMo Services, so each
command has a single service-based execution model.
| nemo data-designer preview submit medical_notes.py --num-records 5 | ||
| ``` | ||
|
|
||
| Generate a larger dataset locally: | ||
|
|
||
| ```bash | ||
| nemo data-designer create run medical_notes.py --num-records 30 | ||
| nemo data-designer create submit medical_notes.py --num-records 30 |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Residual "local" prose left behind by the run → submit rename. Both tutorials now issue submit commands under headings and lead-ins that still promise local execution, so each page contradicts itself.
docs/data-designer/tutorials/seeding.mdx#L259-L265: delete this block; it duplicates the "Submit to NeMo Services" commands at Lines 271-272. Also drop the "locally" labels at Lines 256 and 262 and the local-execution claim at Line 247.docs/data-designer/tutorials/basics.mdx#L205-L205: rename the Line 196 heading "Local CLI Execution" to "CLI Submission", remove the "run locally through the CLI" option at Line 194, and drop "local" from Line 285.
📍 Affects 2 files
docs/data-designer/tutorials/seeding.mdx#L259-L265(this comment)docs/data-designer/tutorials/basics.mdx#L205-L205
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/data-designer/tutorials/seeding.mdx` around lines 259 - 265, Update
docs/data-designer/tutorials/seeding.mdx at lines 259-265 by deleting the
duplicate submit command block, removing “locally” from the labels at lines 256
and 262, and removing the local-execution claim at line 247. In
docs/data-designer/tutorials/basics.mdx at line 205, rename “Local CLI
Execution” to “CLI Submission,” remove the “run locally through the CLI” option
at line 194, and remove “local” from line 285.
| | Mode | Use Case | SDK Call | | ||
| |------|----------|----------| | ||
| | **Local execution** | Rapid prototyping, metric development, and synchronous workflows | `evaluator.run(metric=metric, dataset=dataset)` | | ||
| | **Platform job** | Durable platform execution with polling and result retrieval | `evaluator.submit(metric=metric, dataset=dataset)` | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the execution-mode table distinct.
Line 21 duplicates the durable-job mode on line 22. Collapse the table to one platform-job mode, or restore the standalone SDK mode.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/evaluator/metrics/llm-as-a-judge.mdx` at line 21, Update the
execution-mode table around the “Platform job” row so it no longer duplicates
the durable-job mode on the adjacent row. Either consolidate the entries into
one platform-job mode or restore the intended standalone SDK mode, while keeping
each mode’s description and example accurate.
| | **Platform job** | Durable platform execution with polling and result retrieval | `evaluator.submit(metric=metric, dataset=dataset)` | | ||
| | **Durable remote job** | Production workloads that should run as platform jobs | `evaluator.submit(metric=metric, dataset=dataset)` | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Remove the duplicate execution mode.
Both rows describe durable platform jobs and use the same evaluator.submit(...) call. Keep one row or define different behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/evaluator/metrics/remote.mdx` around lines 24 - 25, Remove the duplicate
execution-mode row from the execution modes table, keeping a single entry for
durable platform jobs with the existing evaluator.submit(metric=metric,
dataset=dataset) example.
|
|
||
| - Prefer CLI surfaces. Generate YAML specs and run `nemo anonymizer ...` commands unless the user explicitly asks for Python. | ||
| - Always iterate via `nemo anonymizer preview run` or `nemo anonymizer preview submit` before running the full job. Previews are cheap and stream a small sample (default 10 records) with full detection traces. | ||
| - Always iterate via `nemo anonymizer preview submit` or `nemo anonymizer preview submit` before running the full job. Previews are cheap and stream a small sample (default 10 records) with full detection traces. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Name the full-job command.
The rule repeats nemo anonymizer preview submit. Replace the second command with nemo anonymizer run submit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md`
at line 37, Update the iteration guidance in SKILL.md so it instructs users to
run “nemo anonymizer preview submit” first, followed by the full-job command
“nemo anonymizer run submit”; remove the duplicated preview command.
| 4. **Build** — Write a YAML preview spec following the Output Template in SKILL.md. Default filename: `<text_column>_preview_spec.yaml` (e.g. `biography_preview_spec.yaml`). | ||
| 5. **Preview** — Run the surface chosen in step 2: | ||
| - Local: `nemo anonymizer preview run --spec-file <path> --workspace <ws>` | ||
| - Plugin service: `nemo anonymizer preview submit --spec-file <path> --workspace <ws>` | ||
| 5. **Preview** — Run `nemo anonymizer preview submit --spec-file <path> --workspace <ws>`. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Verify platform readiness before submission.
Steps 4 and 5 submit work after only checking /openapi.json. Follow SETUP.md, set NMP_BASE_URL=http://localhost:8080, and require /health/ready to succeed before preview submit.
As per coding guidelines, local NeMo CLI operations must set NMP_BASE_URL and verify /health/ready.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md`
around lines 22 - 23, Update the Build and Preview workflow steps to require
local platform readiness before submission: set
NMP_BASE_URL=http://localhost:8080, follow SETUP.md, and verify that
/health/ready succeeds before running nemo anonymizer preview submit. Do not
allow preview submission based only on /openapi.json.
Source: Coding guidelines
| 2. **Confirm the plugin service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, the plugin service isn't loaded — `nemo setup` does not auto-mount it. Tell the user to run `nemo services run` (no `--services` flag) and rerun the check. | ||
| 3. **Confirm input source** — Decide which supported input source you're working with: an `http(s)://` URL or a NeMo Platform fileset reference. If the user named a local file, ask whether to upload it to a fileset first (see `references/inputs.md`). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Set the platform endpoint and readiness check before CLI calls.
This workflow probes http://localhost:8080/openapi.json, but it does not set NMP_BASE_URL or check /health/ready. Later nemo commands can target a different platform or run before the local service is ready. Follow SETUP.md, set the base URL before all nemo commands, and require a successful readiness response.
Suggested setup guard
+export NMP_BASE_URL=http://localhost:8080
+curl --fail --silent "$NMP_BASE_URL/health/ready" >/dev/null
-curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'
+curl --fail --silent "$NMP_BASE_URL/openapi.json" | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'As per coding guidelines, local NeMo CLI commands must set NMP_BASE_URL=http://localhost:8080 and verify /health/ready before proceeding.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| 2. **Confirm the plugin service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, the plugin service isn't loaded — `nemo setup` does not auto-mount it. Tell the user to run `nemo services run` (no `--services` flag) and rerun the check. | |
| 3. **Confirm input source** — Decide which supported input source you're working with: an `http(s)://` URL or a NeMo Platform fileset reference. If the user named a local file, ask whether to upload it to a fileset first (see `references/inputs.md`). | |
| 2. **Confirm the plugin service is mounted.** Run: |
🧰 Tools
🪛 LanguageTool
[grammar] ~11-~11: Ensure spelling is correct
Context: ...cal file, ask whether to upload it to a fileset first (see references/inputs.md). 4. ...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.md`
around lines 10 - 11, Update the workflow setup before the plugin-service probe
and every subsequent nemo command: set NMP_BASE_URL to http://localhost:8080 and
verify that /health/ready returns successfully. Stop and instruct the user to
resolve service readiness if the check fails, then retain the existing
openapi.json mounting check and input-source flow.
Source: Coding guidelines
| There is no plugin-specific CLI command for running an audit yet. Submit | ||
| audits through the SDK or the platform jobs service using the `auditor.audit` | ||
| job entry point. | ||
|
|
||
| ## SDK quickstart | ||
|
|
||
| Every CLI verb has a matching Python SDK method on `client.auditor`, plus | ||
| `client.auditor.run(...)` for in-process execution that bypasses the jobs | ||
| service. | ||
| `client.auditor.submit(...)` for audit job submission. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep the CLI and skill instructions consistent.
This paragraph says that no plugin-specific CLI command exists. plugins/nemo-auditor/src/nemo_auditor/skills/auditor/SKILL.md documents nemo auditor audit submit on Line 81-83 and Line 103. Document the supported submit command here, or remove it from the skill if it is not registered.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-auditor/README.md` around lines 47 - 54, Align the CLI
documentation in the README with the registered command described in the auditor
skill. Verify whether `nemo auditor audit submit` is supported; if registered,
document its usage in the README and update the statement that no
plugin-specific CLI command exists, otherwise remove the command references from
`SKILL.md`.
| def main(argv: list[str] | None = None) -> None: | ||
| """Run the task entry point from either platform env vars or CLI args.""" | ||
| argv = sys.argv[1:] if argv is None else argv | ||
| if not argv: | ||
| run_from_env() | ||
| return | ||
|
|
||
| parser = argparse.ArgumentParser(prog="python -m nemo_safe_synthesizer_plugin.tasks.safe_synthesizer") | ||
| subparsers = parser.add_subparsers(dest="command", required=True) | ||
| local_parser = subparsers.add_parser("run-local", help="Run a Safe Synthesizer job from a local spec file.") | ||
| local_parser.add_argument("--spec-file", required=True, type=Path) | ||
| local_parser.add_argument("--workspace", default="default") | ||
| local_parser.add_argument("--output-dir", required=True, type=Path) | ||
| local_parser.add_argument("--data-source", type=Path) | ||
|
|
||
| args = parser.parse_args(argv) | ||
| if args.command == "run-local": | ||
| run_local( | ||
| spec_file=args.spec_file, | ||
| workspace=args.workspace, | ||
| output_dir=args.output_dir, | ||
| data_source=args.data_source, | ||
| ) | ||
| return | ||
|
|
||
| parser.error(f"unknown command: {args.command}") | ||
| """Run the task entry point from platform environment variables.""" | ||
| if argv: | ||
| raise SystemExit("This task module no longer accepts local execution commands.") | ||
| run_from_env() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Read process arguments before starting the task.
When callers invoke main() with its default value, supplied command-line arguments are ignored and run_from_env() starts. Normalize argv from sys.argv[1:] when it is None before the rejection check.
Proposed fix
+import sys
+
def main(argv: list[str] | None = None) -> None:
"""Run the task entry point from platform environment variables."""
+ argv = sys.argv[1:] if argv is None else argv
if argv:
raise SystemExit("This task module no longer accepts local execution commands.")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def main(argv: list[str] | None = None) -> None: | |
| """Run the task entry point from either platform env vars or CLI args.""" | |
| argv = sys.argv[1:] if argv is None else argv | |
| if not argv: | |
| run_from_env() | |
| return | |
| parser = argparse.ArgumentParser(prog="python -m nemo_safe_synthesizer_plugin.tasks.safe_synthesizer") | |
| subparsers = parser.add_subparsers(dest="command", required=True) | |
| local_parser = subparsers.add_parser("run-local", help="Run a Safe Synthesizer job from a local spec file.") | |
| local_parser.add_argument("--spec-file", required=True, type=Path) | |
| local_parser.add_argument("--workspace", default="default") | |
| local_parser.add_argument("--output-dir", required=True, type=Path) | |
| local_parser.add_argument("--data-source", type=Path) | |
| args = parser.parse_args(argv) | |
| if args.command == "run-local": | |
| run_local( | |
| spec_file=args.spec_file, | |
| workspace=args.workspace, | |
| output_dir=args.output_dir, | |
| data_source=args.data_source, | |
| ) | |
| return | |
| parser.error(f"unknown command: {args.command}") | |
| """Run the task entry point from platform environment variables.""" | |
| if argv: | |
| raise SystemExit("This task module no longer accepts local execution commands.") | |
| run_from_env() | |
| import sys | |
| def main(argv: list[str] | None = None) -> None: | |
| """Run the task entry point from platform environment variables.""" | |
| argv = sys.argv[1:] if argv is None else argv | |
| if argv: | |
| raise SystemExit("This task module no longer accepts local execution commands.") | |
| run_from_env() |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.py`
around lines 396 - 400, Update main to normalize a None argv to sys.argv[1:]
before the rejection check, then preserve the existing SystemExit behavior for
any supplied arguments and call run_from_env only when the normalized list is
empty.
|
fadfffc to
68e6e54
Compare
mckornfield
left a comment
mckornfield
left a comment
There was a problem hiding this comment.
questions/comments, I just stuck to NSS
| ``` | ||
|
|
||
| ## Run the local task directly | ||
| NeMo Safe Synthesizer no longer exposes a host-local workload command. Create synthesis jobs through the platform Jobs API or SDK so scheduling, storage, logs, and artifacts all flow through the supported job path. |
There was a problem hiding this comment.
for the "no longer", do we just want to burn the ships and not act like we ever supported it? this goes for all the plugins, but we'll still have the old behavior documented in the old docs versions
There was a problem hiding this comment.
bumping this, I think as far as confusing agents, we should not have any negative text.
| NeMo Safe Synthesizer has the following additional requirements: | ||
|
|
||
| - An NVIDIA GPU **on the host machine** with 80GB+ VRAM (check with `nvidia-smi`). This is separate from any GPU inside a NIM container; Safe Synthesizer training runs directly on the host. | ||
| - A GPU-capable Jobs backend with enough VRAM for the selected model and dataset. |
There was a problem hiding this comment.
we should probably recommend the 80GB+ VRAM still, even if we're making this more generic to docker/k8s
| """Safe Synthesizer commands.""" | ||
|
|
||
| @runtime_app.command("setup") | ||
| def setup_runtime_command( |
There was a problem hiding this comment.
is this still required? I guess we're getting rid of "run", but we can still do submit with the local process? or is it docker only?
| ## `nemo anonymizer preview` | ||
|
|
||
| Both `preview run` and `preview submit` take a spec file matching `PreviewRequest`. | ||
| Both `preview submit` and `preview submit` take a spec file matching `PreviewRequest`. |
There was a problem hiding this comment.
simple search/replace garbles this line
mckornfield
changed the title
run (local execution) CLI verbs across all servicesrun (local execution) CLI verbs across all services
mckornfield
left a comment
mckornfield
left a comment
There was a problem hiding this comment.
there's some wording things I found, also going to do a triple review, but I think we should try and get this in ASAP
| </Note> | ||
| | Seed source | CLI `submit` / SDK today | Use case | | ||
| |-------------|--------------------------|----------| | ||
| | **Local files or DataFrames** | Not supported | Upload local files to Filesets first. | |
There was a problem hiding this comment.
if things are unsupported, should we just not list them? this goes for all the docs
There was a problem hiding this comment.
These two seed sources are supported in the upstream library, and in the current main state (in which local run still exists) they are supported in that mode, but are not supported in remote/submit mode. Basically, it made more sense when this was a table that included a "CLI run" column.
I think fast-follow, plugin-specific PRs will be easier to manage...
| ``` | ||
|
|
||
| ## Run the local task directly | ||
| NeMo Safe Synthesizer no longer exposes a host-local workload command. Create synthesis jobs through the platform Jobs API or SDK so scheduling, storage, logs, and artifacts all flow through the supported job path. |
There was a problem hiding this comment.
bumping this, I think as far as confusing agents, we should not have any negative text.
| "results to a fileset, but no platform SDK was available. " | ||
| "Set NMP_BASE_URL (so the local CLI can build a default SDK), " | ||
| "pass an explicit sdk via NemoJobScheduler.run_local(sdk=...), " | ||
| "Submit the job through the Jobs API/SDK, pass sdk to " |
There was a problem hiding this comment.
"pass sdk to" doesn't make sense to me
| if signature is None: | ||
| return | ||
|
|
||
| def _collapsed_run(typer_ctx: typer.Context, **kwargs: object) -> None: |
There was a problem hiding this comment.
ah yes, this stupid shim
| ### Local vs. remote | ||
|
|
||
| - **Local** mirrors what `nemo data-designer <preview|create> run` accepts: the engine compiles the config and resolves model providers. Providers can be defined locally **or** referenced by name from the Inference Gateway — both are first-class. | ||
| - **Local** checks library-level compatibility: the engine compiles the config and resolves model providers. Providers can be defined locally **or** referenced by name from the Inference Gateway — both are first-class. |
There was a problem hiding this comment.
guess I'm confused what "locally" means in this context, like a local run of the platform, or local to the platform itself?
There was a problem hiding this comment.
Outdated and incorrect. Again, likely easier to fix in a fast-follow. Once local run mode is removed, data designer on the platform will only support IGW model providers
| "nemo evaluator evaluate run", | ||
| "nemo evaluator agent-evaluate run", | ||
| "client.evaluator.run(", | ||
| "client.evaluator." + "run(", |
| pretrained_model_tmp.cleanup_tmp_dir() | ||
|
|
||
|
|
||
| def main(argv: list[str] | None = None) -> None: |
There was a problem hiding this comment.
P2 / non-blocking — R9
main() never reads sys.argv when invoked as python -m …. Callers that still pass run-local (or any args) get argv=None, skip the rejection, and fall through to run_from_env().
Normalize with argv = sys.argv[1:] if argv is None else argv before the guard, or drop the dead check.
|
|
||
| job_cls = discover_jobs()["example.say-hello"] | ||
| NemoJobScheduler().run_local(job_cls, {"name": "Alice"}) | ||
| NemoJobScheduler().submit_remote(job_cls, {"name": "Alice"}) |
There was a problem hiding this comment.
P2 / non-blocking — R10
This submit_remote(job_cls, …) example omits base_url, which the scheduler requires (raises ValueError when missing).
Also, the surfaces table at line 12 still says the platform auto-generates run / submit / explain and drives local runs — that row was not updated with the rest of this file.
| "nemo evaluator evaluate run", | ||
| "nemo evaluator agent-evaluate run", | ||
| "client.evaluator.run(", | ||
| "client.evaluator." + "run(", |
There was a problem hiding this comment.
P2 / non-blocking — R11
Splitting "client.evaluator." + "run(" hides the literal from repo-wide grep with no runtime benefit — restore the plain string.
Related: skills/nemo-evaluator-plugin/SKILL.md still says the local path “is being retired … even though --help still lists it”, and this test asserts "is being retired". After this PR both APIs are gone; burn the ships and drop the retirement wording + assertion.
|
|
||
|
|
||
| def test_preview_run_saves_expected_artifacts(tmp_path: Path) -> None: | ||
| def test_preview_local_verb_is_not_registered(tmp_path: Path) -> None: |
There was a problem hiding this comment.
P2 / non-blocking — R12
This file lost regression coverage that was not about local execution — notably dataframe-seed rejection (LOCAL_DATAFRAME_SEED_ERROR_MESSAGE / Pydantic ValueError wrapping) and clear errors for bad config sources. Those validators still live on the submit path.
Prefer porting those asserts onto preview submit / create submit rather than only checking "No such command".
|
|
||
| if "://" in agent: | ||
| raise LocalRunError( | ||
| raise RunDependencyError( |
There was a problem hiding this comment.
P2 / non-blocking — R13
RunDependencyError is treated by run_task as a DI/plugin-author failure and re-raised verbatim. Raising it here for user/spec problems (URL agent, unsupported config_format) can surface as an unhandled traceback instead of a mapped job failure.
Prefer a user-facing validation error type (or let the dispatcher map these to exit code 1).
| class LocalRunError(RuntimeError): | ||
| """Raised when a required ``sdk`` / ``async_sdk`` parameter on | ||
| :meth:`NemoJob.run` has no handle to bind.""" | ||
| class RunDependencyError(RuntimeError): |
There was a problem hiding this comment.
P2 / non-blocking — R14
LocalRunError was renamed to RunDependencyError and removed from __all__ with no alias. In-tree call sites were updated, but out-of-tree plugins that catch/import LocalRunError will break for a rename unrelated to removing the run verb.
Consider LocalRunError = RunDependencyError (and keeping it in __all__) for one release.
| NeMo Safe Synthesizer has the following additional requirements: | ||
|
|
||
| - An NVIDIA GPU **on the host machine** with 80GB+ VRAM (check with `nvidia-smi`). This is separate from any GPU inside a NIM container; Safe Synthesizer training runs directly on the host. | ||
| - A GPU-capable Jobs backend with enough VRAM for the selected model and dataset. |
There was a problem hiding this comment.
P3 / non-blocking — R16
Dropping host-specific framing is fine, but the concrete 80GB+ VRAM sizing guidance went with it. The 101 tutorial still recommends 80GB+; keeping that number here (Jobs backend / GPU-capable executor) avoids a guidance regression.
|
|
||
| - Prefer CLI surfaces. Generate YAML specs and run `nemo anonymizer ...` commands unless the user explicitly asks for Python. | ||
| - Always iterate via `nemo anonymizer preview run` or `nemo anonymizer preview submit` before running the full job. Previews are cheap and stream a small sample (default 10 records) with full detection traces. | ||
| - Always iterate via `nemo anonymizer preview submit` or `nemo anonymizer preview submit` before running the full job. Previews are cheap and stream a small sample (default 10 records) with full detection traces. |
There was a problem hiding this comment.
P3 / non-blocking — R17
Search/replace left a duplicated command: preview submit or preview submit. Second should be the full-job surface (nemo anonymizer run submit).
Same class of residue elsewhere in the PR: docs/evaluator/metrics/llm-as-a-judge.mdx still has a ## Local Execution heading, and docs/evaluator/metrics/results.mdx still has ## Get Results from a Local Run above a submit snippet.
| help_result = runner.invoke(app, ["greet", "run", "--help"]) | ||
| help_result = runner.invoke(app, ["greet", "submit", "--help"]) | ||
| assert help_result.exit_code == 0 | ||
| assert "--name" in help_result.output |
There was a problem hiding this comment.
P3 / non-blocking — R18
assert "--name" in help_result.output fails when Rich colorizes help (-- / name split across ANSI codes). The replacement hook itself works; strip ANSI (or assert a color-stable substring) so this does not flake on local TTY runs.
|
|
||
|
|
||
| def test_create_run_reports_artifacts_and_dataset_path(tmp_path: Path) -> None: | ||
| def test_create_local_verb_is_not_registered(tmp_path: Path) -> None: |
There was a problem hiding this comment.
nit / non-blocking — R19
Module/test names still say local (test_preview_local_cli.py, test_*_local_verb_is_not_registered) though they only assert the removed run verb. Rename when convenient so future greps do not treat this as live local-execution coverage.
|
P2 / non-blocking — R15 (no diff line to attach) Fern nav was not updated with the Safe Synthesizer page retitle:
Both still point at |
| }, | ||
| {"input": "Explain quantum physics", "output": "I don't know."}, | ||
| ], | ||
| aggregate_fields=("rubric_distribution", "mode_category"), |
There was a problem hiding this comment.
P1 / blocking — R2
Mechanical run → submit kept aggregate_fields=... on evaluator.submit(...). Evaluator.submit does not accept that kwarg (keyword-only; no **kwargs) — these snippets raise TypeError.
aggregate_fields belongs on result fetch: job.get_result(aggregate_fields=...). Same issue a few lines below with ("std_dev", "variance").
| |----------------------------------|---------------------------------|--------------------------------------------------------------| | ||
| | `nemo anonymizer validate` | Manual Typer command | Validate an `AnonymizerConfig` (and optional `model_configs`).| | ||
| | `nemo anonymizer preview run` | Generated from `NemoFunction` | Local streaming preview. | | ||
| | `nemo anonymizer preview submit` | Generated from `NemoFunction` | Local streaming preview. | |
There was a problem hiding this comment.
P1 / blocking — R3
Incomplete search/replace left duplicate command rows: both preview submit and both run submit, with the first of each still describing local execution.
Also fix nearby garbles: “Both preview submit and preview submit” (L39), the source-kind table with two identical preview submit columns (one still marking local paths yes), and the model_configs note that says required and optional for the same commands.
| |---------|--------------------|-------------------------| | ||
| | `preview run` | Local CLI process | Optional | | ||
| | `create run` | Local CLI process | Optional | | ||
| | `preview submit` | Local CLI process | Optional | |
There was a problem hiding this comment.
P1 / blocking — R4
Mode table now lists preview submit / create submit twice with opposite meanings (“Local CLI process | Optional” vs API/Jobs).
Sections below still say “Preview Locally” / “Create Locally” and claim submit “runs in your current Python environment” / “executes the plugin job locally.” Remove the local path entirely (or rewrite to remote-only submit docs) so this matches execution-modes.mdx and the removed CLI wrappers.
| a helpful error message in phase 1 (until MR 1.3 / MR 1.4 wire them). | ||
| - Invalid JSON on ``--config`` / ``--config-file`` exits 1 cleanly. | ||
| """ | ||
| """Unit tests for the generated submit-only plugin CLI.""" |
There was a problem hiding this comment.
P1 / blocking — R5
This suite dropped from ~67 tests to ~12. Along with run coverage, it also deleted regression tests for surviving submit/explain surfaces: base-URL precedence, CLI auth-header forwarding, connect/HTTP error exit codes, -o / --options-file handling, function NDJSON streaming, _api_segment_for_function, and submit-side auto-flag help.
test_scheduler.py shows the surgical pattern (delete only run_local cases). Please restore submit/explain coverage rather than treating the whole file as local-run debris.
| ) | ||
|
|
||
| result = NemoJobScheduler().run_local( | ||
| result = NemoJobScheduler().submit_remote( |
There was a problem hiding this comment.
P1 / blocking — R8
run_local → submit_remote left sdk=client (not a submit_remote parameter → TypeError) and does not pass the required base_url (raises ValueError when missing).
Same pattern in the other submit_remote examples in this file, plugins/nemo-optimization/examples/hermes-optimize/README.md, and the minimal ARCHITECTURE.md example.
| ) | ||
| print( | ||
| NemoJobScheduler().run_local( | ||
| NemoJobScheduler().submit_remote( |
There was a problem hiding this comment.
P1 / blocking — R8 (same as optimization.mdx)
submit_remote(..., sdk=client) is invalid — NemoJobScheduler.submit_remote accepts base_url / workspace / http_client / headers, not sdk. Pass base_url= explicitly (or rely on env resolution if documented) and drop sdk=.
| sibling agent, or by editing the eval-config YAML. | ||
| - `nemo auditor targets create <target> -d '{"model": "<new-model>", "type": "<type>"}'` then | ||
| `nemo auditor audit run --spec '{"config": "default/<config>", "target": "default/<target>"}'` | ||
| `nemo auditor audit submit --spec '{"config": "default/<config>", "target": "default/<target>"}'` |
There was a problem hiding this comment.
P1 / blocking — R7
This file was only partially migrated (auditor run → submit below). It still directs agents to deleted verbs elsewhere in the same skill:
nemo agents evaluate runnemo agents optimize runnemo agents optimize-skills run
Generated CLI now exposes only submit / explain. Same leftover nemo agents optimize run prose remains in docs/agents/optimization.mdx and plugins/nemo-optimization/examples/hermes-optimize/README.md.
|
P1 / blocking — R1 (file not in this PR’s diff)
Verified with |
|
P1 / blocking — R6 (dotted Authoritative plugin-author skills under
Also |
Signed-off-by: Ryan S <267728323+ironcommit@users.noreply.github.com>
68e6e54 to
b6222a6
Compare
Clean up incomplete run→submit search/replace leftovers in docs and author skills, restore critical submit CLI regression tests, and fix invalid submit_remote(sdk=...) examples after rebasing onto main. Signed-off-by: Matt Kornfield <mkornfield@nvidia.com>
…iders Clarify that local file/DataFrame seeds remain library-valid but not on the platform path, and that remote validate/submit is IGW-provider-only. Signed-off-by: Matt Kornfield <mkornfield@nvidia.com>
4 participants
Summary
Remove the
run(in-process / local execution) CLI verb from all plugin jobs and functions, leavingsubmit(remote platform execution) as the only execution path. This simplifies the CLI surface, removes dual-path maintenance burden, and aligns the developer experience with the platform's remote-first execution model.Changes
Plugin framework (
nemo_platform_plugin)commands.py: Strip therunverb generator for bothNemoJobandNemoFunction; onlysubmit(andexplainfor jobs) are generated.scheduler.py: RemoveNemoJobScheduler.run_local()and all in-process execution machinery.cli.py/cli_state.py/cli_renderer.py: Update type literals, docstrings, and renderer context to reflect submit-only surfaces. Retainis_localandverbfields for third-party renderer compatibility.function.py/job.py/job_context.py/run_dependencies.py: Remove local-run wiring and related helpers.Per-plugin updates
Evaluator.run()SDK method andfs_utils.py; deleteresources.py(local execution resources). Update all evaluator tests.runCLI paths, updatecli.py/inputs.py, and adjust tests.say_hellojob to submit-only; update tests.Documentation (43
.mdxfiles + READMEs + skill files)nemo <plugin> <job> runinstructions withsubmitequivalents.Deleted files
plugins/nemo-evaluator/src/nemo_evaluator/sdk/fs_utils.pyplugins/nemo-safe-synthesizer/tests/e2e/test_local_synthesis.pyplugins/nemo-safe-synthesizer/tests/unit/test_local_run.pyType of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified below