From b07d631bd886305e92cfb32e6424eefaf7df567a Mon Sep 17 00:00:00 2001 From: Nico Tonozzi Date: Fri, 31 Jul 2026 16:02:49 -0700 Subject: [PATCH 1/4] docs(experimentalist): recommend Docker Sandboxes Document an isolated laptop workflow while preserving the current architecture caveat. Signed-off-by: Nico Tonozzi --- docs/get-started/example-agent.mdx | 29 ++++++++++++++++++++++---- plugins/nemo-experimentalist/README.md | 28 +++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 4 deletions(-) diff --git a/docs/get-started/example-agent.mdx b/docs/get-started/example-agent.mdx index edffb4eb27..1cb196cbf8 100644 --- a/docs/get-started/example-agent.mdx +++ b/docs/get-started/example-agent.mdx @@ -7,7 +7,7 @@ This provides a guide for how you can quickly load agent traces into your NeMo P ## Prerequisites -- Git, GNU Make, uv, pnpm, Docker +- Git, GNU Make, uv, pnpm, Docker, and Docker Sandboxes (`sbx`) - An NVIDIA Inference Gateway virtual key (`sk-...`) from `inference.nvidia.com`. ## 1. Clone and bootstrap NeMo Platform @@ -97,10 +97,31 @@ uv run --frozen nemo workspaces create canonical-tau3-airline \ --exist-ok ``` -Now we're ready to run the experimentalist! +Run the experimentalist in a Docker Sandbox so that both the optimization +process and Harbor's task containers are isolated from the host. The sandbox +uses `host.docker.internal` to reach the NeMo Platform services running on the +host: ```bash -uv run --frozen nemo experimentalist run \ +repo="$(git rev-parse --show-toplevel)" +sbx create --name nemo-experimentalist shell "$repo" +sbx exec --workdir "$repo" \ + --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \ + --env INFERENCE_API_KEY \ + --env INFERENCE_API_BASE \ + --env OPENAI_API_KEY \ + --env OPENAI_BASE_URL \ + --env EXPERIMENTALIST_API_KEY \ + --env EXPERIMENTALIST_API_BASE \ + --env EXPERIMENTALIST_SMART_MODEL_NAME \ + --env EXPERIMENTALIST_MID_MODEL_NAME \ + --env EXPERIMENTALIST_FAST_MODEL_NAME \ + --env TAU2_USER_MODEL \ + --env TAU2_NL_ASSERTIONS_MODEL \ + --env AUT_MODEL_NAME \ + nemo-experimentalist \ + uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \ + nemo experimentalist run \ --no-insight \ --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \ --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \ @@ -110,7 +131,7 @@ uv run --frozen nemo experimentalist run \ --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \ --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \ --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \ - --base-url "$NMP_BASE_URL" + --base-url http://host.docker.internal:8080 ``` The trace records include the Experimentalist evaluation ID and Tau3 task ID. diff --git a/plugins/nemo-experimentalist/README.md b/plugins/nemo-experimentalist/README.md index 4ccd8c7f72..e699488297 100644 --- a/plugins/nemo-experimentalist/README.md +++ b/plugins/nemo-experimentalist/README.md @@ -61,6 +61,34 @@ a baseline agent on Harbor-compatible train and validation datasets, proposes candidate mutations, and records its artifacts under the selected experiment directory. +### Recommended laptop isolation + +Use [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) instead of a +privileged Docker-in-Docker container or a host Docker socket mount. The +Experimentalist runs inside an isolated microVM, while Harbor uses that +sandbox's private Docker daemon for task containers: + +```bash +repo="$(git rev-parse --show-toplevel)" +sbx create --name nemo-experimentalist shell "$repo" +sbx exec --workdir "$repo" \ + --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \ + --env EXPERIMENTALIST_API_BASE \ + --env EXPERIMENTALIST_API_KEY \ + --env EXPERIMENTALIST_SMART_MODEL_NAME \ + --env EXPERIMENTALIST_MID_MODEL_NAME \ + --env EXPERIMENTALIST_FAST_MODEL_NAME \ + nemo-experimentalist \ + uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \ + nemo experimentalist run +``` + +Append the run options described below. `UV_PROJECT_ENVIRONMENT` keeps the +sandbox's Linux environment separate from the host checkout's `.venv`. On +Apple silicon, Harbor tasks that publish only `linux/amd64` images do not run +in the `linux/arm64` sandbox. This currently includes the Terminal-Bench +`fix-git` task; use an x86_64 machine or VM for that suite. + Configure the models before running an experiment: ```bash From 91dbe604050721697e116e20e1f28d8e1a5418fb Mon Sep 17 00:00:00 2001 From: Brian Newsom Date: Tue, 4 Aug 2026 14:47:55 -0600 Subject: [PATCH 2/4] docs: harden experimentalist sandbox workflow Signed-off-by: Brian Newsom --- docs/get-started/example-agent.mdx | 78 +++++++++++++++++++------- plugins/nemo-experimentalist/README.md | 29 ++++++++-- 2 files changed, 81 insertions(+), 26 deletions(-) diff --git a/docs/get-started/example-agent.mdx b/docs/get-started/example-agent.mdx index 1cb196cbf8..7ff6d506c6 100644 --- a/docs/get-started/example-agent.mdx +++ b/docs/get-started/example-agent.mdx @@ -7,7 +7,9 @@ This provides a guide for how you can quickly load agent traces into your NeMo P ## Prerequisites -- Git, GNU Make, uv, pnpm, Docker, and Docker Sandboxes (`sbx`) +- Git, GNU Make, uv, and pnpm +- Docker Engine 29.6.2 or later +- [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) (`sbx`) 0.37.1 or later - An NVIDIA Inference Gateway virtual key (`sk-...`) from `inference.nvidia.com`. ## 1. Clone and bootstrap NeMo Platform @@ -98,13 +100,19 @@ uv run --frozen nemo workspaces create canonical-tau3-airline \ ``` Run the experimentalist in a Docker Sandbox so that both the optimization -process and Harbor's task containers are isolated from the host. The sandbox -uses `host.docker.internal` to reach the NeMo Platform services running on the -host: +process and Harbor's task containers use the sandbox's microVM and private +Docker daemon. Clone mode gives the experimentalist a private writable clone +instead of write access to the host checkout. The host checkout remains +available read-only at `/run/sandbox/source`; the prepared evaluation dataset +is read from there. The sandbox uses `host.docker.internal` to reach the NeMo +Platform services running on the host: ```bash repo="$(git rev-parse --show-toplevel)" -sbx create --name nemo-experimentalist shell "$repo" +sbx create --clone --name nemo-experimentalist shell "$repo" + +# Harbor's Supabase storage download fails over SBX's HTTP/2 forward proxy. +# Route only that public host through SBX's policy-enforced transparent proxy. sbx exec --workdir "$repo" \ --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \ --env INFERENCE_API_KEY \ @@ -120,22 +128,52 @@ sbx exec --workdir "$repo" \ --env TAU2_NL_ASSERTIONS_MODEL \ --env AUT_MODEL_NAME \ nemo-experimentalist \ - uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \ - nemo experimentalist run \ - --no-insight \ - --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \ - --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \ - --train-dataset plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \ - --validation-dataset plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \ - --workspace canonical-tau3-airline \ - --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \ - --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \ - --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \ - --base-url http://host.docker.internal:8080 + sh -lc ' + export NO_PROXY="${NO_PROXY:+$NO_PROXY,}ofhuhcpkvzjlejydnvyd.supabase.co" + export no_proxy="$NO_PROXY" + exec uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \ + nemo experimentalist run \ + --no-insight \ + --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \ + --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \ + --train-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \ + --validation-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \ + --workspace canonical-tau3-airline \ + --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \ + --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \ + --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \ + --base-url http://host.docker.internal:8080 + ' ``` +The sandbox needs outbound access to the package, model, registry, Harbor +dataset, and NeMo Platform endpoints used by the run. `host.docker.internal` +is translated to the host's loopback interface so the sandbox can reach the +NeMo Platform service on port 8080. + + +Clone mode protects the host checkout from writes, but it does not isolate +secrets or host services from code in the sandbox. The complete host repository, +including ignored `.env` files, is readable at `/run/sandbox/source`. Values +passed with `sbx exec --env` are readable by the optimizer, candidate agents, +Harbor verifier, and their subprocesses. Use dedicated, revocable, +spending-limited credentials, and do not expose unrelated host services or +network destinations. Optimizer and task code can use any outbound access +granted to the sandbox. + + The trace records include the Experimentalist evaluation ID and Tau3 task ID. -After the run completes, inspect +After the run completes, use `sbx exec` to inspect `plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize/run.json` -for the selected winner and compare the `agent-0` and `agent-1` directories to -review the code change that was evaluated. +in the sandbox for the selected winner and compare the `agent-0` and `agent-1` +directories to review the code change that was evaluated. Copy artifacts you +want to retain to the host with `sbx cp` before removing the sandbox. + +Stopping preserves the VM, its private Git clone, experiment output, installed +packages, and Docker image/build cache. Remove the sandbox when you no longer +need that state: + +```bash +sbx stop nemo-experimentalist +sbx rm nemo-experimentalist +``` diff --git a/plugins/nemo-experimentalist/README.md b/plugins/nemo-experimentalist/README.md index e699488297..9a05777b0a 100644 --- a/plugins/nemo-experimentalist/README.md +++ b/plugins/nemo-experimentalist/README.md @@ -66,11 +66,16 @@ directory. Use [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) instead of a privileged Docker-in-Docker container or a host Docker socket mount. The Experimentalist runs inside an isolated microVM, while Harbor uses that -sandbox's private Docker daemon for task containers: +sandbox's private Docker daemon for task containers. Clone mode gives the +Experimentalist a private writable clone instead of write access to the host +checkout. This flow requires Docker Engine 29.6.2 or later and was tested with +Docker Sandboxes (`sbx`) 0.37.1. + +Create the sandbox and run the Experimentalist: ```bash repo="$(git rev-parse --show-toplevel)" -sbx create --name nemo-experimentalist shell "$repo" +sbx create --clone --name nemo-experimentalist shell "$repo" sbx exec --workdir "$repo" \ --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \ --env EXPERIMENTALIST_API_BASE \ @@ -84,10 +89,22 @@ sbx exec --workdir "$repo" \ ``` Append the run options described below. `UV_PROJECT_ENVIRONMENT` keeps the -sandbox's Linux environment separate from the host checkout's `.venv`. On -Apple silicon, Harbor tasks that publish only `linux/amd64` images do not run -in the `linux/arm64` sandbox. This currently includes the Terminal-Bench -`fix-git` task; use an x86_64 machine or VM for that suite. +sandbox's Linux environment separate from the host checkout's `.venv`. Clone +mode prevents sandbox writes to the host checkout, but it is not a secret +isolation boundary: the complete host repository, including ignored `.env` +files, remains readable at `/run/sandbox/source`. Values passed with `sbx exec +--env` are readable by the optimizer, candidate agent, verifier, and their +subprocesses. Use dedicated, revocable, spending-limited keys. Optimizer and +task code can use any outbound access granted to the sandbox. The sandbox must +be able to reach the package, model, registry, Harbor dataset, and NeMo Platform +endpoints required by the run. + +Copy experiment artifacts you want to retain to the host with `sbx cp`. +`sbx stop nemo-experimentalist` preserves the VM, output, packages, and private +Docker cache; `sbx rm nemo-experimentalist` deletes them. On Apple silicon, +Harbor tasks that publish only `linux/amd64` images do not run in the +`linux/arm64` sandbox. This currently includes the Terminal-Bench `fix-git` +task; use an x86_64 machine or VM for that suite. Configure the models before running an experiment: From 2060003ac40fee5c9601cc9656abe8f0f73e6e41 Mon Sep 17 00:00:00 2001 From: Brian Newsom Date: Tue, 4 Aug 2026 14:55:26 -0600 Subject: [PATCH 3/4] docs: simplify sandbox command invocation Signed-off-by: Brian Newsom --- docs/get-started/example-agent.mdx | 31 ++++++++++++------------------ 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/docs/get-started/example-agent.mdx b/docs/get-started/example-agent.mdx index 063dd4b229..d95701192d 100644 --- a/docs/get-started/example-agent.mdx +++ b/docs/get-started/example-agent.mdx @@ -112,9 +112,6 @@ Platform services running on the host: ```bash repo="$(git rev-parse --show-toplevel)" sbx create --clone --name nemo-experimentalist shell "$repo" - -# Harbor's Supabase storage download fails over SBX's HTTP/2 forward proxy. -# Route only that public host through SBX's policy-enforced transparent proxy. sbx exec --workdir "$repo" \ --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \ --env INFERENCE_API_KEY \ @@ -130,22 +127,18 @@ sbx exec --workdir "$repo" \ --env TAU2_NL_ASSERTIONS_MODEL \ --env AUT_MODEL_NAME \ nemo-experimentalist \ - sh -lc ' - export NO_PROXY="${NO_PROXY:+$NO_PROXY,}ofhuhcpkvzjlejydnvyd.supabase.co" - export no_proxy="$NO_PROXY" - exec uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \ - nemo agents experimentalist run \ - --no-insight \ - --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \ - --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \ - --train-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \ - --validation-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \ - --workspace canonical-tau3-airline \ - --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \ - --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \ - --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \ - --base-url http://host.docker.internal:8080 - ' + uv run --frozen --python 3.13 --package nemo-experimentalist-plugin \ + nemo agents experimentalist run \ + --no-insight \ + --agent plugins/nemo-experimentalist/examples/tau3-nooa-agent \ + --agent-spec plugins/nemo-experimentalist/examples/tau3-nooa-agent/AGENT-SPEC.md \ + --train-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/train \ + --validation-dataset /run/sandbox/source/plugins/nemo-experimentalist/tmp/tau3-airline/experimentalist/validation \ + --workspace canonical-tau3-airline \ + --framework-skills plugins/nemo-experimentalist/framework-skills/nooa \ + --config plugins/nemo-experimentalist/examples/tau3-nooa-agent/experimentalist-smoke.yaml \ + --experiment-dir plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist \ + --base-url http://host.docker.internal:8080 ``` The sandbox needs outbound access to the package, model, registry, Harbor From e88172625eeed23db4a6e8e6ea6044b0a8b47a34 Mon Sep 17 00:00:00 2001 From: Brian Newsom Date: Tue, 4 Aug 2026 15:12:29 -0600 Subject: [PATCH 4/4] docs: add sandbox artifact commands Signed-off-by: Brian Newsom --- docs/get-started/example-agent.mdx | 24 +++++++++++++++++++----- plugins/nemo-experimentalist/README.md | 4 +++- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/get-started/example-agent.mdx b/docs/get-started/example-agent.mdx index d95701192d..9eb524a195 100644 --- a/docs/get-started/example-agent.mdx +++ b/docs/get-started/example-agent.mdx @@ -158,11 +158,25 @@ granted to the sandbox. The trace records include the Experimentalist evaluation ID and Tau3 task ID. -After the run completes, use `sbx exec` to inspect -`plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize/run.json` -in the sandbox for the selected winner and compare the `agent-0` and `agent-1` -directories to review the code change that was evaluated. Copy artifacts you -want to retain to the host with `sbx cp` before removing the sandbox. +After the run completes, inspect the run summary and compare the baseline with +the candidate. The `diff` command exits with status 1 when it finds candidate +changes, which is expected: + +```bash +artifact_dir="plugins/nemo-experimentalist/tmp/tau3-airline-experimentalist/eval-and-optimize" +sbx exec --workdir "$repo" nemo-experimentalist \ + sed -n '1,240p' "$artifact_dir/run.json" +sbx exec --workdir "$repo" nemo-experimentalist \ + diff -ru "$artifact_dir/agents/agent-0" "$artifact_dir/agents/agent-1" +``` + +Copy artifacts you want to retain to the host before removing the sandbox: + +```bash +mkdir -p tmp/tau3-airline-artifacts +sbx cp "nemo-experimentalist:$repo/$artifact_dir" \ + tmp/tau3-airline-artifacts/ +``` Stopping preserves the VM, its private Git clone, experiment output, installed packages, and Docker image/build cache. Remove the sandbox when you no longer diff --git a/plugins/nemo-experimentalist/README.md b/plugins/nemo-experimentalist/README.md index 41dd4d1610..adfa4e3921 100644 --- a/plugins/nemo-experimentalist/README.md +++ b/plugins/nemo-experimentalist/README.md @@ -96,7 +96,9 @@ task code can use any outbound access granted to the sandbox. The sandbox must be able to reach the package, model, registry, Harbor dataset, and NeMo Platform endpoints required by the run. -Copy experiment artifacts you want to retain to the host with `sbx cp`. +Copy experiment artifacts you want to retain to the host with `sbx cp`; the +[complete example](../../docs/get-started/example-agent.mdx#5-optimize-performance-with-the-experimentalist) +includes concrete inspection, comparison, and copy commands. `sbx stop nemo-experimentalist` preserves the VM, output, packages, and private Docker cache; `sbx rm nemo-experimentalist` deletes them. On Apple silicon, Harbor tasks that publish only `linux/amd64` images do not run in the