feat(opensandbox): add automatic job attribution (team/user/workload) to sandbox metadata - #2020
Merged
Merged
Conversation
Contributor
|
🌿 Preview your docs: https://nvidia-preview-hemil-opensandbox-attribution.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
1 similar comment
Contributor
|
🌿 Preview your docs: https://nvidia-preview-hemil-opensandbox-attribution.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
hemildesai
marked this pull request as ready for review
July 30, 2026 07:29
ananthsub
approved these changes
Jul 30, 2026
… to sandbox metadata Resolves RL-1013. Every sandbox created through the OpenSandbox provider now carries team/user/workload attribution in its metadata, which OpenSandbox stores as Kubernetes labels on the sandbox and exposes through the list API's metadata filter (e.g. team=my-team), so cluster operators can attribute running sandboxes to the job that created them. - new nemo_gym/sandbox/attribution.py: resolve_attribution() resolves each field from explicit config, then NEMO_GYM_TEAM/NEMO_GYM_USER/ NEMO_GYM_WORKLOAD env vars, then Slurm job env vars (SLURM_JOB_ACCOUNT/ SLURM_JOB_USER/SLURM_JOB_NAME), then the OS login name for user; unresolvable fields are omitted rather than guessed. Shared module so other providers can adopt the same convention. - OpenSandboxProvider grows an 'attribution' config group (enabled: true by default, plus team/user/workload overrides); create() merges attribution under the spec's metadata, so explicit sandbox_spec.metadata and default_metadata keys always win, and values flow through the existing Kubernetes-label sanitization. - documented in configs/opensandbox.yaml and the OpenSandbox fern page. Carrier choice (from SDK research): Sandbox.create(metadata=...) is the only create-time channel that is stored on the sandbox, queryable (SandboxFilter metadata), and patchable post-create; extensions are opaque/unqueryable and connection headers are per-request only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Hemil Desai <hemild@nvidia.com>
…labels
Attribution keys are now emitted as nemo-gym.nvidia.com/team|user|workload
(configurable via attribution.key_prefix; "" restores bare keys). OpenSandbox
propagates sandbox metadata as Kubernetes labels on the sandbox resources, so
prefixed keys give cleanly namespaced, cluster-level attribution:
kubectl get pods -l nemo-gym.nvidia.com/team=my-team
matching the existing <project>.nvidia.com/ label-key convention used for
run-level sandbox tooling, and avoiding collisions with other tenants' labels
on shared clusters.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
…ed labels Address review findings from replaying attribution against a real Kubernetes eval deployment, where the previous fallback chain resolved to user=root with team/workload omitted: - Ignore "root" from the OS-login fallback: containers default to root, so it attributes the image, not a person. Explicit NEMO_GYM_USER=root is still honored. - Fall back workload to NEMO_GYM_CONFIG_PATH, the server instance name the gym CLI already sets on every server process it spawns, so workload resolves with zero plumbing outside Slurm. - Add a run attribution key (NEMO_GYM_RUN_ID, else generated once per process and logged at first create) so one launch's sandboxes can be listed and garbage-collected exactly; team/user/workload cannot distinguish two runs of the same workload by the same user. - Validate attribution.key_prefix as a DNS-1123 label-key prefix at config time instead of failing server-side at create with an opaque error. - Docs: correct the "every sandbox" claim (fields are omitted when unresolvable) and add a Kubernetes deployment section showing how to set NEMO_GYM_* on pod specs directly or via the downward API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Hemil Desai <hemild@nvidia.com>
ananthsub
force-pushed
the
hemil/opensandbox-attribution
branch
from
July 31, 2026 17:32
9824ec2 to
4b19975
Compare
ananthsub
enabled auto-merge (squash)
July 31, 2026 17:33
bxyu-nvidia
pushed a commit
that referenced
this pull request
Jul 31, 2026
…esource requests/limits (#2212) ## Summary Hardens the OpenSandbox provider and the `mini_swe_agent_2` eval path so that large sandboxed agent evals (SWE-bench Verified scale: hundreds of concurrent rollouts against a Kubernetes-backed OpenSandbox deployment) run end-to-end reliably: every finished rollout is delivered and recorded, and sandbox-infrastructure noise is bounded instead of silently zeroing rewards. Diagnosed on multi-node runs at concurrency 300–1500; the same failure signatures appear at concurrency 8, just less often. ### 1. `Server disconnected without sending a response` The SDK's default httpx pool keeps idle connections for 30s (`opensandbox.config.connection.with_transport_if_missing`), but the OpenSandbox server's uvicorn keep-alive reaper closes idle sockets after ~5s. Agent workloads idle between sandbox commands (model think time), so commands routinely reuse a socket the server already closed — surfacing as `httpx.RemoteProtocolError` → `SandboxInternalException`, which permanently kills the rollout. **Fix:** the provider injects a transport whose `keepalive_expiry` sits *below* the server's keep-alive timeout (default 3s, `connection.keepalive_expiry_s`): - `connection.transport_backend: httpx` (default) — stock `httpx.AsyncHTTPTransport(limits=..., retries=connect_retries)`. No new required dependencies. - `connection.transport_backend: aiohttp` (opt-in) — aiohttp pool under the SDK's httpx surface via [`httpx-aiohttp`](https://github.com/karpetrosyan/httpx-aiohttp); falls back to the httpx transport with a warning when absent. - `keepalive_expiry_s: null` disables injection entirely (SDK default transport). ### 2. 502 `could not connect to the backend sandbox endpoint='<podIP>:44772'` on first command With `create.skip_health_check: true`, `Sandbox.create` returns before the pod's exec daemon is listening; the first command races pod startup and the server proxy 502s. Under a create burst this killed 7–12 rollouts per run. **Fix:** `create.skip_health_check: false` (create waits for readiness, bounded by the spec's `ready_timeout_s`; `create.timeout_s` kept above the ready timeout). **`command_retries` stays at 0.** Retrying a command the server may have already started would execute it twice, and agent commands are frequently mutating. The keepalive bound removes the stale-connection failures retries were compensating for. Raise it only for idempotent workloads. ### 3. Separate resource requests and limits `Sandbox.create` accepts distinct `resource` (limits) and `resource_requests` maps; a lone `resource` map is applied by the server as requests=limits. This exposes the requests side via `sandbox_spec.provider_options.resource_requests` (same keys as `SandboxSpec.resources`). Motivation: SWE-bench test suites OOM-killed sandbox pods at 2Gi, but raising a single combined map to 8Gi would 4× the cluster reservation; with the split, limits rise while requests stay small and pods pack densely. Requires `opensandbox>=0.1.15` (lower bound raised here). ### 4. Sandbox teardown off the rollout's critical path `env.cleanup()` ran in the rollout Ray task's `finally` block, so a finished result only became fetchable after the sandbox DELETE returned — and a teardown failure re-raised *over* the successful eval, degrading it to a reward-0 error row. Teardown is also effectively one-shot (`Sandbox.stop` latches `_closed` before dispatch), so a single failed DELETE leaked the sandbox while still costing the rollout. **Fix:** cleanup runs on a best-effort daemon thread; results return immediately, teardown errors are logged rather than raised, and orphans remain covered by the provider's sandbox TTL. ### 5. High-concurrency rollout delivery (`mini_swe_agent_2`) Two coupled fixes so every finished rollout is delivered at high concurrency (note: at very high concurrency the single-process policy-model proxy remains a separate bottleneck — connection-refused storms confirmed by a controlled A/B — with its scaling fix tracked separately): - **await the Ray ObjectRef** instead of `asyncio.to_thread(ray.get, ...)`: the default executor caps at `min(32, cpu+4)` threads, each pinned for a full rollout, so delivery stalls at ~32 concurrent rollouts and finished tasks queue behind blocked `ray.get` calls. - **bounded litellm retries (`num_retries=5`, config-overridable)** — no retry means one transient LLM-call failure kills a whole rollout; unbounded retries make failures look like hangs — and **`num_cpus=0.25`** on the rollout Ray task so concurrency is not capped at cluster core count (rollouts are I/O-bound). ### 6. `datasets` declared in the base agent config The struct-mode config merge rejects keys absent from the base config, so a benchmark config using `_inherit_from` could not add its dataset list (`ConfigKeyError: Key 'datasets' is not in struct`). Declared empty in the base server config, matching how other agents (e.g. `swe_agents`) expose it. ## Validation - Requests/limits split (paired multi-node runs, identical except the variable under test): infra-failed rollouts 41/316 (**13.0%**) at 2Gi requests=limits → 19/288 (**6.6%**) at 8Gi limits / 2Gi requests; pass@1 58.0% → **67.1%**. - Keepalive bound + health-checked create: zero `Server disconnected` events and zero create-race 502s across all subsequent runs (previously 2–12 per run). - Full stack, end-to-end from this branch: single-pass SWE-bench Verified (500 instances, concurrency 500+) delivered **>98% of rollouts** with ~1.6% residual sandbox-infra failures (server-side proxy 502s on established sandboxes, tracked separately) and zero client-transport failures. Before the delivery fixes (#4/#5), an identically shaped run lost the majority of *completed* evaluations at the wall — hundreds of finished evals, only tens recorded. ## Relationship to #2020 Complementary, no overlap; #2020 (job attribution metadata) has since merged and this branch is updated on top of it. Field-validated together — the attribution labels are what make post-cancellation sandbox garbage collection safely scoped to a single job. ## Testing - `tests/unit_tests/test_opensandbox_provider.py`: transport-backend coverage (httpx default with keepalive expiry, custom pool settings, fallback when `httpx_aiohttp` is unavailable, `null` disables injection), aiohttp opt-in (importorskip-guarded), and requests/limits plumbing tests. 16 passed with and without `httpx-aiohttp`. - `responses_api_agents/mini_swe_agent_2/tests/test_app.py`: updated for the awaited ObjectRef (awaitable `FakeObjectRef`); assertions strengthened to check the Ray call's params. - CI green except a pre-existing `tau2` failure on `main` (reproduces on unrelated PRs). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
… to sandbox metadata (NVIDIA-NeMo#2020) ## Summary Fixes RL-1013 ([Linear issue](https://linear.app/nvidia/issue/RL-1013/add-job-attribution-teamuserworkload-to-opensandbox)). Every sandbox created through the OpenSandbox provider now automatically carries `nemo-gym.nvidia.com/team` / `nemo-gym.nvidia.com/user` / `nemo-gym.nvidia.com/workload` / `nemo-gym.nvidia.com/run` keys in its metadata. OpenSandbox propagates sandbox metadata as **Kubernetes labels** on the sandbox resources (BatchSandbox CR + pod), so sandboxes are attributable both through the OpenSandbox list API and directly at the cluster level: ```bash kubectl get pods -l nemo-gym.nvidia.com/team=my-team ``` ## Why metadata (SDK research) From the `opensandbox` SDK (0.1.9) source, there are exactly three candidate carriers for attribution: | Carrier | Verdict | |---|---| | `Sandbox.create(metadata=...)` | ✅ **Chosen.** Becomes Kubernetes labels on the sandbox (server enforces K8s label rules: values ≤63 chars, `opensandbox.io/` prefix reserved for the platform itself), is queryable via `SandboxManager.list_sandbox_infos(SandboxFilter(metadata={...}))`, and is patchable post-create (`PATCH /v1/sandboxes/{id}/metadata`). | | `extensions` | ❌ Opaque provider pass-through, not queryable, immutable post-create. | | `ConnectionConfig.headers` / `user_agent` / `api_key` | ❌ Per-request HTTP headers only; not stored on the sandbox. | K8s **annotations** are not settable through the SDK/API at all — labels are the only k8s-level channel it propagates. Keys use the Kubernetes prefixed-key convention (`nemo-gym.nvidia.com/`) for namespacing and provenance, matching the existing `<project>.nvidia.com/` label-key convention used by run-level sandbox tooling; the prefix is configurable (`key_prefix: ""` restores bare `team`/`user`/`workload` keys). The provider already sanitizes metadata values to K8s label rules (`_metadata_value`: charset replacement + 63-char truncation) and pipes `spec.metadata` into `Sandbox.create`, so this change only adds the automatic injection. ## Design - **`nemo_gym/sandbox/attribution.py`** (shared, provider-agnostic so docker/openshell/ecs can adopt it later): `resolve_attribution()` resolves each field in order — explicit config → `NEMO_GYM_TEAM` / `NEMO_GYM_USER` / `NEMO_GYM_WORKLOAD` env vars → Slurm job env vars (`SLURM_JOB_ACCOUNT` / `SLURM_JOB_USER` / `SLURM_JOB_NAME`) → OS login name (`user` only; `root` is ignored since containers default to it) → the gym CLI's `NEMO_GYM_CONFIG_PATH` server instance name (`workload` only). Unresolvable fields are **omitted, never guessed**. A `run` key (`NEMO_GYM_RUN_ID`, else generated once per process and logged at first create) scopes sandboxes to one launch so an interrupted run's sandboxes can be listed and garbage-collected exactly. - **`OpenSandboxProvider`**: new `attribution` config group (`enabled: true` by default; `team`/`user`/`workload` overrides; `key_prefix` for the label-key namespace). `create()` merges attribution **under** the spec's metadata, so explicit `sandbox_spec.metadata` / `default_metadata` keys always win; values then flow through the existing K8s-label sanitization. - Documented in `configs/opensandbox.yaml` and `fern/.../sandbox/opensandbox.mdx` (new *Job Attribution* section). ## Testing - `tests/unit_tests/test_sandbox_attribution.py`: resolution precedence (config > NEMO_GYM_* > Slurm > login name), blank-value handling, omission when unresolvable, `os.environ` default. - `tests/unit_tests/test_opensandbox_provider.py`: attribution lands in the SDK `create` call's `metadata` with prefixed keys (and sanitization applied, e.g. `nemo rl` → `nemo_rl`), explicit spec metadata beats attribution, config overrides beat env detection, `enabled: false` disables injection, and `key_prefix` variants (bare, custom, trailing-slash normalization). - Full unit suite: 1152 passed; the 6 failures are pre-existing terminal-formatting assertions that fail identically on `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
…esource requests/limits (NVIDIA-NeMo#2212) ## Summary Hardens the OpenSandbox provider and the `mini_swe_agent_2` eval path so that large sandboxed agent evals (SWE-bench Verified scale: hundreds of concurrent rollouts against a Kubernetes-backed OpenSandbox deployment) run end-to-end reliably: every finished rollout is delivered and recorded, and sandbox-infrastructure noise is bounded instead of silently zeroing rewards. Diagnosed on multi-node runs at concurrency 300–1500; the same failure signatures appear at concurrency 8, just less often. ### 1. `Server disconnected without sending a response` The SDK's default httpx pool keeps idle connections for 30s (`opensandbox.config.connection.with_transport_if_missing`), but the OpenSandbox server's uvicorn keep-alive reaper closes idle sockets after ~5s. Agent workloads idle between sandbox commands (model think time), so commands routinely reuse a socket the server already closed — surfacing as `httpx.RemoteProtocolError` → `SandboxInternalException`, which permanently kills the rollout. **Fix:** the provider injects a transport whose `keepalive_expiry` sits *below* the server's keep-alive timeout (default 3s, `connection.keepalive_expiry_s`): - `connection.transport_backend: httpx` (default) — stock `httpx.AsyncHTTPTransport(limits=..., retries=connect_retries)`. No new required dependencies. - `connection.transport_backend: aiohttp` (opt-in) — aiohttp pool under the SDK's httpx surface via [`httpx-aiohttp`](https://github.com/karpetrosyan/httpx-aiohttp); falls back to the httpx transport with a warning when absent. - `keepalive_expiry_s: null` disables injection entirely (SDK default transport). ### 2. 502 `could not connect to the backend sandbox endpoint='<podIP>:44772'` on first command With `create.skip_health_check: true`, `Sandbox.create` returns before the pod's exec daemon is listening; the first command races pod startup and the server proxy 502s. Under a create burst this killed 7–12 rollouts per run. **Fix:** `create.skip_health_check: false` (create waits for readiness, bounded by the spec's `ready_timeout_s`; `create.timeout_s` kept above the ready timeout). **`command_retries` stays at 0.** Retrying a command the server may have already started would execute it twice, and agent commands are frequently mutating. The keepalive bound removes the stale-connection failures retries were compensating for. Raise it only for idempotent workloads. ### 3. Separate resource requests and limits `Sandbox.create` accepts distinct `resource` (limits) and `resource_requests` maps; a lone `resource` map is applied by the server as requests=limits. This exposes the requests side via `sandbox_spec.provider_options.resource_requests` (same keys as `SandboxSpec.resources`). Motivation: SWE-bench test suites OOM-killed sandbox pods at 2Gi, but raising a single combined map to 8Gi would 4× the cluster reservation; with the split, limits rise while requests stay small and pods pack densely. Requires `opensandbox>=0.1.15` (lower bound raised here). ### 4. Sandbox teardown off the rollout's critical path `env.cleanup()` ran in the rollout Ray task's `finally` block, so a finished result only became fetchable after the sandbox DELETE returned — and a teardown failure re-raised *over* the successful eval, degrading it to a reward-0 error row. Teardown is also effectively one-shot (`Sandbox.stop` latches `_closed` before dispatch), so a single failed DELETE leaked the sandbox while still costing the rollout. **Fix:** cleanup runs on a best-effort daemon thread; results return immediately, teardown errors are logged rather than raised, and orphans remain covered by the provider's sandbox TTL. ### 5. High-concurrency rollout delivery (`mini_swe_agent_2`) Two coupled fixes so every finished rollout is delivered at high concurrency (note: at very high concurrency the single-process policy-model proxy remains a separate bottleneck — connection-refused storms confirmed by a controlled A/B — with its scaling fix tracked separately): - **await the Ray ObjectRef** instead of `asyncio.to_thread(ray.get, ...)`: the default executor caps at `min(32, cpu+4)` threads, each pinned for a full rollout, so delivery stalls at ~32 concurrent rollouts and finished tasks queue behind blocked `ray.get` calls. - **bounded litellm retries (`num_retries=5`, config-overridable)** — no retry means one transient LLM-call failure kills a whole rollout; unbounded retries make failures look like hangs — and **`num_cpus=0.25`** on the rollout Ray task so concurrency is not capped at cluster core count (rollouts are I/O-bound). ### 6. `datasets` declared in the base agent config The struct-mode config merge rejects keys absent from the base config, so a benchmark config using `_inherit_from` could not add its dataset list (`ConfigKeyError: Key 'datasets' is not in struct`). Declared empty in the base server config, matching how other agents (e.g. `swe_agents`) expose it. ## Validation - Requests/limits split (paired multi-node runs, identical except the variable under test): infra-failed rollouts 41/316 (**13.0%**) at 2Gi requests=limits → 19/288 (**6.6%**) at 8Gi limits / 2Gi requests; pass@1 58.0% → **67.1%**. - Keepalive bound + health-checked create: zero `Server disconnected` events and zero create-race 502s across all subsequent runs (previously 2–12 per run). - Full stack, end-to-end from this branch: single-pass SWE-bench Verified (500 instances, concurrency 500+) delivered **>98% of rollouts** with ~1.6% residual sandbox-infra failures (server-side proxy 502s on established sandboxes, tracked separately) and zero client-transport failures. Before the delivery fixes (NVIDIA-NeMo#4/NVIDIA-NeMo#5), an identically shaped run lost the majority of *completed* evaluations at the wall — hundreds of finished evals, only tens recorded. ## Relationship to NVIDIA-NeMo#2020 Complementary, no overlap; NVIDIA-NeMo#2020 (job attribution metadata) has since merged and this branch is updated on top of it. Field-validated together — the attribution labels are what make post-cancellation sandbox garbage collection safely scoped to a single job. ## Testing - `tests/unit_tests/test_opensandbox_provider.py`: transport-backend coverage (httpx default with keepalive expiry, custom pool settings, fallback when `httpx_aiohttp` is unavailable, `null` disables injection), aiohttp opt-in (importorskip-guarded), and requests/limits plumbing tests. 16 passed with and without `httpx-aiohttp`. - `responses_api_agents/mini_swe_agent_2/tests/test_app.py`: updated for the awaited ObjectRef (awaitable `FakeObjectRef`); assertions strengthened to check the Ray call's params. - CI green except a pre-existing `tau2` failure on `main` (reproduces on unrelated PRs). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes RL-1013 (Linear issue). Every sandbox created through the OpenSandbox provider now automatically carries
nemo-gym.nvidia.com/team/nemo-gym.nvidia.com/user/nemo-gym.nvidia.com/workload/nemo-gym.nvidia.com/runkeys in its metadata. OpenSandbox propagates sandbox metadata as Kubernetes labels on the sandbox resources (BatchSandbox CR + pod), so sandboxes are attributable both through the OpenSandbox list API and directly at the cluster level:Why metadata (SDK research)
From the
opensandboxSDK (0.1.9) source, there are exactly three candidate carriers for attribution:Sandbox.create(metadata=...)opensandbox.io/prefix reserved for the platform itself), is queryable viaSandboxManager.list_sandbox_infos(SandboxFilter(metadata={...})), and is patchable post-create (PATCH /v1/sandboxes/{id}/metadata).extensionsConnectionConfig.headers/user_agent/api_keyK8s annotations are not settable through the SDK/API at all — labels are the only k8s-level channel it propagates. Keys use the Kubernetes prefixed-key convention (
nemo-gym.nvidia.com/) for namespacing and provenance, matching the existing<project>.nvidia.com/label-key convention used by run-level sandbox tooling; the prefix is configurable (key_prefix: ""restores bareteam/user/workloadkeys).The provider already sanitizes metadata values to K8s label rules (
_metadata_value: charset replacement + 63-char truncation) and pipesspec.metadataintoSandbox.create, so this change only adds the automatic injection.Design
nemo_gym/sandbox/attribution.py(shared, provider-agnostic so docker/openshell/ecs can adopt it later):resolve_attribution()resolves each field in order — explicit config →NEMO_GYM_TEAM/NEMO_GYM_USER/NEMO_GYM_WORKLOADenv vars → Slurm job env vars (SLURM_JOB_ACCOUNT/SLURM_JOB_USER/SLURM_JOB_NAME) → OS login name (useronly;rootis ignored since containers default to it) → the gym CLI'sNEMO_GYM_CONFIG_PATHserver instance name (workloadonly). Unresolvable fields are omitted, never guessed. Arunkey (NEMO_GYM_RUN_ID, else generated once per process and logged at first create) scopes sandboxes to one launch so an interrupted run's sandboxes can be listed and garbage-collected exactly.OpenSandboxProvider: newattributionconfig group (enabled: trueby default;team/user/workloadoverrides;key_prefixfor the label-key namespace).create()merges attribution under the spec's metadata, so explicitsandbox_spec.metadata/default_metadatakeys always win; values then flow through the existing K8s-label sanitization.configs/opensandbox.yamlandfern/.../sandbox/opensandbox.mdx(new Job Attribution section).Testing
tests/unit_tests/test_sandbox_attribution.py: resolution precedence (config > NEMO_GYM_* > Slurm > login name), blank-value handling, omission when unresolvable,os.environdefault.tests/unit_tests/test_opensandbox_provider.py: attribution lands in the SDKcreatecall'smetadatawith prefixed keys (and sanitization applied, e.g.nemo rl→nemo_rl), explicit spec metadata beats attribution, config overrides beat env detection,enabled: falsedisables injection, andkey_prefixvariants (bare, custom, trailing-slash normalization).main.🤖 Generated with Claude Code