feat: Switchyard OpenCode integration - #2080
Open
drifold wants to merge 12 commits into
Open
Conversation
Adds opencode_source='opencode': runs upstream opencode headless with an opencode.json provider pointed at Switchyard (no fork), captures the session tree via opencode's native correlation headers (X-Session-Id / x-parent-session-id), and reconstructs root and subagent rollouts from the retrieved sessions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>
…mparison OpenCode re-serializes tool-call arguments differently across turns (compact vs. pretty-printed JSON), which failed the message-history extension check even though the tokens were contiguous. Canonicalize arguments for the comparison only; reconstructed training items keep the model's original arguments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>
…system-prompt instructions Signed-off-by: drifold <drifold@nvidia.com>
nemo-automation-bot
Bot
added
the
community-request
Adds switchyard_spawn_routing_profile: when set, each agent run spawns one dedicated token-capture Switchyard instance on a free port (stateful token injection requires all of a session's calls to reach one process), scopes records under the run's persistent_dir, and reaps the instance after trace retrieval in run(). Mutually exclusive with switchyard_base_url; behavior is unchanged when absent. The agent script is built after the instance URL is known, since it embeds the Switchyard routing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>
github-actions
Bot
added
the
sla:triage-overdue
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
11 tasks
…nfigs as nv-opencode Signed-off-by: drifold <drifold@nvidia.com>
…ed context length Fetch the served model's max_model_len from Switchyard's /v1/models (surfaced there for token-capture routes) before building the opencode agent command, and declare limit.context/limit.output on the generated provider entry. Without limits, opencode requests a flat 32000 max_tokens and never compacts (limit.context defaults to 0); with them it budgets like any registry-known model. Output is reserved as min(32000, context // 4) so the compaction threshold (context - output) keeps most of the window for history. Fetch failure degrades to today's limit-less behavior with a warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: drifold <drifold@nvidia.com>
…ends Distribute per-rollout Switchyard spawns across all vLLM generation servers using a round-robin counter (_switchyard_spawn_count % N). Previously all spawns used policy_base_url[0], leaving servers 1-15 idle while server 0 was saturated, causing 3600s timeouts for all concurrent rollouts. Signed-off-by: drifold <drifold@nvidia.com>
Three fixes needed when running 1000+ concurrent rollouts on a single node: 1. Attribute fix: _spawn_switchyard used self.config.ng_global_config_dict_str (SWEBenchWrapperConfig, request-level) instead of params.ng_global_config_dict_str (SWEBenchWrapperInstanceConfig, which carries the server-level global config). This caused AttributeError -> 500 on every /run call. 2. SWITCHYARD_POLICY_MODEL: setting SWITCHYARD_VLLM_BASE_URL via spawn_env caused the wrapper to skip its NEMO_GYM_CONFIG_DICT lookup entirely, leaving SWITCHYARD_POLICY_MODEL unset. Fix: wrapper now derives the model name independently, regardless of whether the URL was pre-set by Gym. 3. Concurrency: 1024 simultaneous Switchyard spawns on one node caused ClientOSError (port exhaustion) and ServerDisconnectedError (uvicorn connection saturation, each /run held open ~168s during spawn+wait). Fix: asyncio.Semaphore(64) limits concurrent spawns; TCP socket probe replaces HTTP /v1/models polling (pure kernel-level, no aiohttp overhead). Signed-off-by: drifold <drifold@nvidia.com>
drifold
force-pushed
the
switchyard-opencode-integration
branch
from
July 29, 2026 08:23
ec12517 to
8afdd31
Compare
… of discarding all of it
reconstruct_switchyard_rollout validates each captured record as it rebuilds a
rollout, and raised on the first record that failed — discarding every record
already validated. A discarded session yields a rollout with no generation
tokens, which makes NeMo-RL reject and retry the sample's entire 16-rollout
prompt group. Breaks are not rare: context compaction legitimately stops later
records from extending earlier ones (~4% of sessions, measured over 300
production traces), and at that rate group-level rejection destroyed 97% of
groups — a 4-hour run produced zero training steps.
Now a mid-session failure returns the already-validated prefix — a complete,
token-exact rollout up to the break — with `partial_reason` set. Callers mask
the sample: real tokens in the batch, zero loss contribution, and the other 15
samples train normally. This was always the intended semantics ("emit what is
trustworthy, mask the sample" — the compaction degrade path already worked this
way); raise-and-discard was the inconsistency.
Deliberate boundaries: a failure on the FIRST record still raises (nothing is
certified, so there is nothing honest to return), and allow_partial=False gives
fail-fast for analysis and the validation unit tests. Partial is the default
because the training pipeline is the primary caller.
Signed-off-by: drifold <drifold@nvidia.com>
…n the rollout's own node; drop shared mode Distribution. Previously the Gym server node hosted every per-rollout Switchyard (~1024 x 94MB at recipe concurrency) and later every trace retrieval — hundreds of Lustre record reads plus CPU-bound token reconstruction per rollout, executed on the server's single event loop. At scale that starved response delivery outright: 2,600 rollouts finished server-side while ~200 responses reached the trainer in 2h and the replay buffer never left zero. Both now live in the rollout's Ray task (scheduling_strategy=SPREAD): the proxy spawns on the agent's node, and after teardown — when the record set is complete by construction — the task reads and reconstructs its own trace and returns it in the task result. The server keeps only in-memory work: applying the trace to the response, masking, and the round-robin backend cursor (carried to the task via switchyard_backend_url). Crossing the server->Ray-worker boundary surfaced three environment assumptions, each fixed here and each caught at full scale first: - the gym venv's bin/ is not on a Ray worker's PATH: resolve the switchyard CLI next to the running interpreter; - NEMO_GYM_CONFIG_DICT does not inherit: set it at the task entry point (the venv's switchyard wrapper derives SWITCHYARD_POLICY_MODEL from it, and get_global_config_dict() otherwise falls through to parsing Hydra CLI args); - proxy readiness now waits 240s, not 60: under Lustre load the CLI's imports alone stall past 60s (measured p90=46s with the distribution truncated at the old timeout), and every timeout kill triggered a 16-rollout group retry — an amplification loop. Dead-on-arrival proxies still fail fast via the process-exit check. Retrieval consumes the longest-valid-prefix semantics from nemo_gym: partial or degraded traces (context compaction) emit real tokens plus mask_sample=True, so a sample excludes itself from the loss without aborting its prompt group. Shared mode (an externally managed proxy at switchyard_base_url, records fetched over HTTP) is removed: a dead proxy strands its tokens behind an unreachable endpoint — retried without bound — which is the failure mode disk retrieval exists to eliminate, and per-rollout spawn is the only configuration injection supports anyway. The instance-level field survives as switchyard_spawned_base_url, internal plumbing from the task's spawn to the harness command builder. Signed-off-by: drifold <drifold@nvidia.com>
Runs the per-rollout Switchyard lifecycle inside a REAL Ray task with the production decorator options — deliberately unmocked. The existing spawn tests mock create_subprocess_exec, which verifies argv but cannot fail on what broke at scale three times: state the Gym server process has that a Ray worker does not (PATH, environment, sys.path). One task round-trip covers CLI resolution without PATH (with a negative control that reports the test inconclusive rather than vacuously green when the worker has the CLI on PATH), proxy readiness (bound + accepting), params crossing the boundary exactly as production sends them (model_dump -> model_validate), task-side trace reconstruction from seeded records, payload survival through the Ray object store, and clean teardown. Skips gracefully where ray or the CLI is unavailable. ~33s wall when fully exercised; reviewers may prefer an integration marker for per-commit CI. Signed-off-by: drifold <drifold@nvidia.com>
drifold
force-pushed
the
switchyard-opencode-integration
branch
from
August 3, 2026 09:30
510bf3f to
afd76b8
Compare
…ports its env
The spawned proxy's config previously lived outside the repo: a routing
profile with hardcoded parsers, plus a wrapper script installed over the
venv's switchyard entrypoint to derive the policy endpoint/model and put
the serving vLLM on PYTHONPATH. Rebuilding the venv silently deleted the
wrapper and broke training.
- switchyard_profile.yaml: shipped, deployment-independent profile. The
endpoint, model, and parsers resolve from ${...} env vars; sampling
params stay numeric literals (env interpolation is string-only). The
default/policy_model routes are aliases of one route: dispatch is an
exact match on the request's model id, and upstream opencode POSTs
"default" while the OpenHands fork POSTs the model server's name.
- _spawn_switchyard_local now exports everything the profile references:
endpoint/model derived from the run's config, parsers and PYTHONPATH
from three new config fields (switchyard_tool_parser,
switchyard_reasoning_parser, switchyard_parser_pythonpath).
Signed-off-by: drifold <drifold@nvidia.com>
linj-glitch
added a commit
that referenced
this pull request
Aug 18, 2026
Serves Gym model calls through a Switchyard routing proxy, so any benchmark
Gym already supports can be evaluated against a router without touching
harness code:
gym eval run --benchmark <name> --model-type switchyard_model
Switchyard is integrated at the model layer rather than the agent layer. The
existing Switchyard work in Gym (#2026 token capture, #2080 OpenCode) wires
one harness at a time under responses_api_agents/swe_agents; putting the
router behind a Responses API model instead makes it available to every
benchmark at once, which is what evaluating the router itself requires.
Two modes. Attach (default) points at a proxy run outside Gym, which keeps
proxy lifecycle and image pinning with the eval harness so a run can be
compared against a scaled-evals run of the same Switchyard commit. Launch
starts the proxy from a routing profile and shuts it down at exit, for a
single-command run.
Gym's rollout-attempt id is forwarded as Switchyard's opaque session id, so
proxy-side routing decisions and costs join back to the rollout that produced
them. The correlation prefix is stripped by Gym's capture middleware before
handlers run, so a thin outer middleware republishes the id for the request.
The nemo-switchyard dependency is scoped to this package rather than Gym's
core dependencies: only runs that route through Switchyard pay for it, and
attach mode needs no dependency at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2026 (base:
feature/switchyard-openhands-integration). Adds upstream opencode as a second harness for the Switchyard token-capture integration — same zero-fork principle as the OpenHands path — plus per-run Switchyard instance spawning for token injection.What
opencode run) with anopencode.jsonprovider pointed at Switchyard — no fork, no harness modification.X-Session-Idper (sub)agent session,x-parent-session-idlinking subagents to the root.subagent_trajectories.switchyard_spawn_routing_profilespawns one dedicated token-capture Switchyard instance per agent run on a free port — stateful token injection (feat: token-level capture for RL training (Switchyard ↔ NeMo Gym) Switchyard#63) requires all of a session's calls to reach one process — and reaps it after trace retrieval inrun().Zero-fork architecture
opencode.jsonprovider id is deliberately notopencode-prefixed — opencode only emits its native correlation headers for non-opencodeproviders; those headers are the whole capture contract.GET /v1/sessions(no harness logsread).
Activation and safety
opencode_source: nv-opencodeopencode_source: opencode, now default+ subagents reconstructswitchyard_trace_errorsetswitchyard_spawn_routing_profileabsentpersistent_dir, instance reaped after retrievalChanges
responses_api_agents/swe_agents/app.py—opencovider JSON), the opencode run command,_list_switchyard_sessions/_reconstruct_sessionssk_id steering via system-prompt instructions (worksaround an opencode quirk where it invents
task_ids).responses_api_agents/swe_agents/app.py—switchyard_spawn_routing_profile/switchyard_spawn_hostconfig (mutually exclusivewith
switchyard_base_url, validated fail-fast), specycle, agent-script build deferred until theinstance URL is known.responses_api_agents/swe_agents/app.py—openco toopencode` (upstream); config comment/docs updatedaccordingly.
responses_api_agents/swe_agents/configs/— splitnch_opencode.yaml/swebench_opencode_training.yamlnow target **upstream** opencode (opencode_source: gs move toswebench_nv_opencode.yaml/swebench_nv_opencode_training.yaml(`opencode_soured). Root README table regenerated.nemo_gym/switchyard_trace.py— canonicalize toolstory-extension comparison only (opencodere-serializes compact vs. pretty-printed JSON across turns); reconstructed items keep the model's original arguments.test_app.pyand`, plus 7 spawn-lifecycle tests (config conflicts,session minting + script rebuild, spawn/teardown with mocked subprocess, readiness-failure reaping).Dependencies
Requires the Switchyard side on NVIDIA-NeMo/Switchya
7b33c2c(native X-Session-Id capture),be0b2c5r the opencode capture path.7f20b2a,940b767,8307fd9(token-continuity ion use-case; spawn mode itself works withcapture-only).Testing
swe_agentsand `switchyardfidelity, 0% masked; with token injection enabled, buous across multi-turn reasoning + tool-calling
conversations (0 fallbacks), and an OpenHands-pattertiguity verification.
Changes from the live description are only: the openthe first two behavior-table rows (fork now keyed onnv-opencode; upstream noted as default), and two addault flip + config split). Everything else — including the - bullets, headings, dependency SHAs, and the vLs unchanged.