Add opencode_hf_sandbox.py: loop-owning training in remote HF sandboxes

[sergiopaniego]

What does this PR do?

Adds examples/scripts/openenv/opencode_hf_sandbox.py, a sibling of opencode.py that runs each loop-owning rollout in its own remote Hugging Face sandbox (HFSandboxBackend) instead of a local subprocess, so rollouts scale out beyond a single node.

It reuses the same DeepCoder task, held-out verifier and hooks as opencode.py. The delta is where the rollout runs: a pre-baked remote sandbox image, and a two-URL topology (trainer to vLLM on localhost for NCCL weight-sync, remote sandboxes reach the same vLLM through a public URL). It also uses a dense terminal reward (fraction of held-out tests passed) and drops empty-trace rollouts as unscorable, so a flaky remote sandbox does not poison the group baseline. Adds the example to example_overview.md and a note in the openenv.md guide.

The example imports HFSandboxBackend / SandboxHandle from opencode_env.sandbox, which is already on OpenEnv main, so it runs against plain OpenEnv main with no extra branch.

Validated end to end on HF Jobs (h200x2): Qwen3-8B, 10 steps, one remote HF sandbox per rollout, with a public trackio dashboard and a pushed model.

AI writing disclosure

• AI-assisted

cc @AmineDiro

---

Note

Low Risk
Documentation and a self-contained example script only; no changes to TRL training or library APIs.

Overview
Adds examples/scripts/openenv/opencode_hf_sandbox.py, a scale-out variant of the loop-owning opencode AsyncGRPO example: each rollout runs in its own remote Hugging Face sandbox (HFSandboxBackend) instead of a local subprocess, with a pre-baked image default and optional cold-install via --sandbox-image.

The script documents a two-URL setup—local --vllm-url for the trainer (NCCL weight sync) and required public --sandbox-vllm-url for in-sandbox proxies—and wires the same DeepCoder held-out verifier, dense test-fraction reward, and trace-filtering hooks into HarnessRolloutWorker.

Docs: example_overview.md and openenv.md now point readers to this example for multi-node rollout scaling.

^{Reviewed by Cursor Bugbot for commit 0746667. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sergiopaniego]

Ran end to end on HF Jobs (h200x2): Qwen3-8B, 10 steps, one remote HF sandbox per rollout. The reward climbs over the run, so the loop-owning path trains on the captured tokens as intended.

• Trackio dashboard: https://huggingface.co/spaces/sergiopaniego/opencode-hf-sandbox
• Trained model: https://huggingface.co/sergiopaniego/Qwen3-8B-opencode-deepcoder-grpo
• HF Jobs launcher: https://gist.github.com/sergiopaniego/ff9622406a2298b4f84300392bead2c2

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[AmineDiro]

🔥 🔥

Niiice to support hf-sandbox 👍🏼