Repository navigation
Add SmolDataEnvs multi-harness RL article and tutorials - #29
Merged
Merged
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3bdb81233c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+55
to
+57
| max_steps=cfg["max_steps"], num_train_epochs=1, shuffle_dataset=False, | ||
| num_generations=cfg["num_generations"], per_device_train_batch_size=1, | ||
| gradient_accumulation_steps=cfg["batch_size"] * cfg["gradient_accumulation_steps"], |
There was a problem hiding this comment.
With the default whitebox configuration, schedule() supplies 2,000 prompts while gradient accumulation consumes 16 prompts per optimizer update, so the finite schedule contains only 125 updates. In Transformers/TRL, a positive max_steps overrides num_train_epochs; setting it to the default 1,000 therefore cycles this dataset roughly eight times instead of stopping after the configured two passes. A full whitebox run consequently repeats task episodes, incurs substantially more sandbox/GPU work, and is no longer comparable to the advertised finite schedule; cap max_steps to the available updates or enforce schedule exhaustion explicitly.
Useful? React with 👍 / 👎.
adithya-s-k
changed the title
…gent The article was drafted on 04-data-agent, which is archived. This adds it as it stood there (19b727e), with its row in content/README.md, so the review rounds that follow apply on top of 05-multi-harness-rl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six comments on the Google Doc, all addressed. - Description was hard to read; it is now a TLDR in the shape he asked for: you can train a small model inside real harnesses, we did it with a 2B, here is what happened. - Opening had a synthetic quality. Rewritten conversationally, and it now names the tools people actually switch between, which is what the half-finished edit in the doc was reaching for. - Added the Anthropic case to the fine-print section. It owns both the model and Claude Code and trains one for the other, which is the cleanest statement of the asymmetry the article is about. - "Do you have results with more well known harnesses?" We do now, so the introduction names Claude Code, Codex, OpenCode and Mini-SWE-Agent where it used to say "many harnesses". - "binary you do not control" is not always true, since plenty of harnesses are a Python package or a node CLI. Now "executable". - Dropped the what-a-harness-is-not table. Every definition survives as prose and the harness/environment distinction, the only one that matters downstream, gets its own paragraph. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016err5aVhdMyM4tmveitkJW
Reconciled against the Google Doc by comparing each doc paragraph to the export base (19b727e) as well as to HEAD, so only genuine in-doc edits were taken and nothing written after the export was reverted. - Introduction opens with the doc's tighter two paragraphs, and the subhead becomes "Model cards name the harness now". - "because almost nobody runs the weights on their own", and the DeepSeek aside folded into the paragraph it comments on. - What is a harness: drops the meta lead-in and starts on the definition; the harness is "the interface" from the model's side. - Description becomes the TLDR line proposed in the doc thread. - Links the agent glossary where the neighbouring terms are distinguished (Sergio's suggestion on the "you could drop this" thread). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N24Lx2drKGJs1gwSgGXgHy
…ry sources Joel Niklaus left nine comments on the shared doc. Each one checked against the paper it concerns rather than answered from memory. - The Orchard passage was wrong in the way Joel suspected. The table said "two systems, three harnesses"; Table 9 of the paper has three systems, five cells and two task sets, and only the unseen harness collapses. The prose now separates the tax (7.5 points moving to a harness the model has seen) from the collapse (58.8 points onto one it has not), names OpenHands as the harness OpenSWE-32B trained in, and the wide table is dropped in favour of the figure, which was overflowing the prose column anyway. - The two failure modes are now named as Orchard names them, degraded resolve rate and catastrophic format failure, so the paragraph about tool names on a laptop connects to the paragraph above it. - "A major lab shipped a model that could not run in a major harness" now names DeepSeek and Terminus. - Heading loses its "this" and "it": "Frontier labs now train across harnesses on purpose". - poolside does have its own harness, `pool`, which they ship and run every evaluation in; the split between SFT on other people's harnesses and RL inside their own is now the point of that paragraph. - "wire dialect" becomes "API dialect" everywhere, defined at first use as the request format a harness speaks, with the four named. - The harness chapter no longer re-defines what the introduction just defined; an accordion in the introduction carries the short version for readers who want it immediately. - Evals now states the single-trial caveat, with poolside's four-run practice and their note about large swings in single-trial scores as the reference point. Ben and Sergio: the description is a "you can do this thing" TLDR rather than a summary of the contents, and drops "black-box" as Joel asked; the glossary link stands in for the table Ben wanted dropped. Figures: d3-harness-lockin rebuilt as the full Table 9 grid, with the unseen harness called out and the malformed-tool-call cells marked rather than scored. d3-overfit-modes now shows what each failure looks like on the wire instead of describing it abstractly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N24Lx2drKGJs1gwSgGXgHy
…d the repo
Full audit against the comment history. Four of Ben's accepted suggestions were
still only in the doc:
- the LangChain aside is gone from the harness definition
- both section headings in the harness chapter are gone ("What a harness owns,
and the one thing it does not", "The harness ecosystem today"), so the chapter
runs as one argument
- the forward reference to the proxy in the next chapter is gone
Joel marked "this section is quite confusing in the current form" resolved
himself, but the section deserved the note: "Frontier labs now train across
harnesses" now separates its two claims, that labs buy harness variety and that
buying it costs them nothing, and takes them one at a time. The poolside
paragraph gains the native behaviours Laguna lists as deliberately preserved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N24Lx2drKGJs1gwSgGXgHy
…ctorial The Seed-OSS model card was the weakest evidence in the introduction. Its two SWE-bench Verified rows differ in sampling as well as scaffold (AgentLess 4*10 is best-of-n), so the paragraph had to spend three sentences defending the example before it could use it. Table 2 of "Stop Comparing LLM Agents Without Disclosing the Harness" is the same claim with the confound removed: three models within three points of each other on a public leaderboard, three harness configurations, the same 100 SWE-bench Verified tasks, same container, same 50-step budget, same per-step timeout, two runs per cell, mean pass@1. Nothing varies but the harness. The result is sharper than the one it replaces: each of the three models is the best model under one of the three configurations. Changing the harness moves GLM-5.1 by 13.0 points while changing the model inside a fixed harness moves the score by 2.5 to 5.0, six of the nine model-pair/harness-pair comparisons are ranking reversals, and harness-induced variance is 7.8x model-induced variance. The named-harness deltas from the same paper's survey (Claude Opus 4.5 45.9 -> 55.4 on SWE-bench Pro moving SEAL -> Claude Code; Grok 4 +14-16 on SWE-bench Verified moving SWE-agent -> xAI's scaffold) follow as the observational version, which also answers Ben's instinct that the examples should involve harnesses people have heard of. Figure rebuilt to match: a slope chart with the winner of each column ringed, per-column label dodging, the cropped y range printed on the axis, and the three configurations spelled out under the legend rather than colliding under the plot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N24Lx2drKGJs1gwSgGXgHy
…eceipts Subtitle and introduction - New subtitle: harnesses change performance, you can now train with RL inside the ones people use, and improve across all of them. - Opening rewritten in plainer language. It states the problem (an open-weight model can struggle in a harness it was never trained in) and ends on the framework line, highlighted: pick a model, pick a harness, pick a sandbox, and train. - Removed the paraphrased OpenEnv quote and the unsupported claim about which harnesses labs train in. - Cut repetition across the sections (the format-failure point was made three times) and condensed the rank-reversal and Orchard passages. Every figure, quote and citation is kept. - "How badly it breaks" is now "When a model leaves the harness it was trained in", and "Model cards name the harness now" is now "Benchmark scores now come with a harness attached". - "What is missing" now frames the gap as a reliable, open way to do multi-harness training: a capture proxy on OpenEnv that records what RL needs from any harness, custom ones included, and plugs into different trainers, with Harbor on top. Adds a "What is Harbor?" accordion that links "RL Coding Environments 101: Why Harbor Exists". - Humanize pass. Prose colons down from about 12 to 6 and no semicolons. Removed mirrored contrasts, punchlines and existence announcers. Restored a citation dropped from the Opus 4.5 figures, and tied Orchard's 45.0 to 64.3 range to SWE-bench Verified. Numbers recomputed against the figure data. Paper receipts - Carousel is more compact and keeps a fixed height, so the page no longer jumps. It rotates every 5 seconds with a progress bar, pauses on hover, focus, off-screen and hidden tab, stops when a paper is picked, has a pause/play control and arrow-key tabs, and does not autoplay under prefers-reduced-motion. - Added Kimi K3 (arXiv 2607.24653, §4.2.1), Qwen3-Coder-Next (arXiv 2603.00729, §3.1.2) and MiMo-V2.6 (official PDF on Hugging Face, §4.2.5, not on arXiv yet). Each passage was verified against the PDF text, and the crops are rendered at 3x with the crop as the only edit. Provenance is in public/papers/README.md. - GLM left out. GLM-5's only relevant line names no harness, and in GLM-4.5 the same sentence means task-specific frameworks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…citations - Replace the evals, async-GRPO and ablation chapters with one training chapter built from the 24 Sep run audit: baselines, the reward and its GRPO effect, per-step training curves with a smoothing slider, pass@1 and tool/token efficiency per harness, a savings heatmap by checkpoint, exposure per run, and the Qwen3.5-2B runs in a collapsible section. - Rework the OpenEnv chapter (capture proxy, Harbor integration, serving) with new system, seams and task-browser figures; remove the Data Agent chapter. - Rewrite the conclusions and apply the humanize pass to the training and conclusions chapters. - Citation and link audit: add GRPO, vLLM, SWE-bench, SWE-bench Pro, Terminal-Bench and REINFORCE references and links to harnesses, sandboxes and docs; fix bib metadata; make quotes verbatim; correct the Harbor counts against v0.22.0. - Recolour the carried-over RL environment figures to the article palette. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arness-rl - Set a site URL so canonical and og:url no longer point at localhost, and allow a canonical override from frontmatter. - Add author, keywords, robots, image size and article tags, and a Twitter creator. - Replace the stale thumbnail with a capture of the hero, served from app/public/og. - Drop @astrojs/sitemap, which crashes on Astro 4, in favour of a static sitemap.xml. - Point the README card and deploy command at the Space's new home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each chapter built its own reference list, so the footer showed several numbered lists with duplicates. They are now merged into one list, one entry per source, sorted by author, with the back links combined. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open the article with ?review and sign in with Hugging Face to comment on text or figures, reply, react, resolve, and suggest deletions or replacements that the owner accepts or rejects. Only the owner and the REVIEWERS list can read or write. Threads are stored as JSON on a bucket mounted at /data, and review/pull_comments.py exports them to markdown. The published page never loads the review layer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…usions Add an SFT vs RL section for LFM2.5-2.6B: OpenCode and multi-harness SFT on Qwen3.8-27B's successful rollouts (FineEnvs/SmolDataEnvs-multiharness-sft), compared with the best RL checkpoints on pass@1 and tool-call savings, with two new charts. Redraw the GRPO group chart with rollouts sorted by tool calls and diverging advantage bars. Restructure the conclusions into findings, practical advice, limits and next steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most figures read theme colours once when they draw, so switching between light and dark left dark panels behind light text. They now redraw on a data-theme change. Also stop the RL animation writing to a counter that no longer exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The figure title bar covered the Comment button, leaving only a sliver of its border. Storage messages and the README now describe mounting a bucket at /data, which is how Spaces provide persistent storage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Hub stores binary files over about 100 KB as Git LFS, and the Space build copied the LFS pointer instead of the image, so the Kimi and Orchard crops and the OG image were broken on the live page. All three are now 256-colour PNGs at the same dimensions, between 54 KB and 72 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Focus the story on LFM2.5-2.6B; the Qwen baseline comparison and the Qwen
tool-call chart move into the collapsible Qwen section, and the reward
charts are labelled by reward rather than by model.
- Rename the OpenEnv chapter to "How the framework works" with a short intro,
introduce SmolDataEnvs at its first mention, move hardware to the setup.
- Define policy, agent, sandbox and benchmark once in the basics; reword the
passages reviewers found unclear; drop the OpenClaw reference and the
flagged phrasing ("on the wire", "left holding", "quietly", "honest").
- Mention Prime Intellect's renderers; give the real reason for pure sampling;
move the SGLang version note to a footnote.
- Shorten figure captions; fix the components link and the episode-control
label in the RL environment figure, and the overfit figure's row height;
label the three harness configurations in the SWE-bench figure.
- Add Joel Niklaus's harness/cost plot to the introduction.
- Add the co-authors and the Liquid AI affiliation; drop the acknowledgements.
- Link the FineEnvs Multi-harness RL collection and the seven trained models.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The article, its four review rounds, the review mode and the deploy setup, under content/articles/multi-harness-rl. Live at https://huggingface.co/spaces/AdithyaSK/multi-harness-rl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sergiopaniego
approved these changes
Oct 1, 2026
sergiopaniego
left a comment
sergiopaniego
left a comment
Collaborator
There was a problem hiding this comment.
thanks!! the tutorial reads really well, and the scripts already match the final #6947 API (factory taking sampling=, no train_turn_fn / agent_turn_fn) :) a few things from me and an agent reviewing it before we do comms:
- the main one is the native OpenCode mode. It runs on
opencode_env, which is deprecated in OpenEnv since 0.7.0 (openenv/OpenEnv#1276), with removal moving to 0.9.0 (openenv/OpenEnv#1292). TRL is also droppingasync_grpo_opencode(huggingface/trl#7457). I'd keep the mode, but say so wherever it shows up (README,train/opencode.py, the Space card and the collection note): it's deprecated, and it's still there for anyone who wants to try it. The recommended path is multi-harness through Harbor, and an OpenCode-only run is just Harbor with one harness (HARNESSES=("opencode",)), which is what the article's LFM run used. No need to rerun anything. check_setup.pyimportsopencode_envin every mode (inline below).- a few links pin this PR's branch and will 404 once it's deleted, and some RESULTS/VALIDATION links point to things readers can't open (inline below).
| | Script | Who runs the tool loop? | Trainer | | ||
| |---|---|---| | ||
| | [train/whitebox.py](train/whitebox.py) | TRL calls Python bash/SETA tools | `GRPOTrainer` | | ||
| | [train/opencode.py](train/opencode.py) | Native OpenCode runs inside a sandbox | `AsyncGRPOTrainer` | |
Collaborator
There was a problem hiding this comment.
This row and the "Then read OpenCode" step (L47) present native OpenCode as a regular path, but it runs on opencode_env, deprecated upstream (openenv/OpenEnv#1276, removal in 0.9.0 via openenv/OpenEnv#1292). Maybe mark it as deprecated here, and say the recommended path is multi-harness, where an OpenCode-only run is Harbor with a single harness (HARNESSES=("opencode",)), like the article's LFM run (L35 already hints at it).
| import json | ||
| import os | ||
|
|
||
| from opencode_env.config import OpenCodeConfig |
Collaborator
There was a problem hiding this comment.
opencode_env is deprecated since OpenEnv 0.7.0 and goes away in 0.9.0, so this Space will stop building on OpenEnv main after that. Since it stays, maybe pin it to OpenEnv v0.7.0 and add a deprecated note to its Space card pointing to the Harbor Space.
| ) | ||
| args = parser.parse_args() | ||
| from harbor_env.harness import HarborSession | ||
| from opencode_env.harness import OpenCodeSessionFactory |
Collaborator
There was a problem hiding this comment.
This import runs in every mode, so whitebox and Harbor installs print the opencode_env FutureWarning, and they'll fail once it's removed. Maybe only import it for --mode opencode / all.
| set -euo pipefail | ||
| # Run from 05-multi-harness-rl in a Python 3.12 virtual environment. | ||
| python -m pip install -r requirements.txt | ||
| python -m pip install --upgrade "trl @ git+https://github.com/huggingface/trl.git@main" |
Collaborator
There was a problem hiding this comment.
Installs TRL and OpenEnv from main (also L8-10 and envs/*/install.sh, which the Space Dockerfiles run). Recording the resolved commits is great, but a Space rebuild after an upstream change can break. Maybe pin TRL 52fb144 and OpenEnv 7ee88d5, the commits VALIDATION.md tested.
| gr.Markdown( | ||
| "This playground uses the same `BashEnvironment` as the whitebox training script. Each browser session has its own sandbox.\n\n" | ||
| "[Source and local setup](https://huggingface.co/spaces/FineEnvs/smoldataenv-multi-harness-whitebox/tree/main) · " | ||
| "[Training tutorial](https://github.com/adithya-s-k/FineEnvs/tree/05-multi-harness-rl/05-multi-harness-rl) · " |
Collaborator
There was a problem hiding this comment.
This points to tree/05-multi-harness-rl/05-multi-harness-rl, which 404s once the branch is deleted. Same in the three envs/*/README.md cards (whitebox L12, opencode L12, harbor L15). tree/main/05-multi-harness-rl and a Space redeploy would fix them.
| self._folder = Path(folder) | ||
| self._calls, self._submitted = 0, False | ||
| self._sandbox = DaytonaSandboxBackend( | ||
| image="docker.io/savatar101/env-data-agent-train:base" |
Collaborator
There was a problem hiding this comment.
The sandbox image is docker.io/savatar101/env-data-agent-train:base, a personal Docker Hub account with a mutable tag (also envs/opencode/.../environment.py:80). Moving it to an org registry and pinning a digest would make the tutorial reproducible.
| | Check | Result | Evidence | | ||
| |---|---|---| | ||
| | Local training, LFM and Qwen × all three modes | All six completed two updates and saved checkpoints | [Validation record](VALIDATION.md) | | ||
| | HF native OpenCode, LFM on A100 | Two updates; both checkpoints persisted | [HF job](https://huggingface.co/jobs/FineEnvs/6abcc3d9031314b6963440a0) | |
Collaborator
There was a problem hiding this comment.
These three HF Jobs links redirect anonymous readers to login. The whitebox one (L16, 6abcd472) shows ERROR next to "completed training and reload evaluation", which is easy to misread. Maybe drop the links or say the pilot itself failed.
|
|
||
| ## Locate an artifact | ||
|
|
||
| Our test and pilot outputs use bucket `FineEnvs/data-agent-daytona-artifacts`, under these run prefixes: |
Collaborator
There was a problem hiding this comment.
FineEnvs/data-agent-daytona-artifacts is private, so readers can't follow these prefixes (same in VALIDATION.md L77). Either make the relevant prefixes public or say it's internal.
|
|
||
| ## Evidence and remaining checks | ||
|
|
||
| Local evidence is under `experiments/tutorial-gpu-smoke-20261001/`: immutable `source-v*` snapshots, `local/` checkpoints and evaluation results, CPU UI checks, the packed-kernel probe and the real Daytona upload check. Repository-local test logs are in ignored `runs/qualification-20261001/`. |
Collaborator
There was a problem hiding this comment.
experiments/tutorial-gpu-smoke-20261001/ isn't committed. Maybe a short summary here or a link to where it lives.
Collaborator
|
one more small thing on the root |
5 of 12 tasks
adithya-s-k
changed the title
sergiopaniego
approved these changes
Oct 1, 2026
sergiopaniego
left a comment
sergiopaniego
left a comment
Collaborator
There was a problem hiding this comment.
approving, this is great!
just some small nits:
- root
README.md:05still has no### 05 · …section like 00–04, and the row only links the folder and is marked 🚧 wip. Maybe add a short section with the article and the collection, mark native OpenCode as deprecated in the row, and drop the wip once it's announced. (the collection link in my previous comment was wrong, this is the right one) - the article isn't in the root "Articles & talks" table yet, and there are two empty cells at the end where it would fit
- the tutorial links pinned to
6f2b6137(theenvs/*/README.mdcards,ui.py,training.mdx,openenv.mdx) predate thecheck_setup.pyfix. Maybe point them totree/main/05-multi-harness-rlafter merge so readers get the latest version
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion code and source for the multi-harness RL article. Three readable training scripts compare SETA whitebox, native OpenCode and Harbor using the same task split and correctness-plus-tool-efficiency reward. Each environment folder is the source for its standalone Docker Space.
The SmolDataEnvs collection links the article, environments, datasets, trained models and dashboards. The article and raw teacher rollouts now live under FineEnvs. The tutorial includes HF Jobs and Slurm commands, checkpoint evaluation, Trackio logging and historical curves with source data.
Installers use TRL main and OpenEnv main, as intended, and record the resolved commits. Native
opencode_envis deprecated and retained for the historical comparison. New runs should use Harbor, includingHARNESSES = ("opencode",)for OpenCode-only training. Whitebox and Harbor preflight no longer import native OpenCode.Validation:
VALIDATION.mdseparates those results from the outstanding HF GPU/reload checks on merged TRL main. No new GPU training is claimed by this review.Space cards and the collection mark native OpenCode as deprecated. Internal job records and private bucket paths are labelled as such. The tutorial still follows upstream main, and its standalone sandbox uses a mutable Docker Hub image; registry/digest pinning remains a reproducibility follow-up. Space concurrency settings are preserved and are not a throughput claim.