openenv/tbench2: per-task sandbox image recipe + Daytona materialization by nblintao · Pull Request #1710 · radixark/miles

nblintao · 2026-07-17T21:01:49Z

First of a three-PR stack (this → TTL/ownership hardening → #1675 rebased on top). Pure additive: nothing imports the modules yet; the openenv TB2 adapter's per-task backend (#1675) adopts them in the follow-up.

What

The build recipe for per-task TB2 sandboxes, next to its only consumer (the miles openenv TB2 adapter). TB2 is a per-task-image benchmark — every task pins its official runtime image in task.toml — so a cloud sandbox serving the env must be built per task: the official task image ⊕ a tbench2_env server layer, one layer, no DinD.

Two files, one per concern (per review):

tb2_task_recipe.py — provider-agnostic recipe: server_layer_commands() emits plain shell commands — uv-managed venv at /opt/envserver running the installed tbench2_env package's source (embedded into the build, so each sandbox runs exactly the version installed locally, no released package needed; the tbench2_env features this relies on are upstream since fix(tbench2_env): canonical test.sh scoring, real timeouts, task-image workdir huggingface/OpenEnv#965 + [fix] [test] attention_output_gate TP slice when num_kv_heads < TP #972), task directory staged from the tasks checkout's pinned-SHA GitHub tarball with solution/ excluded. Nothing Daytona-specific; the same layers can back a Dockerfile or another provider.
tb2_task_sandbox.py — Daytona materialization: create_task_sandbox() creates per-episode declaratively from the Image definition — no named snapshots, so no org snapshot quota; repeat creates hit Daytona's build cache (~1 min warm, ~10 min first build per task). Daytona doesn't run the image CMD, so it execs server_cmd() and waits for /health. A bake CLI (python tb2_task_sandbox.py) can pre-register named snapshots as a warm cache.
Verifier-asset hygiene (mirrors the official harness's stage-at-verify model): solution/ never enters the image; server_cmd() sets TB2_WITHHOLD_TESTS=1 so the server pulls tests/ into memory at reset() and keeps it off the agent-visible filesystem outside the verify window.

The embedded env source is located from the installed tbench2_env package and requires an editable/checkout install; _env_src_dir() fails fast with instructions instead of erroring mid-image-build.

Testing

pytest examples/experimental/openenv/tests/test_tb2_task_recipe.py -q → 3 passed (solution-exclusion anchored to the one task; withhold gate and staged-task default present in server_cmd()). Embedding smoke against a real editable install: source resolves and tars to ~16KB, well under Daytona's 64KB Dockerfile-line ceiling.

The recipe itself is the one already validated end-to-end on the #1675 branch: golden sweep 82/89 with 0 infra errors, full 89-task live-policy sweep with 0 systematic infra failures.

Post-reorg sanity (2026-07-17): fresh end-to-end validation of the full stack — golden regression, live TTL/label checks, API-policy episodes — is recorded in #1675's Validation section.

🤖 Generated with Claude Code

gemini-code-assist · 2026-07-17T21:01:52Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Shi-Dong

I'd actually suggest splitting the Daytona support and the Daytona-agnostic generic sandbox support into two different files (still in the same PR).

The recipe module split (review feedback on #1710) moved the provider-agnostic image recipe into tb2_task_recipe; the README and the adapter/eval docstrings still said it lives in tb2_task_sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

nblintao · 2026-07-20T02:25:03Z

I'd actually suggest splitting the Daytona support and the Daytona-agnostic generic sandbox support into two different files (still in the same PR).

Good point! Done.

The recipe module split (review feedback on #1710) moved the provider-agnostic image recipe into tb2_task_recipe; the README and the adapter/eval docstrings still said it lives in tb2_task_sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Shi-Dong

Approved with minor comments.

Shi-Dong · 2026-07-22T11:46:06Z

Perhaps rename to tb2_task_daytona.py to show that this script is for Daytona only? We might need other adapters if we switch to a different sandbox provider.

Fixed. Renamed the 2 files to tb2_sandbox_recipe and tb2_sandbox_daytona to make it more clean.

Shi-Dong · 2026-07-22T11:52:20Z

+    One uniform path for every task: download the checkout's pinned-commit
+    GitHub tarball and extract just this task. Deterministic (the SHA pins
+    the content — note: the committed tree, not uncommitted local edits) and
+    payload-free, so build commands stay far from Daytona's 64KB


Nit: you might want to remove the reference to Daytona in the docstring since this script is supposed to be sandbox-agnostic.

Shi-Dong · 2026-07-22T11:56:23Z

+
+def _dir_tar_b64(paths: list[Path], arcnames: list[str], max_bytes: int) -> str:
+    buf = io.BytesIO()
+    with tarfile.open(fileobj=buf, mode="w:gz") as tar:


Seems that tarfile automatically encodes current timestamp into the tarball, which means that every tarball produces a different base64 hash. It feels to me that this might make it impossible for Daytona to reuse images.

Good catch! Fixed.

TB2 is a per-task-image benchmark: every task pins its official runtime image in task.toml, so a cloud sandbox serving the env must be built per task — the official task image plus a tbench2_env server layer, one layer, no DinD. This module owns that recipe, miles-side, next to its only consumer (the per-task Daytona backend of the openenv TB2 adapter). Provider-agnostic core: _server_layer_commands() emits plain shell commands (uv-managed venv at /opt/envserver running the installed tbench2_env package's source, embedded into the build; task dir staged from the tasks checkout's pinned-SHA GitHub tarball with solution/ excluded), so the same layers can back a Dockerfile or another provider. Daytona materialization: create_task_sandbox() creates per-episode declaratively from the Image definition — no named snapshots, no org snapshot quota; repeat creates hit Daytona's build cache — then execs server_cmd() and waits for /health. A bake CLI can pre-register named snapshots as a warm cache. The embedded env source is located from the installed tbench2_env package and requires an editable/checkout install (pyproject.toml must be present); _env_src_dir() fails fast with instructions instead of erroring mid-build. Nothing imports this module yet; the adapter's per-task backend adopts it in a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…half Review feedback on #1710: the module mixed two concerns its own docstring already kept apart. tb2_task_recipe.py now owns everything Daytona-agnostic (server layer commands, env-source embedding, task staging, server_cmd, task.toml reading, /health polling); tb2_task_sandbox.py keeps its name, CLI, and import surface but is purely the Daytona materialization (declarative Image build, Resources, create_task_sandbox, bake). The two symbols that crossed the new module boundary go public with the move: server_layer_commands and resolve_docker_image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The tbench2_env fixes are upstream now (huggingface/OpenEnv#965 + #972); embedding the installed source remains the mechanism that guarantees the sandbox runs exactly the version validated locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review follow-ups on the per-task sandbox recipe: - _dir_tar_b64 is now byte-for-byte deterministic for identical source: gzip mtime=0 suppresses the header compression timestamp and _tar_filter zeroes per-entry mtimes/owners. The b64 is embedded in a build command, so the previous per-call drift changed the image definition on every episode create — defeating the provider build cache the declarative path relies on, and preventing pre-baked snapshots from ever matching. Guarded by a regression test. - rename the module pair to tb2_sandbox_recipe (provider-agnostic recipe) + tb2_sandbox_daytona (Daytona materialization): the subject of both is the sandbox, the trailing token is the role/provider, and future backends slot in as tb2_sandbox_<provider>. - drop the Daytona reference from _task_layer_command's docstring; the recipe module is provider-agnostic and the concrete ceiling is already documented at _MAX_INLINE_TAR_BYTES. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The recipe module split (review feedback on #1710) moved the provider-agnostic image recipe into tb2_task_recipe; the README and the adapter/eval docstrings still said it lives in tb2_task_sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…half Review feedback on #1710: the module mixed two concerns its own docstring already kept apart. tb2_task_recipe.py now owns everything Daytona-agnostic (server layer commands, env-source embedding, task staging, server_cmd, task.toml reading, /health polling); tb2_task_sandbox.py keeps its name, CLI, and import surface but is purely the Daytona materialization (declarative Image build, Resources, create_task_sandbox, bake). The two symbols that crossed the new module boundary go public with the move: server_layer_commands and resolve_docker_image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The recipe module split (review feedback on #1710) moved the provider-agnostic image recipe into tb2_task_recipe; the README and the adapter/eval docstrings still said it lives in tb2_task_sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…half Review feedback on #1710: the module mixed two concerns its own docstring already kept apart. tb2_task_recipe.py now owns everything Daytona-agnostic (server layer commands, env-source embedding, task staging, server_cmd, task.toml reading, /health polling); tb2_task_sandbox.py keeps its name, CLI, and import surface but is purely the Daytona materialization (declarative Image build, Resources, create_task_sandbox, bake). The two symbols that crossed the new module boundary go public with the move: server_layer_commands and resolve_docker_image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The recipe module split (review feedback on #1710) moved the provider-agnostic image recipe into tb2_task_recipe; the README and the adapter/eval docstrings still said it lives in tb2_task_sandbox. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This was referenced Jul 17, 2026

openenv/tbench2: orphan TTL + ownership labels for per-task sandboxes #1711

Merged

openenv/tbench2: Daytona sandbox mode — one cloud sandbox per episode, as its own agent function #1675

Merged

nblintao force-pushed the tao/tb2-sandbox-recipe branch from 293d343 to 6bf15c1 Compare July 17, 2026 21:36

nblintao requested a review from Shi-Dong July 17, 2026 21:45

Shi-Dong reviewed Jul 18, 2026

View reviewed changes

nblintao force-pushed the tao/tb2-sandbox-recipe branch from 6bf15c1 to 3726743 Compare July 20, 2026 02:21

Shi-Dong approved these changes Jul 22, 2026

View reviewed changes

nblintao and others added 4 commits July 22, 2026 09:42

nblintao force-pushed the tao/tb2-sandbox-recipe branch from eb66c44 to 3534d28 Compare July 22, 2026 13:44

nblintao merged commit 0d9e565 into main Jul 22, 2026
36 checks passed

nblintao deleted the tao/tb2-sandbox-recipe branch July 22, 2026 14:01

This was referenced Jul 28, 2026

E2B sandbox backend (E2B Cloud / self-hosted AgentENV) + dedicated AgentENV recipe #1913

Merged

openenv/tbench2: drop the daytona bake CLI, which nothing can consume #2136

Merged

Navigation Menu

Uh oh!

openenv/tbench2: per-task sandbox image recipe + Daytona materialization - #1710

openenv/tbench2: per-task sandbox image recipe + Daytona materialization#1710
nblintao merged 4 commits into
mainfrom
tao/tb2-sandbox-recipe

nblintao commented Jul 17, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jul 17, 2026

Uh oh!

Shi-Dong left a comment

Uh oh!

nblintao commented Jul 20, 2026

Uh oh!

Shi-Dong left a comment

Uh oh!

Shi-Dong Jul 22, 2026

Uh oh!

nblintao Jul 22, 2026

Uh oh!

Shi-Dong Jul 22, 2026

Uh oh!

nblintao Jul 22, 2026

Uh oh!

Shi-Dong Jul 22, 2026

Uh oh!

nblintao Jul 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nblintao commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Testing

Uh oh!

gemini-code-assist Bot commented Jul 17, 2026

Uh oh!

Shi-Dong left a comment

Choose a reason for hiding this comment

Uh oh!

nblintao commented Jul 20, 2026

Uh oh!

Shi-Dong left a comment

Choose a reason for hiding this comment

Uh oh!

Shi-Dong Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

nblintao Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Shi-Dong Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

nblintao Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Shi-Dong Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

nblintao Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nblintao commented Jul 17, 2026 •

edited

Loading

nblintao Jul 22, 2026 •

edited

Loading