Skip to content

Add skiplist and more robust calculation on val - #965

Merged
yueming-yuan merged 1 commit into
mainfrom
fix/skip_weight_version_val
Apr 9, 2026
Merged

Add skiplist and more robust calculation on val#965
yueming-yuan merged 1 commit into
mainfrom
fix/skip_weight_version_val

Conversation

@maocheng23

Copy link
Copy Markdown
Contributor

Summary

  • Add weight_versions and metadata to the skiplist in log_rollout_data to avoid logging non-metric fields
  • Handle nested lists (e.g. from async rollout) by flattening before averaging
  • Skip non-numeric values gracefully instead of crashing

Test plan

  • Verify logging works correctly with fully async rollout (which produces weight_versions and metadata fields)
  • Confirm no regression in standard rollout logging

🤖 Generated with Claude Code

@yueming-yuan
yueming-yuan merged commit 8d66ac1 into main Apr 9, 2026
4 checks passed
@yueming-yuan
yueming-yuan deleted the fix/skip_weight_version_val branch April 9, 2026 22:21

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the rollout data logging utility to include 'weight_versions' and 'metadata' in the exclusion list and improves the handling of nested lists during data processing. The reviewer identified potential runtime errors in the new logic, including risks of IndexError, TypeError, and ZeroDivisionError, and provided a more robust implementation to filter numeric items and safely calculate averages.

Comment on lines +157 to +163
flat = val
if isinstance(val[0], (list, tuple)):
flat = [x for sublist in val for x in sublist]
# Skip non-numeric values (e.g. strings from async rollout metadata)
if flat and not isinstance(flat[0], (int, float)):
continue
val = sum(flat) / len(flat)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current logic for processing list values has a few potential issues that could lead to runtime errors:

  • IndexError: isinstance(val[0], ...) on line 158 will fail if val is an empty list.
  • TypeError: sum(flat) on line 163 will fail if flat contains a mix of numeric and non-numeric types (e.g., [1, 2, 'a']), as the type check on line 161 only inspects the first element.
  • ZeroDivisionError: len(flat) on line 163 can be zero if val is an empty list or a list of empty lists, leading to a division by zero.

To make this more robust, I suggest checking if the list is empty before accessing elements, filtering all items for numeric types, and then calculating the average only if there are numeric items.

flat = val
if val and isinstance(val[0], (list, tuple)):
    flat = [x for sublist in val for x in sublist]
numeric_items = [item for item in flat if isinstance(item, (int, float))]
if not numeric_items:
    continue
val = sum(numeric_items) / len(numeric_items)

GuanxingLu pushed a commit to GuanxingLu/miles that referenced this pull request Apr 21, 2026
DavidBellamy added a commit to LLM360/miles that referenced this pull request Apr 21, 2026
…region clusters (#10)

* Revert "[BUGFIX] [P2PRDMA] Add rollout post-processing after P2PRDMA weight updates" (radixark#882)

* [Fix] fix ci (radixark#894)

* Avoid threading for ray getting object (radixark#886)

* Add explicit errors for unsupported Megatron profiles (radixark#887)

* Add nvfp4 quantizer files (radixark#907)

* Bump flash-linear-attention version to 0.4.2 (radixark#892)

* [BUGFIX] Invoke "post_process_quantization" by default after weight updating (radixark#890)

Co-authored-by: Yueming Yuan <yym022502@gmail.com>

* Add heartbeat and id to session server (radixark#866)

* fix: adding thin glm5 image to docker build + latest tag sync (radixark#871)

* Add consistent hashing routing policy for rollout (radixark#891)

Co-authored-by: Yueming Yuan <yueming@Mac.attlocal.net>

* [example] add retool v2 example with multi-turn framework interfaces (radixark#654)

Co-authored-by: GuanxingLu <gxlu02@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Expose rollout-batch-size, n-samples-per-prompt, global-batch-size as CLI args in swe-agent-v2 (radixark#954)

Co-authored-by: Shi Dong <shi.dong@radixark.ai>

* chore: remove obsolete swe-agent server.py and run-qwen3.sh (radixark#952)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add weight staleness control for fully async rollout (radixark#958)

* Fix/pause generation mode (radixark#924)

Co-authored-by: Yueming Yuan <yym022502@gmail.com>

* [v0.5.10][1] Bump sglang to v0.5.10 (radixark#898)

* [v0.5.10][2] Fix apply_chat_template behavior for transformers >=5.0 (radixark#926)

Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [v0.5.10][3] Fix processor return_tensors duplicate kwarg for transformers >=5.0 (radixark#927)

Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [v0.5.10][4] Fix _no_split_modules set not subscriptable in transformers >=5.0 (radixark#931)

* [v0.5.10][5] Disable piecewise cuda graph to avoid NVLS oom (radixark#935)

* [v0.5.10][6][FSDP] fix outdated weight update logic in FSDP (radixark#948)

Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [v0.5.10][7][FSDP] move FSDP to experimental and disable by default (radixark#961)

* Add skiplist and more robust calculation on val (radixark#965)

* [fix] tiny fix debug rollout only in weight version check (radixark#967)

* feat: real cp support with relayout fix for qwen3.5 train/rollout mismatch (radixark#885)

* [AMD] Upgrade to sglv0.5.10 (radixark#973)

* switch model to actor (radixark#756)

* [fix] support general logic to bypass fp32 downcast and fix qwen35 A_log dtype (radixark#975)

Co-authored-by: yueming-yuan <yym022502@gmail.com>

* fix: populate prefix_cache_info in OpenAI/session rollout path (radixark#960)

* Remove prepare_harbor_tasks.py; use harbor-private adapters (radixark#982)

* [fix] Skip flush_cache in in_place mode and add fully async example (radixark#974)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* GLM47 full cmd for async and sync reasoning (radixark#986)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: handle non-tool appended messages in TITO incremental tokenization (radixark#949)

Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>

* [docker] Add sgl-model-gateway install and download .tar.gz assets (radixark#895)

* [ci] fix hf rate limit error by caching tokenizer loading (radixark#1014)

Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>

* Use load_generate_function in legacy sglang_rollout path (radixark#1016)

* Update CODEOWNERS to add new reviewers (radixark#1021)

* Support moe lora for gpt-oss (radixark#798)

Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>

* [fix] restore expert_bias to fp32 before bridge weight export (radixark#811)

* [chore] drop legacy transformers upgrade pin for glm47-flash and qwen35 (radixark#1018)

* [fix] Enforce param dtype before wrap ddp (radixark#992)

Co-authored-by: Zhichen Zeng <zczeng@uw.edu>

* [upgrade] update Megatron-Bridge source and LoRA CI to megatron e2e tests and  (radixark#1023)

* [CI] Drop --use-miles-router from R3 tests and add r3 comparasion test between sgl & miles router (radixark#1015)

* wandb: raise init_timeout, add retry wrapper, fix shared-mode init for cross-region clusters

In online + shared mode, both `init_wandb_primary` and `init_wandb_secondary`
make HTTPS round-trips to wandb cloud (login + run create/attach). On
high-latency cross-region clusters (e.g. Abu Dhabi MBZUAI ↔ wandb-cloud
US-West) with concurrent actor bursts, a single round-trip can exceed the
wandb SDK's 90s default `init_timeout` — tearing down the whole run
with a silent handshake abort. Observed on RL360 job 1564420, which
forced `WANDB_MODE=offline` as a global default ever since (see
https://github.com/LLM360/RL360/issues/87).

The issue's original diagnosis assumed a local primary↔secondary socket
handshake race. That's not how shared mode works — per wandb's own
feature PR (wandb/wandb#6882), each writer spawns
an independent wandb-core that talks to the cloud directly; aggregation
is server-side by run_id. No local socket exists. The failure mode is
pure network/latency, not a local readiness race.

Changes
-------

- Bump `init_timeout` to 300s for primary and secondary Settings.
  Configurable via `WANDB_INIT_TIMEOUT_SECS` env var for tuning.
- Wrap both init paths in a bounded exponential-backoff retry
  (`_wandb_init_with_retry`) that re-attempts on wandb.errors.CommError
  and wandb.errors.UsageError. 3 attempts with 5→10→20s backoff by
  default, tunable via `WANDB_INIT_RETRY_ATTEMPTS` /
  `WANDB_INIT_RETRY_BACKOFF_SECS`.
- Add `x_label` tagging per wandb distributed-training docs: primary
  gets `rank_<rank>_primary`, secondaries get `rank_<rank>_secondary`.
  Enables per-rank console-log filtering in the wandb UI.
- Drop `reinit=True` from secondary init_kwargs. Shared mode natively
  supports concurrent writers on a single run; `reinit=True` triggered
  stale-state warnings on secondary actors without functional benefit.

Followups this change enables
-----------------------------

- `WANDB_MODE=offline` can be removed from scale.yaml's extra_env
  default once a pilot run confirms online mode boots cleanly.
- The tmux-based `~/bin/wandb-sync-rl360.sh` workaround on David's M2
  account becomes obsolete (no more offline-only default).
- Near-realtime wandb dashboards replace the ~2-minute-lag offline
  sync; per-rank system metrics via x_label filtering.

---------

Co-authored-by: JD <jaedon.guo@gmail.com>
Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
Co-authored-by: Ziang Li <ziangli@umich.edu>
Co-authored-by: Zhichen Zeng <zczeng@uw.edu>
Co-authored-by: JensenFire <xinji1@microsoft.com>
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
Co-authored-by: Douglas Yang <douglasyang88@gmail.com>
Co-authored-by: Yueming Yuan <yueming@Mac.attlocal.net>
Co-authored-by: Huapeng Zhou <73010314+PopSoda2002@users.noreply.github.com>
Co-authored-by: GuanxingLu <gxlu02@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Shi-Dong <Shi-Dong@users.noreply.github.com>
Co-authored-by: Shi Dong <shi.dong@radixark.ai>
Co-authored-by: Jiajun Li <48857426+guapisolo@users.noreply.github.com>
Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Yuzhen Zhou <82826991+zyzshishui@users.noreply.github.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yisheng Gong <yishenggong9437@gmail.com>
matthewyryang pushed a commit to LLM360/miles that referenced this pull request May 13, 2026
…region clusters (#10)

* Revert "[BUGFIX] [P2PRDMA] Add rollout post-processing after P2PRDMA weight updates" (radixark#882)

* [Fix] fix ci (radixark#894)

* Avoid threading for ray getting object (radixark#886)

* Add explicit errors for unsupported Megatron profiles (radixark#887)

* Add nvfp4 quantizer files (radixark#907)

* Bump flash-linear-attention version to 0.4.2 (radixark#892)

* [BUGFIX] Invoke "post_process_quantization" by default after weight updating (radixark#890)

Co-authored-by: Yueming Yuan <yym022502@gmail.com>

* Add heartbeat and id to session server (radixark#866)

* fix: adding thin glm5 image to docker build + latest tag sync (radixark#871)

* Add consistent hashing routing policy for rollout (radixark#891)

Co-authored-by: Yueming Yuan <yueming@Mac.attlocal.net>

* [example] add retool v2 example with multi-turn framework interfaces (radixark#654)

Co-authored-by: GuanxingLu <gxlu02@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Expose rollout-batch-size, n-samples-per-prompt, global-batch-size as CLI args in swe-agent-v2 (radixark#954)

Co-authored-by: Shi Dong <shi.dong@radixark.ai>

* chore: remove obsolete swe-agent server.py and run-qwen3.sh (radixark#952)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add weight staleness control for fully async rollout (radixark#958)

* Fix/pause generation mode (radixark#924)

Co-authored-by: Yueming Yuan <yym022502@gmail.com>

* [v0.5.10][1] Bump sglang to v0.5.10 (radixark#898)

* [v0.5.10][2] Fix apply_chat_template behavior for transformers >=5.0 (radixark#926)

Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [v0.5.10][3] Fix processor return_tensors duplicate kwarg for transformers >=5.0 (radixark#927)

Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [v0.5.10][4] Fix _no_split_modules set not subscriptable in transformers >=5.0 (radixark#931)

* [v0.5.10][5] Disable piecewise cuda graph to avoid NVLS oom (radixark#935)

* [v0.5.10][6][FSDP] fix outdated weight update logic in FSDP (radixark#948)

Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [v0.5.10][7][FSDP] move FSDP to experimental and disable by default (radixark#961)

* Add skiplist and more robust calculation on val (radixark#965)

* [fix] tiny fix debug rollout only in weight version check (radixark#967)

* feat: real cp support with relayout fix for qwen3.5 train/rollout mismatch (radixark#885)

* [AMD] Upgrade to sglv0.5.10 (radixark#973)

* switch model to actor (radixark#756)

* [fix] support general logic to bypass fp32 downcast and fix qwen35 A_log dtype (radixark#975)

Co-authored-by: yueming-yuan <yym022502@gmail.com>

* fix: populate prefix_cache_info in OpenAI/session rollout path (radixark#960)

* Remove prepare_harbor_tasks.py; use harbor-private adapters (radixark#982)

* [fix] Skip flush_cache in in_place mode and add fully async example (radixark#974)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* GLM47 full cmd for async and sync reasoning (radixark#986)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: handle non-tool appended messages in TITO incremental tokenization (radixark#949)

Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>

* [docker] Add sgl-model-gateway install and download .tar.gz assets (radixark#895)

* [ci] fix hf rate limit error by caching tokenizer loading (radixark#1014)

Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>

* Use load_generate_function in legacy sglang_rollout path (radixark#1016)

* Update CODEOWNERS to add new reviewers (radixark#1021)

* Support moe lora for gpt-oss (radixark#798)

Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>

* [fix] restore expert_bias to fp32 before bridge weight export (radixark#811)

* [chore] drop legacy transformers upgrade pin for glm47-flash and qwen35 (radixark#1018)

* [fix] Enforce param dtype before wrap ddp (radixark#992)

Co-authored-by: Zhichen Zeng <zczeng@uw.edu>

* [upgrade] update Megatron-Bridge source and LoRA CI to megatron e2e tests and  (radixark#1023)

* [CI] Drop --use-miles-router from R3 tests and add r3 comparasion test between sgl & miles router (radixark#1015)

* wandb: raise init_timeout, add retry wrapper, fix shared-mode init for cross-region clusters

In online + shared mode, both `init_wandb_primary` and `init_wandb_secondary`
make HTTPS round-trips to wandb cloud (login + run create/attach). On
high-latency cross-region clusters (e.g. Abu Dhabi MBZUAI ↔ wandb-cloud
US-West) with concurrent actor bursts, a single round-trip can exceed the
wandb SDK's 90s default `init_timeout` — tearing down the whole run
with a silent handshake abort. Observed on RL360 job 1564420, which
forced `WANDB_MODE=offline` as a global default ever since (see
https://github.com/LLM360/RL360/issues/87).

The issue's original diagnosis assumed a local primary↔secondary socket
handshake race. That's not how shared mode works — per wandb's own
feature PR (wandb/wandb#6882), each writer spawns
an independent wandb-core that talks to the cloud directly; aggregation
is server-side by run_id. No local socket exists. The failure mode is
pure network/latency, not a local readiness race.

Changes
-------

- Bump `init_timeout` to 300s for primary and secondary Settings.
  Configurable via `WANDB_INIT_TIMEOUT_SECS` env var for tuning.
- Wrap both init paths in a bounded exponential-backoff retry
  (`_wandb_init_with_retry`) that re-attempts on wandb.errors.CommError
  and wandb.errors.UsageError. 3 attempts with 5→10→20s backoff by
  default, tunable via `WANDB_INIT_RETRY_ATTEMPTS` /
  `WANDB_INIT_RETRY_BACKOFF_SECS`.
- Add `x_label` tagging per wandb distributed-training docs: primary
  gets `rank_<rank>_primary`, secondaries get `rank_<rank>_secondary`.
  Enables per-rank console-log filtering in the wandb UI.
- Drop `reinit=True` from secondary init_kwargs. Shared mode natively
  supports concurrent writers on a single run; `reinit=True` triggered
  stale-state warnings on secondary actors without functional benefit.

Followups this change enables
-----------------------------

- `WANDB_MODE=offline` can be removed from scale.yaml's extra_env
  default once a pilot run confirms online mode boots cleanly.
- The tmux-based `~/bin/wandb-sync-rl360.sh` workaround on David's M2
  account becomes obsolete (no more offline-only default).
- Near-realtime wandb dashboards replace the ~2-minute-lag offline
  sync; per-rank system metrics via x_label filtering.

---------

Co-authored-by: JD <jaedon.guo@gmail.com>
Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
Co-authored-by: Ziang Li <ziangli@umich.edu>
Co-authored-by: Zhichen Zeng <zczeng@uw.edu>
Co-authored-by: JensenFire <xinji1@microsoft.com>
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
Co-authored-by: Douglas Yang <douglasyang88@gmail.com>
Co-authored-by: Yueming Yuan <yueming@Mac.attlocal.net>
Co-authored-by: Huapeng Zhou <73010314+PopSoda2002@users.noreply.github.com>
Co-authored-by: GuanxingLu <gxlu02@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Shi-Dong <Shi-Dong@users.noreply.github.com>
Co-authored-by: Shi Dong <shi.dong@radixark.ai>
Co-authored-by: Jiajun Li <48857426+guapisolo@users.noreply.github.com>
Co-authored-by: guapisolo <guapisolo@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Yuzhen Zhou <82826991+zyzshishui@users.noreply.github.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yisheng Gong <yishenggong9437@gmail.com>
nblintao added a commit that referenced this pull request Jul 16, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 17, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 17, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 17, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 20, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 21, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 22, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 22, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 22, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 24, 2026
The daytona-SDK preflight closed one launch-time gap but not the other
half of the same failure mode: the per-task leg also hard-depends on
tbench2_env.task_snapshots.make_daytona / create_task_sandbox, which
only exist on the openenv #965/#972/#966 branch. An upstream-main
tbench2_env install passes the daytona check, then every episode's
sandbox start fails, the sample aborts, the group drops, and the
rollout loop refills forever — exactly the silent GPU-burning churn
the first preflight was written to prevent. Check the recipe symbols
at launch and fail with the install instruction instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 29, 2026
…pter compensation

With huggingface/OpenEnv#1012 bringing docker mode up to the canonical
scoring contract (#965/#972 had it in local mode only), both legs speak one
protocol: raw exec commands (the server resolves the task image's WORKDIR)
and the standard `evaluate` action (canonical tests/test.sh, server-side).

- delete the adapter-side compensation machinery the shared leg carried for
  older deployments: _apply_workdir, _CANONICAL_EVAL_CMD, the reward /
  test.sh-rc markers and their parsers, the OPENENV_TASK_WORKDIR /
  OPENENV_TB2_TESTS_SRC knobs, and the testsh_rc metrics plumbing.
- the native_evaluate episode-wiring parameter goes with it: legs now differ
  only in run_body (how an env comes into being) and post_episode (the
  shared server keeps its trial-dir purge; a sandbox needs none).
- runtime contract guard, replacing the source preflight that is impossible
  against a remote server: evaluate must carry the canonical harness marker
  (info.harness == "tests/test.sh"). An older deployment's bare-pytest
  reward looks valid but is untrustworthy -- it is dropped with a warning,
  so an out-of-date server surfaces as every sample dropping, never as a
  plausible reward curve. Composes with the existing error/None-reward drop.

Behavior change: episodes whose task dir ships no tests/test.sh (scored by
the server's pytest fallback, harness marker absent) are now dropped too --
all 89 official TB2 tasks and the synth pools ship test.sh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nblintao added a commit that referenced this pull request Jul 29, 2026
The clone instruction still pointed at the #965/#972 merge — enough for the
Daytona leg (local mode), but the shared docker leg scores through the
docker-mode canonical contract that only exists from 04d259ea6. And the
launcher preflight's failure message still said 'not upstream main', from
the era when the fixes lived only on a fork branch — upstream main IS the
correct install source now, as the README already says.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants