Repository navigation
Discard infrastructure failures in agentic training and evaluation - #7
Merged
celve merged 6 commits intoOct 21, 2026
Conversation
An eval attempt that ends ABORTED (a sandbox that never came up, a session that could not be collected, an agent function's InfraAbort) has reward None, and the default eval logger scored it 0.0, so one dead sandbox lowered the held-out score as if the policy had failed the task. When every attempt of a dataset aborted, the sample metrics crashed on round(None). The default logger now drops every sample of an aborted attempt before scoring, logs eval/<dataset>-aborted_ratio (per attempt, not per sample) and eval/<dataset>-aborted/<exit_status> counts, logs no score when nothing is left, and computes pass@k over prompts whose attempts were all scored. Datasets without samples, and samples with a missing reward that did not abort, are scored as before.
…count With aborted attempts left out, pass@k decided a prompt was complete by counting sample rows, so an attempt split into several samples counted several times. It also grouped indices by the global --n-samples-per-eval-prompt, while both eval loops number a dataset's attempts by its own resolved value, so a dataset with an override was cut into fabricated prompts. A custom eval output without sample indices crashed. Rewards are now taken once per attempt, prompts are grouped with the dataset's resolved n_samples_per_eval_prompt (the global value when no dataset config matches), and pass@k is skipped with a warning when samples carry no index. The fixed-size reshape for datasets with nothing aborted uses the dataset's value too.
An agent function had no way to say that an episode failed for a reason the policy could not have caused. Returning None, or raising, kept whatever the session recorded, and the reward model then scored it 0, so GRPO pushed down what the policy did in an episode the sandbox service broke. - InfraAbort(exit_status), in miles.rollout.agentic.agent_function: an agent function raises it to mark its sample ABORTED with the cause in metadata["exit_status"], so the group is dropped. It keeps its cause across the subprocess agent boundary. - Miles's own aborts on the session path record a cause too: NoModelCalls, CollectFailed and AllTruncated. - rollout/aborted/drop_<exit_status> counts dropped groups by cause, in both synchronous rollout loops and the fully-async buffer. - --max-consecutive-aborted-groups, 4 x the rollout batch size by default and 0 to disable, fails a synchronous rollout whose groups keep aborting, rather than drawing new prompts forever while the sandbox service is down. The interface matches Tao Lin's upstream InfraAbort pull request (2801), so the two can converge.
Harbor, in-process (the path computer-1 runs on): - A trial that failed before its agent started raises InfraAbort: SandboxUnavailable when the sandbox never became ready, AgentSetupFailed when the harness could not be installed. Harbor stamps each phase's timing when the phase begins, and a multi-step trial counts as started once any step ran. - A trial whose agent lost the model service also raises it (ServerUnreachable). That covers Harbor's typed errors for installed agents and litellm's APIConnectionError and ServiceUnavailableError for in-process ones such as computer-1. - Everything else after the agent started still scores 0, timeouts included. HUD: an episode that never ran, or recorded no model call, is now ABORTED rather than FAILED with reward 0. HUD 0.6 returns a failed run instead of raising when provisioning, connecting or starting the task fails, and remove_sample kept that 0 in its group's baseline.
- Harbor no longer discards installed agents' Api* errors. Harbor gets them by regex over the agent's own output, which the policy writes, so a policy could print a provider error to escape a low reward. Only litellm's exceptions from in-process agents remain. Multi-step trials now read step exceptions too, where Harbor records a step's model-service failure. - HUD discards a run only when it never launched (no runtime) or its first model request failed. An AgentStep alone does not prove a response: HUD records a failed request as one. - On the session path, an InfraAbort keeps its cause when collecting the samples then fails with any error, not only a timeout. - The aborted-group guard now aborts in-flight generation, agent abort hook included, before failing the rollout.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f53498151
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+889
to
+891
| "--max-consecutive-aborted-groups", | ||
| type=int, | ||
| default=None, |
There was a problem hiding this comment.
When a user passes a negative value such as --max-consecutive-aborted-groups -1, the parser accepts it, and aborted_streak_error treats it as enabled: even a successful group produces streak 0, which is not less than -1, so the rollout immediately aborts with the misleading message that zero groups aborted. Validate this launch constraint as nonnegative at startup, since only 0 and positive limits have defined behavior. .claude/rules/launch-and-request-args.mdL69-L71
Useful? React with 👍 / 👎.
Owner
Author
There was a problem hiding this comment.
Fixed in f255def: miles_validate_args now refuses a negative value at startup. 0 still disables the guard, and a test covers the default, 0, and -1.
A negative limit parsed fine and then stopped the rollout on its first group, whatever that group's outcome, with a message claiming zero groups had aborted. Only 0, which disables the guard, and positive counts mean anything.
celve
changed the base branch from
upstream/main
to
LIN-1985/investigate-existing-recipes-for
October 21, 2026 13:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An agentic episode that fails because the sandbox service or the model service broke used to train as an attempt the policy failed. GRPO then pushed down whatever the policy did in that episode. Evaluation scored the attempt 0, and crashed outright when every attempt of a dataset aborted. This PR lets harnesses discard such episodes, discards them in Miles's bundled Harbor and HUD paths, and leaves them out of evaluation.
The rule
Discard only what the policy cannot have caused. Anything it can cause stays scored, or it would learn to trigger that outcome to escape a low reward.
mainalready didmax_seq_lenChanges
Training:
InfraAbort(exit_status)inmiles.rollout.agentic.agent_function. An agent function raises it to mark its sampleABORTEDwith the cause inmetadata["exit_status"], so Miles drops the group. It keeps its cause across the subprocess agent boundary, and when collecting the session's samples then fails. ReturningNoneor raising anything else still keeps the session, scored by the reward model.NoModelCalls,CollectFailedorAllTruncated.rollout/aborted/drop_<exit_status>counts dropped groups by cause, in both synchronous loops and the fully-async buffer.--max-consecutive-aborted-groupsfails a synchronous rollout once that many groups in a row abort. It first aborts in-flight generation, agent abort hook included. The default is 4 × the rollout batch size, and 0 disables it. Before this, a sandbox outage made the loop draw new prompts forever.Examples:
computer-1runs on):SandboxUnavailableorAgentSetupFailed. Harbor stamps each phase's timing when the phase begins; a multi-step trial counts as started once any step ran.ServerUnreachable. That means litellm'sAPIConnectionErrororServiceUnavailableError, on the trial or on any step of a multi-step trial.Api*errors: Harbor derives those by regex over the agent's own output, which the policy writes.ABORTEDinstead ofFAILEDwith reward 0, whose 0 still entered the group's baseline underremove_sample. They are: an episode that never launched (HUD 0.6 returns a run with no runtime rather than raising), one whose first model request failed (HUD records that as anAgentStepwith an error and no response), and one wheretask.runraised. A graded episode keeps its real reward.Evaluation (
log_eval_rollout_data):eval/<dataset>-aborted_ratio(per attempt) andeval/<dataset>-aborted/<exit_status>counts are logged.mainit crashed onround(None).n_samples_per_eval_prompt, the value the eval loops number attempts by.Relation to upstream
The training-side interface (
InfraAbort, theexit_statusnames, the drop metric) matches Tao Lin'sradixark/miles#2801andradixark/miles#2802, so the two can converge. Both have been open since August, and#2801no longer applies tomain. For Harbor, this departs from nanjiangwill'sradixark/miles#3617. That PR discards installed agents'ApiInternalServerErrorand kin, which a policy can trigger by printing the matching text. Nothing upstream covers evaluation.Left out:
#2802, which aren't on the computer-use path.Verification
All on CPU: macOS, Python 3.12, with the CPU-CI requirements and
sglang-milessource onPYTHONPATH.main, the eval tests fail, including theround(None)crash.tests/fast/rollout,tests/fast/ray/rollout,tests/fast/examples/experimental,tests/fast/utilsandtests/fast/routerpass, except tests that fail the same way onmainon this machine: the session-server benchmark, the Anthropic conversion contract, process and worker lifecycle, Ray object-store and node-IP fixtures, and HTTP concurrency timing.InfraAbort("SandboxUnavailable"). The rollout dropped that group, refilled with the next prompt, kept its three-turn episode, and reportedrollout/aborted/drop_SandboxUnavailable: 1.AgentStep, a lost cause on HTTP collection errors, and a guard that stopped without draining. All are fixed in the last commit.