Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,26 @@ filesystem also cannot see `is_incomplete`, so it would train on a rollout that
knowing, which is the failure this whole path exists to prevent. Serving that flag alongside the
records is what a remote read path needs before it is safe to have one.

## How a rollout's calls are linked

By default the calls of a rollout are chained by their token prefixes: a call is parented to the
earlier call whose prompt plus generation is the longest prefix of this call's prompt. That is
correct, and it has one blind spot. If a harness retries a call, both attempts share a prompt and
differ only in what they generated, so inference cannot tell which one the harness kept.

The model server therefore identifies the parent at request time instead. A harness must echo the
conversation to continue it, so hashing the model-authored turns of an incoming request finds the
call that produced the last one. Nothing is added to the wire and nothing depends on the harness
preserving a field Gym invented.

Two things are checked before a match is used. It must be unique, because two calls that produced
byte-identical output cannot be told apart. And the conversation those turns arrived with must still
match what that call was sent, which is what rules out a compacted or summarized history whose model
output is unchanged.

A miss is not a problem: the builder falls back to prefix inference, which is what it did before.
`parent_link_fallbacks` on the per-rollout metrics reports how often that happened and why.

### Driving rollouts yourself

`gym eval run` finalizes each record for you. A framework that calls `run_examples` directly does
Expand Down Expand Up @@ -244,6 +264,24 @@ underscores, starting with a letter or digit. An id outside that is refused rath

## Extensions

### Prefix supply

By default the inference engine builds each prompt by re-rendering the whole conversation through the
chat template. That can produce a different sequence than the model sampled. A tool-call parser
truncates the assistant turn at the tool call and discards anything generated after it; a chat
template can re-tokenize an assistant turn differently; a reasoning model's template may drop earlier
thinking entirely. In each case the new prompt no longer extends the previous one and the rollout
cannot be chained.

Prefix supply sends the engine the exact tokens of the call being continued, so the new prompt
extends the previous one by construction. Enable it by overlaying
`responses_api_models/vllm_model/configs/vllm_model_supply_prefix.yaml`.

It requires a backend that honours `required_prefix_token_ids` on both the chat and the tokenize
endpoints, and it applies only when a parent resolved. Otherwise the request is forwarded unchanged
and starts a new chain, which costs a cold cache but is never wrong. Each record carries whether its
prefix was supplied, so the rate is auditable after a run.

### Sampling pin

`sampling_overrides` on the model server forces the sampling parameters on every request, overriding
Expand All @@ -260,3 +298,8 @@ contract that accepts one.
context-compaction summary. Those are real policy output and are currently trained on. A compaction
summary is long enough that it can outweigh the rollout it summarizes, which shows up as
`chains_per_rollout_mean` above 1 and `delivered_fraction_mean` below 1.0.

**Multiple server workers.** The index that identifies which call a request continues is per process.
With more than one worker, calls from one rollout can be handled by different workers, and those
calls fall back to inferring the parent from token prefixes. Inference is correct but cannot
disambiguate a retry, and prefix supply does not apply to them.
Loading