feat(observability): return the environment session id and allow correlation without full capture

[waple0820]

Closes #2610

Builds on #2611 (both touch BaseVerifyResponse); please review that one first.

#2114 landed the rollout correlation contract, but two gaps keep incident attribution manual.

It is one-directional. BaseSeedSessionResponse is an empty model and BaseVerifyResponse carries no identifier, so the environment-side handle — the container, browser context or provider session that actually consumed quota — never reaches the training side. #2122 states this explicitly: "No rollout identifier is added to public request schemas."

It is gated on model-call capture. server_utils.py:341-348 only applies the rollout prefix to resources servers when observability_enabled is true, and that flag defaults off, so current_rollout_id() is None inside resources servers exactly in the runs where an incident happens.

What this changes

class BaseSeedSessionResponse(BaseModel):
    env_session_id: Optional[str] = None   # container / browser context / provider session

class BaseVerifyResponse(BaseVerifyRequest):
    env_session_id: Optional[str] = None   # same handle, returned with the score

plus a rollout_correlation_enabled global-config key that turns on the resources-server prefix without turning on capture. It defaults to false, so nothing changes for existing runs — happy to flip the default if you would rather have correlation on by default.

Why it matters

When a run degrades, the training side has rollout ids, the provider has environment session ids, and the only join key is a timestamp. Reconstructing which orphaned sessions belonged to which aborted rollouts across our own incident (1172 of 1280 rollouts over 20 steps) was done by hand; with a returned session id it would have been a groupby. This is also the environment-side half of #238.

No training framework needs a transport change: verl and NeMo-RL already carry the whole verify response as full_result.

Terminology note: this is exclusively the environment session created by /seed_session, not the vLLM router KV-cache affinity of #2570 / #2347 / #2369.

Tests

Round trip through seed and verify, plus test_rollout_correlation.py: correlation is off by default, and the new flag prefixes resources-server calls without full observability.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[waple0820]

Rebased onto main and out of draft.

This PR used to be stacked on #2611 and carried its commit, so the diff looked larger than the change. Unstacked — the two are independent now and can land in either order.

One connection to the error-handling design in #2750 that was not in the original description: the ReplayPolicy contract there puts POST at before_delivery by default, which is right for a request that performs work. But /seed_session allocates a resource, and under that rule a response lost on the way back leaves the session alive with the caller holding no id for it — nothing a release or a sweeper can act on. A caller-minted env_session_id makes /seed_session eligible for deduplicated instead: replay is safe, and a lost response is still releasable. Detail in #2750 (comment).

The field here is optional and defaults to absent, so it does not require that policy change — it just makes it possible.