feat(vllm_model): add extra_body_override_keys for server-authoritative sampling

[GNroy]

Summary

VLLMModelConfig.extra_body supplies defaults only. Every merge site does

body_dict = extra_body | body_dict

so a value set explicitly on the request always wins. That is a sensible default, and the
/v1/completions builder documents it as deliberate ("extra_body fields do NOT override
request-level fields"). But it leaves an operator with no way to make the server authoritative
for a sampling parameter.

This PR adds an opt-in escape hatch. It does not change the default.

Why an operator would want this

When a model server generates rollouts for RL training, the sampling parameters actually used must
match the training config — otherwise the samples are off-policy but are attributed to the policy.

Agent harnesses routinely hardcode sampling parameters on internal or auxiliary calls: a title
generator or a context compressor that pins temperature low for a deterministic sub-task. Those
calls go through the same model server as the rollout itself, so the server silently emits
off-policy generations that are indistinguishable downstream.

With NeMo-RL this is not even silent. The async vLLM worker asserts that a request's temperature
and top_p match the generation config; a mismatched auxiliary call 500s, the client retries, and
once retries are exhausted the entire trajectory is discarded. In a multi-turn agentic run this
is enough to drop the large majority of rollouts in a step, so the async collector never fills one.

Worth noting top_p is already protected in practice: a harness that never sets it leaves it unset,
and model_dump(exclude_unset=True) drops unset fields, so the configured value survives the merge.
Only explicitly set values are affected.

Change

One new optional config field:

extra_body_override_keys: Optional[List[str]] = None

Keys listed here take precedence over an explicit request value:

policy_model:
  responses_api_models:
    vllm_model:
      extra_body:
        temperature: 1.0
        top_p: 1.0
      extra_body_override_keys: [temperature, top_p]

Applied at all three merge sites — the Responses-native path, the Chat-Completions path, and
_build_completion_body_from_chat_body (the /v1/completions backend). Covering only some of them
would give an operator the guarantee on one path and silently drop it on another, which is the exact
failure mode the field exists to prevent.

Blast radius

Defaults to None, so behaviour is unchanged unless the field is set. The merge is otherwise
untouched, and the configured extra_body dict is not mutated — it is shared across requests, and
there is a test for that specifically.

The VLLMModel subclasses (genrm_model, local_vllm_model, local_vllm_model_proxy) inherit the
field; it is optional, so none of them need changes.

Tests

Seven unit tests in TestMergeExtraBody, all of which fail against main and pass with the change:

• request wins by default (the existing contract, pinned so this PR cannot regress it)
• extra_body still supplies defaults for keys absent from the request
• an override key wins over an explicit request value
• override is scoped to listed keys only
• an override key absent from extra_body injects nothing
• the configured extra_body dict is not mutated
• the /v1/completions path honours overrides too

Full vllm_model suite: 112 passed. ruff check / ruff format clean.

Alternatives considered

1. Reverse the merge globally — breaks the documented "config supplies defaults" contract for
   every existing user.
2. Hardcode ("temperature", "top_p") — works, but is arbitrary from the library's point of
   view and surprising for non-RL users.
3. Fix the agent harnesses not to hardcode sampling params — the cleaner layering argument, and
   worth doing regardless. This is the defensive half: a server generating training data should be
   able to enforce its own sampling config without trusting every harness that calls it.

Questions for reviewers

• Is extra_body_override_keys the right name and shape, or would a boolean
  extra_body_is_authoritative be preferable?
• The override currently also applies to per-request metadata.extra_body overrides — whatever ends
  up in the merged extra_body wins for listed keys. Is that the behaviour you want?
• CONTRIBUTING asks for an issue before features. Happy to move this to an issue first if you would
  rather discuss the shape before reviewing code.

[ananthsub]

Hi @GNroy , this PR has similarities with #2190 - could you confirm whether 2190 would meet your requirements?

[GNroy]

Confirmed — #2190 meets our requirements. I'll close this in its favour.

One detail worth noting, since it differs from the case #2190 leads with: ours wasn't an absent parameter. The agent harness set temperature explicitly on auxiliary, non-rollout calls (context compression, title generation), so the outbound body carried the harness's value instead of the trainer's. sampling_overrides handles it because it replaces rather than only fills in — so the conflicting-parameter case is real in the wild, not just theoretical. top_k: -1 and top_p: 1.0 also match what we need.

Happy to close now, or when #2190 lands — whichever you prefer.

[ananthsub]

closing out this PR in favor of #2190 which merged