docs: add a sampling-params reference for the generation backends

[yupengtang]

What does this PR do ?

Adds docs/design-docs/sampling-params.md, a reference for how each generation backend interprets temperature / top_p / top_k.

generation.md lists them as config fields but never says how the backends read them, and they genuinely disagree, including on what "disabled" means. The page covers the two questions in #3776: how to set the params across backends, and what actually reaches each engine.

Linked from the design-docs toctree in docs/index.md and from the GenerationConfig block in docs/design-docs/generation.md.

Issues

Closes #3776.

Usage

The headline rule the page exists to state:

you write	vLLM	SGLang	TRT-LLM	Megatron
top_k: null	sent as -1, unrestricted	key omitted, unrestricted	sent as 0, unrestricted	sent as 0, unrestricted
top_k: -1	unrestricted	unrestricted	raises, require top_k >= 0	unrestricted
top_k: 0	unrestricted	rejected, top_k must be -1 (disable) or at least 1	unrestricted	unrestricted

So null is the only value that means "unrestricted" everywhere, and examples/configs/evals/eval.yaml's top_k: -1 # -1 means disable raises at request time on a TRT-LLM run. I left that config alone since it is a behaviour change rather than a docs one; happy to fix it here or in a follow-up, whichever you prefer.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Docs-only, so no tests. python -m sphinx -b html docs/ <out> builds, the new page renders with its three tables, and no warning mentions any of the three files.

Additional Information

Verification

Rebased onto main on Oct 12. Several things the page describes moved after I opened this, so I went through every statement again, against the current code and against the engines at the versions pyproject.toml pins:

• vLLM is now 0.29.0 (build: bump vllm to 0.29.0 #4080). Its SamplingParams documents top_k: 0 as "consider all tokens" and still accepts -1. The first version of this page said 0 was a literal top-k of 0 on vLLM. That is no longer true, so the null / -1 / 0 table now has one column per engine.
• SGLang rejects top_k: 0. At the pinned rev (3003d70f), SamplingParams.verify raises top_k must be -1 (disable) or at least 1, and NeMo-RL forwards a configured 0 as it is. This is the mirror of the TRT-LLM -1 case and the page now says so.
• TRT-LLM is now 1.3.0rc28 (build: bump TensorRT-LLM to feat/rl_rc (1.3.0rc28) #4358) and still raises require top_k >= 0 in its own SamplingParams, whose docstring documents 0 as "all logits". There is no such check in nemo_rl/models/generation/trtllm/.
• Megatron does not reject -1. Megatron-LM's SamplingParams turns any top_k <= 0 into 0. The first version put Megatron in the same column as TRT-LLM for that row, which was wrong.
• fix(megatron): pass training sampling defaults to the HTTP server #4516 gave the Megatron HTTP server its sampling defaults from policy.generation. The per-path table has a row for it, and there is a short new section on what each HTTP server does with values sent on the request: vLLM and TRT-LLM assert them against the config, Megatron uses the config only as a default.

Two details differ from the table in the issue:

Greedy is not uniform across all paths. Every path that takes a greedy flag resolves it to temperature=0.0, top_k=1: BaseVllmGenerationWorker._build_sampling_params, generate_text / generate_text_async, sglang_generation.py, trtllm_worker_async.py, megatron_worker.py. The HTTP servers have no greedy parameter at all; over HTTP, greedy is whatever the caller puts on the request.

SGLang drops the key. It does not send top_k: -1; it omits top_k from the request dict when the resolved value is -1.

Scope

Kept to the suggested scope: the per-path table, the null vs -1 vs 0 rule, the vLLM-HTTP top_k gap, greedy, and a pointer from generation.md, plus the HTTP request section above. It references #2053 for where top-p/top-k support came from and #3537 for the TRT-LLM direct-vs-HTTP alignment.

I did not change any behaviour. In particular the vLLM HTTP path still ignores top_k, and the page says so explicitly rather than pretending the config applies. If you would rather that path thread top_k like TRT-LLM's now does, that is a code change and I am happy to open it separately.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yupengtang]

Rebased on main. vLLM, TRT-LLM and the Megatron HTTP server all changed after I opened this, so I went through the page again against the current code; what changed is listed in the description. Happy to adjust anything once someone has a chance to look.