[06/06] Sampling Parameters | Documentation

docs/adding_models.rst

@@ -241,8 +241,9 @@ request, and the engine passes each config to its resource when the request is i
 There are two ``ResourceReqConfig`` subclasses:
 
 - ``SamplingReqConfig`` holds ``temperature``, ``top_k``, ``top_p``,
-  ``repetition_penalty``, ``min_p`` and ``ignore_eos``. The conductor fills in the
-  per-request seed. ``min_p`` follows the HF processor order (after the penalty and
+  ``repetition_penalty``, ``min_p``, ``penalize_prompt`` and ``ignore_eos``. The
+  conductor fills in the per-request seed, derived per resource key so each sampler of a
+  request draws its own stream. ``min_p`` follows the HF processor order (after the penalty and
   temperature, before top-k/top-p) and needs ``enable_min_p=True`` on the node's
   ``SamplerSpec``, which adds the filter to that node's captured sampler only.
   Further knobs, each needing a ``SamplerSpec`` capability (CUDA only):
@@ -268,7 +269,8 @@ Orpheus shows the simple case. It has one sampler, and reads the parameters from
 
    def get_request_resource_configs(self, partition_fwd_args, model_kwargs=None):
        model_kwargs = model_kwargs or {}
-       keys = ["temperature", "top_p", "repetition_penalty", "ignore_eos"]
+       keys = ("temperature", "top_p", "top_k", "min_p",
+               "repetition_penalty", "penalize_prompt", "ignore_eos")
        return {
            SAMPLER: SamplingReqConfig(
                **{k: model_kwargs.get(k, getattr(self.config, k)) for k in keys}
@@ -279,6 +281,25 @@ BAGEL (``bagel_model.py``) returns both config types. A BAGEL request uses
 classifier-free guidance or does not, and that choice determines which cache labels the
 request reads.
 
+Sampling knobs mean the same thing on every model (:ref:`generation-parameters`), and
+the server range-checks them before your model sees them. A model's part of that
+contract:
+
+- Default each knob the request leaves unset from the checkpoint's own
+  ``generation_config.json`` (``mstar.model.utils.load_generation_defaults``),
+  falling back to your config only when the checkpoint ships none.
+- Raise ``ValueError`` (a 400) for an input you cannot serve; never clamp, coerce, or
+  fall back silently. A ``SamplerSpec`` without ``enable_repetition_penalty`` or
+  ``enable_min_p`` refuses those knobs at admission, so do not default a penalty onto it.
+- Declare every ``model_kwargs`` key you read in ``request_kwargs()``; any other key a
+  client sends is logged as ignored by the server. ``test_model_contracts.py`` fails for a
+  registered model that does not override it.
+- Bound a request's length, or a denoise loop's step count, through ``get_loop_caps``: the
+  conductor refuses a cap past the loop's ``max_iters`` instead of cutting the request short,
+  so do not clamp it, or check it against a typed-in limit, yourself.
+- Draw any model-local randomness from the request's seed (``random_seed`` on the
+  forward-pass info), never the global RNG.
+
 ``get_sampling_config(node_name, model_kwargs)`` still exists as a helper for assembling
 sampling parameters. The engine no longer reads it directly. Pass its result into a
 ``SamplingReqConfig`` here.
@@ -556,9 +577,9 @@ The spec types are:
        ``high_freq_factor`` and ``old_context_len``. A model with learned position
        embeddings also declares this spec, because it needs the position counter. Such a
        model never applies RoPE.
-   * - ``SamplerSpec(vocab_size=..., enable_repetion_penalty=...)``
+   * - ``SamplerSpec(vocab_size=..., enable_repetition_penalty=...)``
      - A sampler with its own per-request parameter buffers, philox stream, and optional
-       seen-token mask. ``enable_repetion_penalty`` declares a capability, not an
+       seen-token mask. ``enable_repetition_penalty`` declares a capability, not an
        intention. It selects which kernel variant is recorded into the captured graph.
        Whether the penalty runs on a given step is decided from the
        ``repetition_penalty`` values of the resident requests.
@@ -592,7 +613,7 @@ appears in no spec, so it receives no resources:
            SamplerSpec(
                resource_key=SAMPLER, nodes={"LLM"},
                vocab_size=self.config.vocab_size,
-               enable_repetion_penalty=True,
+               enable_repetition_penalty=True,
            ),
            PositionSpec(
                resource_key=ROPE, nodes={"LLM"},
@@ -644,10 +665,10 @@ to N-1, using its own vocabulary size and no repetition penalty:
 
    SamplerSpec(resource_key=TALKER_SAMPLER, nodes={"Talker"},
                vocab_size=self.config.talker_text.vocab_size,
-               enable_repetion_penalty=True),
+               enable_repetition_penalty=True),
    SamplerSpec(resource_key=CODE_PRED_SAMPLER, nodes={"Talker"},
                vocab_size=self.config.code_predictor.vocab_size,
-               enable_repetion_penalty=False),
+               enable_repetition_penalty=False),
 
 The forward looks up each sampler by key and calls ``.sample()`` on it::
 
@@ -2010,7 +2031,7 @@ own KV cache and attention, a second KV cache for the encoder context, a
 ``CrossAttentionSpec`` over that second cache, a ``PositionSpec``, and a ``SamplerSpec``.
 The ``PositionSpec`` exists only for the position counter. Whisper uses learned position
 embeddings, so the planned ids index an ``embed_positions`` table instead of driving RoPE.
-The ``SamplerSpec`` sets ``enable_repetion_penalty=False``, because ASR transcription
+The ``SamplerSpec`` sets ``enable_repetition_penalty=False``, because ASR transcription
 decodes greedily.
 
 The page counts show how to size a cache for a specific model instead of using the

docs/clients.rst

@@ -97,10 +97,10 @@ Each message is one request with the ``/generate`` fields — ``text``, ``files`
 as a msgpack binary frame (``data`` raw bytes). Replies use the same encoding as the
 message: one frame per result chunk, ``{"request_id", "modality", "data", "metadata"}``,
 then ``{"request_id", "finish": true}``. A rejected message answers
-``{"request_id", "error": ...}`` and the socket stays open; a request that fails after it
-was accepted ends with one such reply. Both carry the HTTP ``status`` when there is one
-(``409`` for a ``request_id`` still in flight, as on ``/generate``), and no
-``finish`` follows. Messages may be pipelined —
+``{"request_id", "error": ..., "status": ...}`` with the HTTP status ``/generate`` would
+have returned (``409`` for a ``request_id`` still in flight), and the socket stays open; a
+request that fails after it was accepted ends the same way, and no ``finish`` follows.
+Messages may be pipelined —
 send the next observation before the current action chunk has returned — and the
 ``request_id`` tells the replies apart. Pipelined messages are submitted in the order
 they were sent, so of two under one ``request_id`` it is the later one that gets the
@@ -129,6 +129,99 @@ they were sent, so of two under one ``request_id`` it is the later one that gets
 ``examples/cosmos3_action_ws_client.py`` is a complete openpi-style client that runs this
 loop at a fixed observation rate and reports chunks/s, actions/s and latency percentiles.
 
+.. _generation-parameters:
+
+Generation parameters
+---------------------
+
+``model_kwargs`` knobs mean the same thing on every model, every endpoint, and every
+execution path (eager or CUDA graph). The server checks them once, before the request is
+queued; a value outside these rules is a ``400`` that names the knob and its range, and a
+streaming request gets that status instead of a ``200`` followed by an in-band error.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 22 30 48
+
+   * - Knob
+     - Allowed
+     - Meaning
+   * - ``temperature``
+     - number ``>= 0``
+     - ``0`` is greedy (the most likely token; ties go to the lower id). Values below
+       ``1e-5`` are greedy too. A greedy request ignores ``top_k``, ``top_p`` and ``min_p``.
+   * - ``top_k``
+     - integer ``>= 0``
+     - ``0`` is off; so is any value at or above the vocabulary size.
+   * - ``top_p``
+     - number in ``(0, 1]``
+     - ``1`` is off. ``0`` is refused: ask for greedy with ``temperature: 0``.
+   * - ``min_p``
+     - number in ``[0, 1]``
+     - ``0`` is off. A model whose sampler has no min-p filter refuses ``min_p > 0``.
+   * - ``repetition_penalty``
+     - number ``> 0``
+     - ``1`` is off. Counts the prompt's tokens as well as generated ones unless
+       ``penalize_prompt`` is ``false``. A model whose sampler has no penalty refuses
+       any value other than ``1``.
+   * - ``penalize_prompt``
+     - ``true`` / ``false``
+     - Whether ``repetition_penalty`` also counts the prompt (default ``true``).
+   * - ``seed``
+     - signed 64-bit integer
+     - Seeds FlashInfer's sampler. A seeded request repeats only when its batch
+       repeats: the draw also depends on its batch position and on the batch's
+       first request, so under continuous batching the same seed can give different
+       tokens. Absent, the server derives a seed from the request id.
+       Each sampler of a multi-stage model draws its own stream from it.
+   * - ``max_output_tokens``
+     - integer ``>= 1``
+     - In the model's output unit (text tokens, codec frames). ``max_tokens``,
+       ``max_completion_tokens`` and ``max_new_tokens`` are accepted as aliases, in
+       that order of increasing precedence below ``max_output_tokens`` itself. A value
+       above the model's limit is refused rather than cut short.
+   * - ``ignore_eos``
+     - ``true`` / ``false``
+     -
+   * - ``num_inference_steps``
+     - integer ``>= 1``
+     - Denoising steps of an image or video model. Past the denoise loop's ``max_iters``
+       is a 400.
+   * - ``num_step``
+     - integer ``>= 1``
+     - OmniVoice's unmask steps. Past the unmask loop's ``max_iters`` is a 400.
+   * - ``rollout_horizon``
+     - integer ``>= 0``
+     - V-JEPA2's rollout length. Past the rollout loop's ``max_iters`` is a 400.
+   * - ``height``, ``width``, ``num_frames``
+     - integer ``>= 1``
+     - Output size of an image or video model, which may refuse sizes its VAE cannot tile.
+   * - ``fps``
+     - number ``> 0``
+     -
+   * - ``guidance_scale``
+     - number
+     - Classifier-free guidance strength.
+   * - ``speed``
+     - number ``> 0``
+     - Speech rate, for a model that reads it (Kokoro, OmniVoice); any other model logs it
+       as ignored. Only a sent ``speed`` is forwarded, not the OpenAI default of 1.0.
+
+A string that spells a number is read as that number, since some JSON clients send
+numbers as strings: ``"0.7"`` or ``" 0.7 "`` for a real-valued knob, ``"5"`` or ``"5.0"``
+for an integer one (``"5e0"``, ``"5.5"`` and ``"0x1"`` are not integers and are a 400).
+Booleans stay strict: ``"false"`` is not a boolean, and ``true`` is not a number. ``null``
+is the same as leaving the knob out: the model's default applies, which is the
+checkpoint's own ``generation_config.json`` value where it ships one. BAGEL and Orpheus
+are the exception: their files are the backbone's chat presets, so only their
+``do_sample: false`` is read and the models' own defaults stay. Multi-stage models
+take a stage prefix (``talker_temperature``, ``code_predictor_top_k``) with the same
+rules; an unprefixed knob sets the primary stage (the thinker of Qwen3-Omni, the talker
+of Qwen3-TTS), and a prefixed one wins over it for its stage.
+
+A knob the model does not read is not an error: the request is served, and the server
+logs its name as ignored (check the server log for typos such as ``temprature``).
+
 Python SDK
 ----------
 
@@ -292,9 +385,11 @@ Per-model notes:
   klein) only: ``size`` as ``WxH`` (multiples of 16); ``seed``; ``num_inference_steps``,
   ``output_format`` (``png`` default, ``jpeg``, ``webp``), ``output_compression`` (JPEG /
   WebP quality) and ``png_compress_level`` (0-9, default 1) via ``extra_body``.
-- **Qwen3-Omni** — text sampling uses ``thinker_*`` keys, speech uses ``talker_*``, and the
-  residual codec groups use ``code_predictor_*``; set the speaker with ``voice`` (default
-  ``Ethan``) and request audio output by including ``"audio"`` in ``modalities``.
+- **Qwen3-Omni** — text sampling uses plain or ``thinker_*`` keys (a ``thinker_*`` key wins),
+  speech uses ``talker_*``, and the residual codec groups use ``code_predictor_*``; set the
+  speaker with ``voice`` (``Ethan`` default, ``Chelsie``, ``Aiden``; any other is a 400) and
+  request audio output by including ``"audio"`` in ``modalities``. ``max_output_tokens``
+  defaults to 2048 and ``talker_max_output_tokens`` to 4096 (the checkpoint's value).
   Non-OpenAI knobs (e.g. ``talker_top_k``, ``code_predictor_top_p``) go through
   ``extra_body``.
 - **Whisper / Higgs-Audio** — ``language`` (ISO-639-1) skips language detection,
@@ -321,6 +416,10 @@ Per-model notes:
   decodes past 30 min, whatever its header says (a 413 here, a 400 from the data
   worker). The decode up to that point still runs, so this bounds memory, not
   time: refusing a 31 min file takes about as long as decoding it.
+  Whisper also reads ``task`` (``transcribe``, the default, or ``translate`` into
+  English). Its ``max_output_tokens`` is at most 444, the 448-position table less the
+  forced prompt; more is a 400. Higgs-Audio's default is 1024. A key the model does not
+  read is logged as ignored on each request that sends it.
 - **Omnilingual ASR** (``omnilingual_asr_ctc_*``): no language code is needed for any of
   its 1,600+ languages, and ``language`` / ``prompt`` / ``temperature`` have no effect. A
   clip of up to 20 s is one request; a longer upload is cut into windows of about equal
@@ -388,4 +487,5 @@ Per-model notes:
   ``temperature`` / ``top_p`` are ignored: Kokoro does not sample.
 - **Orpheus** — set the speaker with ``voice`` — one of ``tara`` (default), ``zoe``,
   ``zac``, ``jess``, ``leo``, ``mia``, ``julia``, ``leah`` (the ``available_voices`` list
-  in the Orpheus config).
+  in the Orpheus config); an empty or unknown voice is a 400. ``max_output_tokens``
+  defaults to 2048.