feat(sglang): tokens in/out with logprobs

[Aphoh]

Summary

Keeps the logprob/token-id pieces needed by RL workflows while preserving OpenAI-compatible response shapes.

This PR:

• adds return_tokens_as_token_ids for OpenAI chat/completions logprob responses
• wires token-id logprob formatting through the SGLang backend
• accepts integer token-id stop arrays such as stop: [576]
• enables /v1/completions chosen-token logprobs when logprobs=0
• gates expensive SGLang top-logprobs behind DYN_SGL_ALLOW_TOP_LOGPROBS=1
• returns backend stop_reason as opt-in nvext.stop_reason metadata instead of adding non-OpenAI fields to choices[]

What's included

• return_tokens_as_token_ids request/output option plumbing in the Rust OpenAI protocol layer.
• SGLang decode handling emits token_id:<id> strings for logprob token fields when requested.
• /v1/completions treats logprobs=0 as chosen-token logprobs only.
• SGLang rejects logprobs >= 1 / prompt_logprobs >= 1 by default to avoid upstream per-position top-token detokenization cost. Set DYN_SGL_ALLOW_TOP_LOGPROBS=1 to opt in while tracking the upstream fix in Batch detokenization across positions in detokenize_top_logprobs_tokens sgl-project/sglang#24447.
• SGLang finish_reason.matched values are carried as Dynamo stop reasons for strings, token IDs, and token-ID arrays.
• Hidden/system stop token IDs are filtered out before nvext.stop_reason, so the response only reports user-provided stop triggers.
• stop: [576] is accepted as a Dynamo token-id stop extension and is threaded through internal StopConditions / SGLang sampling params.
• stop_reason is returned only when requested with nvext.extra_fields: ["stop_reason"], and is emitted at response-level nvext.stop_reason.
• docs/components/frontend/nvext.md documents stop_reason and the distinction between token-id logprob display and token-id stop input.

What's not included

• No /v1/tokenize endpoint.
• No /v1/detokenize endpoint.
• No tokenizer trait API expansion for standalone tokenization.
• No model-card local-file reuse changes.
• No vLLM token-id/logprob wiring in this PR.
• No per-choice stop_reason response shape for n > 1; Dynamo currently serves this as a response-level field for single-choice requests, and n > 1 will need an indexed/per-choice nvext shape.
• No support for "token_id:576" as token-id stop input; that string remains a literal string stop sequence.

Stop contract

Dynamo keeps the public response shape aligned with the OpenAI completions API while accepting token-id stops as a Dynamo extension:

• stop: "..." stops on that string.
• stop: ["A", "B"] stops on either string.
• stop: [32, 34] stops on either token ID.
• stop: "token_id:576" and stop: ["token_id:576"] are treated as literal string stops, not token-id shorthand.
• stop: 576 is rejected; scalar numeric stops are not accepted.
• choices[].finish_reason remains the OpenAI-compatible value, for example "stop".
• When requested with nvext.extra_fields: ["stop_reason"], Dynamo returns the backend matched stop reason at response-level nvext.stop_reason: string stops return strings, and token-id stops return numbers.
• choices[].stop_reason is intentionally omitted because it is not part of the normal OpenAI completions API. Dynamo currently serves nvext.stop_reason as a response-level single-choice field; n > 1 will require an indexed/per-choice nvext shape.

API notes

When return_tokens_as_token_ids: true, logprob token fields use token_id:<id> instead of decoded text:

{
  "logprobs": 0,
  "return_tokens_as_token_ids": true
}

Example emitted token field:

{"token": "token_id:12345", "logprob": -0.5}

That display format is output-only. To stop on a token ID, use an integer stop array:

{
  "prompt": [1, 2, 3],
  "stop": [32, 34],
  "nvext": {
    "extra_fields": ["stop_reason"]
  }
}

Do not use "stop": ["token_id:576"] for token-id stops; Dynamo treats it as a literal string stop sequence.

Example response shape for a matched string or token-id stop:

{
  "choices": [
    {
      "finish_reason": "stop"
    }
  ],
  "nvext": {
    "stop_reason": 576
  }
}

choices[].stop_reason is intentionally omitted because it is not part of the normal OpenAI completions API.

Validated against the temporary Dynamo /v1/completions stack:

Stop input	Result
stop: [" The"] where " The" is decoded token ID 576	Works, finish_reason="stop", response-level nvext.stop_reason=" The"
stop: " The"	Works, finish_reason="stop", response-level nvext.stop_reason=" The"
stop: ["token_id:576"]	Treated as a literal string, generation continues to length
stop: "token_id:576"	Treated as a literal string, generation continues to length
stop: [32, 34]	Accepted as token-id stop input; stops on either token ID
stop: 576	Rejected; scalar numeric stop is not accepted
top-level stop_token_ids: [576]	Not part of the public OpenAI HTTP shape
nvext: {"stop_token_ids": [576]}	Not the supported stop-token input shape

Test plan

• cargo fmt --all -- --check
• git diff --check
• python3 -m py_compile components/src/dynamo/frontend/sglang_processor.py components/src/dynamo/sglang/protocol.py
• python3 -m py_compile components/src/dynamo/frontend/tests/test_sglang_processor_unit.py components/src/dynamo/sglang/tests/test_sglang_decode_handler.py
• cargo test -p dynamo-protocols stop
• cargo test -p dynamo-llm stop_contract --no-default-features
• cargo test -p dynamo-llm test_stop --no-default-features
• cargo test -p dynamo-llm user_stop_token_reports_distinct_trigger --no-default-features
• cargo test -p dynamo-llm stop_reason --no-default-features
• pre-commit hooks during commit
• PYTHONPATH=components/src pytest components/src/dynamo/frontend/tests/test_sglang_processor_unit.py -k "stop_token_id_array_maps_to_stop_token_ids or string_stops_remain_string_stops or token_id_display_string_remains_string_stop" locally, blocked by missing sglang package in this environment
• PYTHONPATH=components/src pytest components/src/dynamo/sglang/tests/test_sglang_decode_handler.py -k "user_stop_token_ids or openai_stop_sampling_params" locally, blocked by missing sglang package in this environment

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-d9900877-4da6-4032-9516-1e88acac3bd9.docs.buildwithfern.com/dynamo/dev

[AndyDai-nv]

LGTM — extending /v1/completions with return_tokens_as_token_ids is the cleanest path for downstream RL frameworks (Miles could directly drop its fallback_tokenize=True re-tokenization on this).

One minor and non-blocking thing, marking as Comment since I haven't run it locally yet