Repository navigation
Return logprob for choices - #87
Merged
Merged
Conversation
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
NorthmanPKU
pushed a commit
to NorthmanPKU/sglang
that referenced
this pull request
May 16, 2025
Co-authored-by: Mengdi Wu <mengdiwu@andrew.cmu.edu>
pi314ever
pushed a commit
to pi314ever/sglang
that referenced
this pull request
Aug 25, 2025
Signed-off-by: Zhou Yuwen <yuwen.zhou@intel.com>
iforgetmyname
added a commit
to iforgetmyname/sglang
that referenced
this pull request
Sep 1, 2025
* support deepep normal * change forward_npu return * fix linting * bugfixes * linting happy * lingting happy
zhuyijie88
pushed a commit
to zhuyijie88/sglang
that referenced
this pull request
Sep 4, 2025
* support deepep normal * change forward_npu return * fix linting * bugfixes * linting happy * lingting happy
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
Oct 22, 2025
Hexq0210
pushed a commit
to Hexq0210/sglang_pro
that referenced
this pull request
Mar 12, 2026
wdunn001
added a commit
to wdunn001/sglang
that referenced
this pull request
May 17, 2026
Closes the loop on @codecai/tool-kit (npm, v0.4.1). The SDK lets
operators publish a tool manifest at build time; this module ships
the engine-side primitives for fetching the manifest, dispatching to
the tool, and reinjecting the response token IDs into the model's
context — all without round-tripping through detokenize/JSON/retok.
## What's in this module
- CodecToolCall / CodecToolResult — msgpack wire shapes for the
/codec/tool/v1/call POST endpoint contract.
- encode_tool_call / decode_tool_result — round-trip helpers.
- RegisteredTool + ToolRegistry — engine-side tool registry. Loaded
via ToolRegistry.from_env(active_tokenizer_hash) which reads
CODEC_TOOL_MANIFEST_URLS, fetches each manifest with urllib
(stdlib only — no heavy HTTP deps at boot), validates the
manifest's tokenizerHash against the active model's tokenizer,
marks mismatched tools as text-fallback mode (the engine still
surfaces them but doesn't dispatch).
- dispatch_call(tool, arguments_json, call_id) — synchronous POST
to the tool's endpoint with the msgpack-framed CodecToolCall.
Returns the parsed CodecToolResult.
- reinject_ids_into_context(ctx_ids, response_ids) — append the
tool's response IDs to the model's generation context.
## env vars
- CODEC_BOLT_ON_DISPATCH=1 — opt the engine into bolt-on dispatch
(default off; off-state preserves v0.2-v0.4 pass-through behaviour
where the engine surfaces <tool_call> regions to the client).
- CODEC_TOOL_MANIFEST_URLS=<url1>,<url2>,... — list of manifest URLs
loaded at engine boot.
- CODEC_TOOL_MANIFEST_REQUIRED=1 — hard-fail engine boot if any
manifest fails to load (default off → drop failed tools + warn).
## What's NOT in this commit
The integration with the streaming loop (_generate_binary_stream)
that detects <tool_call> via ToolWatcher and routes to dispatch_call
+ reinject. That integration touches the KV-cache-aware regeneration
path and needs careful review against SGLang's batching semantics;
filed as the next pass after this contract module lands.
Smoke-tested standalone: encode_tool_call → decode_tool_result
round-trip works; ToolRegistry register/lookup works; reinjection
append works. msgpack hex for {tool_name='time', call_id='tc_1',
arguments_json='{}', tokenizer_hash='sha256:abc'} captured for the
cross-fork parity test against the vllm mirror.
Scoped per Codec/docs/engine-fork-tasks/v0.5-rollout.md § Task sgl-project#87.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wdunn001
added a commit
to wdunn001/sglang
that referenced
this pull request
May 17, 2026
…enerate_binary_stream Completes the bolt-on tool dispatcher integration on sglang's side. The dispatcher contract module (commit c193c36) shipped the primitives; this commit wires them into the streaming loop: 1. When CODEC_BOLT_ON_DISPATCH=1 AND a ToolWatcher is active, load the tool registry from CODEC_TOOL_MANIFEST_URLS at the start of the stream (cached per stream, not per chunk). 2. When ToolWatcher completes a region, parse_tool_call as before AND look up the tool name in the registry. If found in dispatch mode, POST CodecToolCall to the tool's endpoint synchronously. 3. On a successful dispatch (is_error=false), append the tool's response_ids to the current frame's new_ids via reinject_ids_into_context. The model "reads" them as its next input. 4. Dispatch failures (network, hash mismatch, 5xx) log a warning but don't tear down the stream — the tool call still surfaces to the client as a tool_calls_payload entry, just without an in-engine response. Same end-user outcome as if the tool wasn't registered. The reinjection here is the simple append model documented in codec_dispatcher.reinject_ids_into_context. KV-cache-aware reinjection (inserting at the position where <tool_call> was detected, with proper KV-cache invalidation downstream of that position) is the follow-up upstream work in tokenizer_manager — out of scope for the dispatcher contract module, but the integration point is now clear (the new_ids extension is the seam where the deeper integration plugs in). env vars (no change): CODEC_BOLT_ON_DISPATCH=1 CODEC_TOOL_MANIFEST_URLS=<url1>,<url2>,... CODEC_TOOL_MANIFEST_REQUIRED=1 With both env vars set + a tool registered in dispatch mode, sglang now handles the full <tool_call>...</tool_call> → tool POST → reinjection loop without ever decoding the model's stream to text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.