Skip to content

perf(api-server): stop keeping finished sequences, keep prompt rendering and tokenizing off the event loop - #2912

Merged
valarLip merged 14 commits into
ROCm:mainfrom
yhl-amd:hyi/api-server-hot-path
Nov 5, 2026
Merged

perf(api-server): stop keeping finished sequences, keep prompt rendering and tokenizing off the event loop#2912
valarLip merged 14 commits into
ROCm:mainfrom
yhl-amd:hyi/api-server-hot-path

review: widen prompt annotations to PrefixEncoding

5cfc14e
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

2 errors
Job log options

This job was cancelled