Skip to content

perf(relay): count streaming completion tokens incrementally instead of buffering the full response - #5481

Closed
jstar0 wants to merge 6 commits into
QuantumNous:mainfrom
jstar0:feat/streaming-token-counting
Closed

perf(relay): count streaming completion tokens incrementally instead of buffering the full response#5481
jstar0 wants to merge 6 commits into
QuantumNous:mainfrom
jstar0:feat/streaming-token-counting

test(service): cover explicit multibyte stream splits

df9ba01
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs