[DO NOT MERGE] perf(multimodal): reduce video decode, Qwen preprocess, and TokenSpeed handoff overhead - #1820

Closed

yechank-nvidia wants to merge 47 commits into

smg-project:mainfrom

yechank-nvidia:yechan/mm-video-decode-seq

Commits on Jun 29, 2026

perf(multimodal): decode sampled video frames sequentially, not per-frame seek
yechank-nvidia
committed
1aeb274
perf(multimodal): reduce vision preprocessing overhead
yechank-nvidia
committed
34023a5
perf(tokenspeed): optimize multimodal tensor handoff
yechank-nvidia
committed
c3e7bf2
fix(multimodal): address review validation gaps
yechank-nvidia
committed
1557dba
fix(multimodal): address follow-up review gaps
yechank-nvidia
committed
bca3c04
style(tokenspeed): apply ruff formatting
yechank-nvidia
committed
2b215d1
test(tokenspeed): skip shm tests without torch
yechank-nvidia
committed
7611aaf
fix(multimodal): satisfy clippy in qwen tests
yechank-nvidia
committed
7e68ea6
test(tokenspeed): clean shm file on failure
yechank-nvidia
committed
9ce64ed
fix(tokenspeed): satisfy clippy in multimodal assembly
yechank-nvidia
committed
fb864c4
fix(responses): sanitize mcp replay errors
yechank-nvidia
committed
2003572
Revert "fix(responses): sanitize mcp replay errors"
yechank-nvidia
committed
93683aa
style(multimodal): apply rustfmt
yechank-nvidia
committed
32e8f70

Commits on Jun 30, 2026

fix(multimodal): address review feedback
yechank-nvidia
committed
87a09ca
style(tokenspeed): apply ruff formatting
yechank-nvidia
committed
53e5238
perf(multimodal): adapt OpenCV decoder threads
yechank-nvidia
committed
56a6f20
perf(multimodal): tune decoder threads across CPU quotas
yechank-nvidia
committed
41dd682
refactor(multimodal): clarify adaptive decoder budgeting
yechank-nvidia
committed
671d34f

Commits on Jul 1, 2026

perf(multimodal): reuse preprocessing worker pool
yechank-nvidia
committed
ee5facd
perf(multimodal): parallelize encoder tensor serialization
yechank-nvidia
committed
d74b216
perf(multimodal): serialize encoder tensors into mapped shm
yechank-nvidia
committed
68207d9
perf(multimodal): skip identity bicubic resize passes
yechank-nvidia
committed
f086ed0
perf(multimodal): defer Qwen video normalization for TokenSpeed
yechank-nvidia
committed
61a3e8b
perf(tokenspeed): use worker vision encoder dtype
yechank-nvidia
committed
6e8466b
perf(multimodal): defer Qwen image normalization
yechank-nvidia
committed
beae33f
perf(multimodal): precompute deferred BF16 normalization
yechank-nvidia
committed
256ee99
perf(multimodal): decode buffered videos without tempfiles
yechank-nvidia
committed
9c44bbf
fix(multimodal): propagate OpenCV probe errors
yechank-nvidia
committed
c36f075
perf(multimodal): patchify resized Qwen video directly
yechank-nvidia
committed
61eab84
Overlap video decode and Qwen preprocessing
yechank-nvidia
committed
58e5a76
Enable adaptive video preprocessing overlap
yechank-nvidia
committed
349528e
Avoid RGB conversion in streamed video preprocessing
yechank-nvidia
committed
5d82941
Remove multimodal performance environment knobs
yechank-nvidia
committed
876f639
Own multimodal execution resources explicitly
yechank-nvidia
committed
32a5419
Model multimodal payload representations explicitly
yechank-nvidia
committed
4028d63
Unify multimodal preprocessing dispatch
yechank-nvidia
committed
b2a4935
Separate multimodal backend assembly
yechank-nvidia
committed
3b98743
Separate multimodal assembly responsibilities
yechank-nvidia
committed
ae75181
Isolate TokenSpeed transport policy
yechank-nvidia
committed
bdbe1fb
Make prepared media modality exhaustive
yechank-nvidia
committed
19b2215
Simplify multimodal assembly intermediate
yechank-nvidia
committed
5ffa5cc
Generalize multimodal processor registry
yechank-nvidia
committed
8cd337c
Carry prepared media through preprocessing
yechank-nvidia
committed
6c84189
Use neutral multimodal assembly metadata
yechank-nvidia
committed
02d18b4
Type multimodal preprocessing requests
yechank-nvidia
committed
4b5504a
Make processor matching deterministic
yechank-nvidia
committed
7285463

Commits on Jul 2, 2026

Address multimodal review findings
yechank-nvidia
committed
d4756df