Improve load balancing behavior for prefix cache-aware routing - #3930

Merged

santhnm2 merged 47 commits into

NVIDIA:mainfrom

santhnm2:prefix_caching_load_balancing

Mar 26, 2026

Commits on Mar 18, 2026

Improve load balancing behavior for prefix cache-aware routing
santhnm2
committed
73c9dd1
Linting
santhnm2
committed
bd16146
Add prioritization for idle ranks
santhnm2
committed
2a04323
Linting
santhnm2
committed
9b49d98
Fix tests
santhnm2
committed
79cf6a7
Fix tests
santhnm2
committed
cdbc032
Fix tests
santhnm2
committed
e1c457e
Fix tests
santhnm2
committed
7792b15
Add scoring function
santhnm2
committed
097b4b4
Linting
santhnm2
committed
0a90d39
Typo
santhnm2
committed
12b79c0
Add vectorized operations
santhnm2
committed
a941be3
Add hash_rank_table.py
santhnm2
committed
0f59faf
Linting
santhnm2
committed
f3f8151
Update pending count decrement
santhnm2
committed
0a7d380
Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
2f5c5d4
Make round robin just do round robin
santhnm2
committed
3b37d2e
Linting
santhnm2
committed
76d9b4a

Commits on Mar 19, 2026

Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
21787fd
Address review feedback
santhnm2
committed
4ca3e8d
Linting
santhnm2
committed
2ec13c6
Update comments
santhnm2
committed
a08ef34
Add add_rank functionality
santhnm2
committed
e80b5bf
Add max_timestamps
santhnm2
committed
e443bdb
Linting
santhnm2
committed
8c215f6
Simplify policy
santhnm2
committed
48fca8d
Linting
santhnm2
committed
cf940b0
Update hash table tests
santhnm2
committed
93eee30
Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
6551751

Commits on Mar 23, 2026

Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
c91fd84
Add assertion on max_requests
santhnm2
committed
877a0bb
Add compaction interval
santhnm2
committed
c7dee8e
Linting
santhnm2
committed
fb832af
Simplify compaction
santhnm2
committed
4d99149
Store initial capacity
santhnm2
committed
cf4396d

Commits on Mar 24, 2026

Commits on Mar 25, 2026

Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
b7d1484
Remove hash rank table and simplify
santhnm2
committed
03cbe39
Merge with main
santhnm2
committed
b0b4bd6

Commits on Mar 26, 2026

Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
e5bd157
Fix tests
santhnm2
committed
e4b2c17
Remove nested for loop
santhnm2
committed
f3e03ad
Merge with main
santhnm2
committed
d2acb7a
Merge remote-tracking branch 'upstream/main' into prefix_caching_load_balancing
santhnm2
committed
9df482c
Fix tests
santhnm2
committed
9228d8f