Skip to content

fix(cache_aware): gate the miss path and ignore idle throughput gauges #2810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
slin1237 wants to merge 5 commits into
base: main
Choose a base branch
from
+365 −73
Open

fix(cache_aware): gate the miss path and ignore idle throughput gauges #2810

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5b09baa
fix(cache_aware): gate the miss path and ignore idle throughput gauges
claude Oct 6, 2026
047612b
fix(cache_aware): gate hash-mode head misses and score idle workers a…
claude Oct 6, 2026
618040d
docs(least_load): say when the default throughput applies
claude Oct 6, 2026
8c70ec2
docs(cache_aware): scope the miss-burst bound to the KV-balanced paths
claude Oct 6, 2026
fc5995b
Merge branch 'main' into claude/charming-cray-3vuhaq-cache-aware
claude Oct 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions bindings/python/src/smg/router_args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,8 +556,8 @@ def add_cli_args(
type=float,
default=RouterArgs.least_load_default_throughput,
help=(
"Fallback generation throughput (tokens/s) for least_load when a"
" backend reports no live throughput"
"Fallback generation throughput (tokens/s) for least_load when no"
" candidate reports a live throughput"
),
)
routing_group.add_argument(
Expand Down
2 changes: 1 addition & 1 deletion model_gateway/src/config/types.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -962,7 +962,7 @@ pub enum PolicyConfig {
#[serde(default = "default_least_load_mean_prefill")]
mean_prefill_tokens: u32,
/// Fallback generation throughput (tokens/s) for the expected-wait term
/// when a backend reports no live `gen_throughput`. Set to the fleet's
/// when no candidate reports a live `gen_throughput`. Set to the fleet's
/// per-replica generation rate; co-tunes with `kv_pressure_weight`.
#[serde(default = "default_least_load_throughput")]
default_throughput: f64,
Expand Down
4 changes: 2 additions & 2 deletions model_gateway/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -460,8 +460,8 @@ struct CliArgs {
#[arg(long, default_value_t = 0.15, help_heading = "Routing Policy")]
least_load_kv_pressure_weight: f64,

/// Fallback generation throughput (tokens/s) for least_load when a backend
/// reports no live throughput
/// Fallback generation throughput (tokens/s) for least_load when no
/// candidate reports a live throughput
#[arg(long, default_value_t = 2000.0, help_heading = "Routing Policy")]
least_load_default_throughput: f64,

Expand Down
Loading
Loading