Skip to content

refactor(policies): rename least_load to kv_pressure_weight - #1632

Merged
slin1237 merged 1 commit into
mainfrom
refactor/least-load-kv-pressure-weight
Jun 10, 2026
Merged

slin1237 merged 1 commit into
mainfrom
refactor/least-load-kv-pressure-weight

Conversation

@slin1237

@slin1237 slin1237 commented Jun 10, 2026 •

Copy link
Copy Markdown
Member

No description provided.

The least_load score is `in_flight + lambda * k/(1-k)`, where k is the
backend KV-cache utilization from the load monitor. `lambda` is opaque to
operators and mathematically misleading: in the M/M/1 model the utilization
k already encodes the arrival/service ratio, so this coefficient is not the
M/M/1 arrival rate — it is simply the weight that converts the KV-congestion
term into request-equivalents so it is commensurable with the in-flight count.

Rename to `kv_pressure_weight` (the policy's own doc already calls it the
"KV-pressure weight") across the config variant, validation, factory,
CLI/binding construction, and the policy's internal field, constant, and
constructor. Pure rename — no behavior change; default stays 1.5.

Signed-off-by: Simo Lin <25425177+slin1237@users.noreply.github.com>
@github-actions github-actions Bot added python-bindings Python bindings changes model-gateway Model gateway crate changes labels Jun 10, 2026
@coderabbitai

coderabbitai Bot commented Jun 10, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR renames the LeastLoadPolicy weight parameter from lambda to kv_pressure_weight across configuration, validation, implementation, instantiation, and Python binding layers, maintaining the same default value and scoring behavior throughout the stack.

Changes

LeastLoad Configuration Parameter Rename

Layer / File(s) Summary
Configuration schema and validation
model_gateway/src/config/types.rs, model_gateway/src/config/validation.rs
PolicyConfig::LeastLoad enum field renamed from lambda to kv_pressure_weight; serde default function renamed from default_least_load_lambda() to default_least_load_kv_pressure_weight() preserving the 1.5 default; validation logic checks the new field with the same finite/non-negative constraints.
Core policy implementation and public API
model_gateway/src/policies/least_load.rs
LeastLoadPolicy struct field renamed to kv_pressure_weight; new exported constant DEFAULT_KV_PRESSURE_WEIGHT replaces DEFAULT_LAMBDA; public constructor with_kv_pressure_weight() replaces with_lambda(); score formula updated to use the renamed field.
Policy instantiation in factory and CLI
model_gateway/src/policies/factory.rs, model_gateway/src/main.rs
PolicyFactory::create_from_config and CliArgs::parse_policy updated to construct LeastLoadPolicy with kv_pressure_weight parameter.
Python bindings exposure
bindings/python/src/lib.rs
Router::to_router_config policy conversion updated to set kv_pressure_weight instead of lambda for PolicyType::LeastLoad.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

  • lightseekorg/smg#1629: Implements the initial least_load policy with lambda parameter; this PR builds on that foundation by renaming the configuration weight parameter to kv_pressure_weight across all usage sites.

Suggested labels

python-bindings, tests, model-gateway

Suggested reviewers

  • CatherineSue
  • key4ng
  • gongwei-130

Poem

🐰 A weight renamed across the stack so wide,
From lambda's grasp to kv_pressure's stride,
Config and validation, factory and bind,
All dancing in sync with semantic redesign—
Same score, new name, configuration refined! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: renaming the least_load policy's lambda parameter to kv_pressure_weight, which is the central refactor across all modified files.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/least-load-kv-pressure-weight

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request renames the lambda parameter to kv_pressure_weight across the codebase, including configuration, validation, and policy implementations, to improve clarity. However, renaming this configuration field directly breaks backward compatibility for existing deployments. It is recommended to add a Serde alias to ensure a smooth upgrade path.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread model_gateway/src/config/types.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5b35134bb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread model_gateway/src/config/types.rs
Comment thread model_gateway/src/config/types.rs

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-executed rename. All references updated consistently across config types, validation, factory, CLI args, Python bindings, and tests. No behavior change, default preserved at 1.5. One minor nit posted about adding a serde alias for the old field name — low risk since the feature only landed today.

Summary: 0 🔴 Important · 1 🟡 Nit · 0 🟣 Pre-existing

@slin1237 slin1237 changed the title refactor(policies): rename least_load lambda to kv_pressure_weight refactor(policies): rename least_load to kv_pressure_weight Jun 10, 2026
@slin1237
slin1237 merged commit 27b528f into main Jun 10, 2026
26 of 27 checks passed
@slin1237
slin1237 deleted the refactor/least-load-kv-pressure-weight branch June 10, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-gateway Model gateway crate changes python-bindings Python bindings changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant