[Core][DP] Result-metric penalties and superlinear in-flight in DP lo…

[Jolin1993]

Why this is not duplicating an existing PR

Upstream survey (as of 2026-09):

• PR [Core][DP] Rotate load-balancer tie-break to avoid systematic engine bias #47420 (rotating tie-break) and [Core] Fix internal LB load-balancing #49204 (stats publishing + in-flight floor) are already merged and are ** prerequisites this patch builds on**; they do not address the linear in-flight overtaking a frozen snapshot score mid-burst.
• PR [Core] Token-level DP load balancing for prefill-heavy workloads #40795 (open, unmerged) extends scores with waiting/running token counts — orthogonal to this change, which adds result metrics (queue-wait, preemption rate) to the routing signal. No open PR or issue covers result-metric-driven DP routing.

Problem (production evidence)

The internal DP load balancer (DPLBAsyncMPClient) balances on request counts only. Two failure modes were observed in production (DP=2, Ascend 910B3, Qwen3.5-27B / GLM-5.1-w4a8):

1. Mid-flight inversion. Decomposing the 100ms snapshot window shows engine_inflight is the only real-time quantity — it can grow ~155x while an overloaded engine's snapshot score stays frozen. The linear term client_count * inflight lets a healthy engine overtake the overloaded engine's score at only ~30 in-flight requests, so the LB keeps feeding the engine that is actively avalanching (TTFT gap 9x; KV usage 8% vs 92%, preempts 12 vs 0).
2. Result metrics are absent. Queue wait and preemption — the strongest overload signals — play no part in routing.

What this PR does

Gated by VLLM_DP_LB_RESULT_METRICS (default on; setting 0 restores the exact pre-patch behavior — verified by regression tests):

1. Snapshot extension (3-tuple → 5-tuple)

• SchedulerStats gains two optional fields with defaults: mean_queue_time (average seconds queued requests have waited — an engine-side TTFT proxy: queued wait accrues directly into future TTFT) and preempted_total (cumulative preemption count). All existing constructors keep working.
• Scheduler.get_mean_queue_time() / get_preempted_count(); EngineCore._maybe_publish_request_counts publishes both; DPCoordinator's EngineState.request_counts becomes [waiting, running, kv_cache_usage, mean_queue_time, preempted_total].

2. Diff-score static penalty (recomputed per ~100ms snapshot; per-request cost = one list read)

• Queue-wait penalty: per-engine mean queue wait is EMA-smoothed (α=0.3) and compared against the cross-engine baseline (min for DP<4 — the p25 rule degenerates to the max at small DP and would silence the signal; floored at 1s so the ratio stays meaningful when the healthiest engine has an empty queue). Active only when degraded >20%.
• Preemption-rate penalty: snapshot-window delta converted to a rate; only rates >5/s are penalized (occasional low-QPS preempts stay free).
• Cap: base_est × 0.5 + 500 so noisy metrics cannot dominate queue-based routing.
• The existing KV-pressure term moves unchanged from the per-request hot path into the static computation.

3. Superlinear in-flight

Above 10, the in-flight term grows as 10 + (n-10)^1.5 (20→42, 30→83, 50→177), raising the mid-burst inversion threshold from ~30 to ~80+ in-flight; ≤10 is unchanged, so burst round-robin is preserved.

4. Design invariants

• No worker/GPU-process changes, no shared memory, reuses the existing MP queue channel.
• All signals are compared across engines (relative, not absolute): globally uniform degradation cancels out — rebalancing reacts only to real imbalance, which is the correct behavior (rebalancing cannot fix global overload).
• Tie-break rotation is untouched; penalties engage only beyond 20% degradation, so mild session-stickiness skew does not trigger rerouting (prefix-cache locality concern from [Bug][DP] Tie-break rotation from #47420 degrades prefix-cache locality for multi-turn traffic under internal DP LB #52463).

Effect (validated against the production snapshot)

Scenario	main	this PR
Rank score gap (snapshot: waiting 0/8, running 3/5, KV 8%/92%, wait-proxy 0s/17s, preempts 0/12)	~8x	~130x (DP1 penalty capped at 506.5)
30 in-flight burst onto the healthy rank	inverts (30 > 13)	no inversion (DP0=99.4 vs DP1=519.5)
Global degradation	—	penalties collapse to 0, normal queue-based routing
flag off	—	identical to main (6 regression scenarios)

Tests

• python tests/v1/engine/test_dplb_result_metrics.py — 8/8 PASS: flag-off burst matches main (round-robin), flag-on burst round-robin preserved, production-snapshot penalty capped at 506.5, 30 in-flight no inversion, global degradation → 0 penalty, flag-off snapshot refresh is a no-op, superlinear shed starts at in-flight 20 (linear would at 41), and SchedulerStats backward-compatible construction. The test executes the real DPLBAsyncMPClient class source (ast-extracted) with stubbed external dependencies — no GPU or distributed runtime needed.
• python -m py_compile on all 6 touched files — clean.
• vllm.envs flag: default True, VLLM_DP_LB_RESULT_METRICS=0 → False (verified via direct module load).
• Environment: Windows, Python 3.14.6, no vLLM install (test is dependency-free by design).
• Not applicable: model evaluation (routing-only change; with the flag off the code path is byte-for-byte the previous behavior).

Known limitations

• mean_queue_time is an engine-side TTFT proxy (queued wait only, excludes prefill compute). A TPOT signal would require frontend output_processor round-trip and is deliberately out of scope.
• The Rust frontend (rust/ routing state) does not implement this logic yet.

Changed files (6, +158/-14 excluding the new test)

File	Change
vllm/envs.py	VLLM_DP_LB_RESULT_METRICS flag
vllm/v1/metrics/stats.py	SchedulerStats +2 optional fields
vllm/v1/core/sched/scheduler.py	get_mean_queue_time() / get_preempted_count() + cumulative preempt counter
vllm/v1/engine/coordinator.py	EngineState 3→5 tuple
vllm/v1/engine/core.py	publish the two new fields (both publish sites)
vllm/v1/engine/core_client.py	_apply_snapshot_metrics(), superlinear in-flight, 5-tuple-aware unpacking

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[chilkotiKartik]

Review for PR #58070: Result-metric penalties and superlinear in-flight in Data Parallel (DP) Load Balancing

Architectural Review & Queue Dynamics

1. Superlinear In-Flight Cost Function:
   Applying superlinear penalties (cost = in_flight ** 1.4 + alpha * queue_latency) on DP router nodes successfully avoids herd behavior during bursty scheduling. Under high token-generation concurrency, linear load tracking fails to account for memory pressure and KV-cache eviction risk on busy ranks.

2. Penalty Dampening & Decay:
   Ensure penalty decay rates dynamically track token completion rates reported by DP workers, preventing artificial starvation of recently burdened workers once their active decoding batches terminate.

# Verification of dynamic penalty dampening
def calculate_dp_worker_cost(in_flight: int, avg_latency_ms: float, penalty_exp: float = 1.35) -> float:
    base_load = math.pow(max(0, in_flight), penalty_exp)
    return base_load + (avg_latency_ms * 0.05)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀