[Phoenix3334]

Motivation

Generation health checks (HEALTH_CHECK_*) are intentionally skipped by the scheduler while normal inference is already making progress. Under DP serving, however, the request passes through DataParallelController before that scheduler-side skip.

With round_robin, a generation health request can advance the same dispatch state used by user requests. For load-aware methods, it can also participate in the user load-budget refresh/update path. A health request that is later discarded can therefore still perturb user-request ownership across DP ranks.

The canonical runtime evidence now localizes the failure chain as:

generation health request
  -> shared user LB state is mutated
  -> user dispatch across DP ranks becomes skewed
  -> per-rank effective batch diverges / running work inventory drops
  -> Input TPS collapses and TTFT tail inflates

DP Attention, long chunked prefill, and PrefillDelayer expose or amplify the resulting ownership skew; they are not the source of the health request mutating DPC routing state.

The root-cause investigation is tracked in #35241.

Modifications

This PR isolates generation-health dispatch state from user load-balancing state in DataParallelController:

• Detect generation health requests before normal user LB dispatch.
• Use an independent round-robin counter for health requests.
• Do not refresh or mutate the user DPBudget for health-only dispatches.
• For batch generate, refresh the user load budget only when the batch contains a user request.
• Preserve explicit routed_dp_rank behavior.
• Preserve FOLLOW_BOOTSTRAP_ROOM semantics for PD/disaggregated serving, since that path uses request-specific routing rather than shared mutable user LB state.

This PR does not change DP Attention, PrefillDelayer, scheduler busy-health skip behavior, or the health endpoint semantics.

Regression Tests

Added CPU unit coverage for:

• independent user-RR and health-RR state;
• health dispatch using only active workers;
• explicit routed_dp_rank not advancing health RR state;
• health dispatch bypassing user load-budget refresh;
• health-only batch dispatch not refreshing user load state;
• FOLLOW_BOOTSTRAP_ROOM health routing compatibility.

python3 -m pytest test/registered/unit/managers/test_data_parallel_controller.py -q
30 passed

git diff --check
clean

pre-commit run --files \
  python/sglang/srt/managers/data_parallel_controller.py \
  test/registered/unit/managers/test_data_parallel_controller.py
all passed

Current-main verification (2026-10-10)

Rebased onto main at 16423cb6d2f9ab365f4361cce3572b38890aaf68, preserving the current load-budget refresh throttle and adapting the test helpers to num_dp_ranks. The PR remains scoped to the same two files.

The targeted CPU test file passes 30 tests on this rebased tree. Targeted pre-commit checks and git diff --check also pass. The B300 runtime evidence below was collected previously on the documented release configurations; GPU performance was not re-measured during this rebase.

Runtime Validation

Validated on 8x NVIDIA B300 with DeepSeek-V4-Pro, TP8/DP8, DP Attention, MegaMoE, PrefillDelayer, round-robin routing, 8k effective prefill chunk, C16/N160, 60k input / 1 output.

The canonical 2x2 campaign used fresh servers per round and held the workload/configuration fixed while varying only (DPC patch, generation health):

Configuration	Result	Input TPS	HEALTH@DPC	user dispatch std	per-rank BS
Original + generation health ON	2/8 LOW	LOW: 68,403 / 82,883	48-61	LOW: 1.12 / 0.71	LOW rounds diverged
Original + generation health OFF	4/4 GOOD	94,783-94,847	0	0.00	uniform
Patched + generation health ON	4/4 GOOD	94,558-94,699	49-50	0.00	uniform
Patched + generation health OFF	4/4 GOOD	94,850-94,934	0	0.00	uniform

Three canonical rounds make the mechanism especially clear:

Original + HEALTH ON, LOW:
  TPS=68,403, HEALTH@DPC=61, user dispatch=29.0+-1.12,
  per-rank BS mean=3.71..7.12

Original + HEALTH OFF, GOOD:
  TPS=94,847, HEALTH@DPC=0, user dispatch=29.0+-0.00,
  per-rank BS mean=10.0 x 8

Patched + HEALTH ON, GOOD:
  TPS=94,583, HEALTH@DPC=50, user dispatch=29.0+-0.00,
  per-rank BS mean=10.0 x 8

The important invariant is that generation-health traffic remains present at DPC after the patch, but it no longer perturbs user routing state.

Runtime reproduction

The validation matrix keeps the server/client workload fixed and varies only generation-health state and whether this DPC patch is applied. The relevant runtime knobs are:

# server-side topology / scheduler configuration
--tp 8 \
--dp 8 \
--enable-dp-attention \
--enable-prefill-delayer \
--moe-a2a-backend megamoe \
--chunked-prefill-size 65536 \
--load-balance-method round_robin

# generation-health control
SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=1  # HEALTH ON
SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=0  # HEALTH OFF

Workload shape:

input length    = 60000
output length   = 1
max concurrency = 16
num prompts     = 160
seed            = 1

With DP Attention, --chunked-prefill-size 65536 corresponds to an 8k effective chunk per DP rank. Each arm is run from a fresh server process; external /health polling remains enabled in all arms so HEALTH OFF isolates generation-health request creation rather than HTTP polling overhead.

The exact benchmark wrapper used for these B300 runs is environment-specific, so the PR records the model-independent server knobs, workload shape, and controlled variables needed to reproduce the failure family rather than claiming a portable one-command benchmark across environments.

Signal interpretation

The expanded evidence also corrected two earlier intermediate-mechanism assumptions:

• IDLE-while-EXTEND was 0 across all 27 analyzed rounds, including severe LOW cases, so it is not used as a required link in the canonical causal chain.
• wait_timeout > 0 is not a LOW-state discriminator: patched HEALTH-ON runs remained GOOD with 12-24 timeouts. PrefillDelayer mixed / timeout activity is treated as downstream scheduler activity or an amplifier, not the root trigger.
• fill% and queue depth also overlap between GOOD and LOW rounds and are not state discriminators.

All canonical runs completed 160/160 requests with no fatal errors. The patch removes the user-ownership skew while preserving generation-health dispatch.

This is a scoped validation for the tested failure family; it is not intended to claim that generation health is the cause of every low-throughput state.

A separate 16k-effective-chunk follow-up with generation health ON also completed 4/4 GOOD after patching, supporting the same invariant across both tested chunk configurations.

Limitations

• Runtime validation is scoped to the tested DeepSeek-V4-Pro, 8x B300, TP8/DP8 DP-Attention serving configuration; it does not establish the incidence rate of this failure across all models, hardware, or DP topologies.
• The controlled experiments establish that generation-health traffic mutating shared DPC user-routing state is causal for this reproduced failure family. Downstream scheduler symptoms such as PrefillDelayer timeouts, queue depth, or aggregate request-count skew are not individually necessary or sufficient failure indicators.
• The patch intentionally isolates routing state rather than redesigning the configured user load-balancing policy. Policy-specific performance differences remain outside the scope of this fix.

Scope

The intended invariant is:

Generation health traffic must not mutate the load-balancing state used by user requests when it is routed through DP serving.

This PR does not make round-robin load-aware and does not change the configured load-balancing policy.

Related: #35241

---

CI States

Latest PR Test (Base): ✅ Run #38103975233
Latest PR Test (Extra): ❌ Run #38103975098
Latest PR Test (AMD ROCm 10): ❌ Run #38103975231

[Phoenix3334]

@Ying1123 @hnyls2002 @merrymercy @xiezhq-hermann
Hi maintainers – this PR is currently merge-blocked and waiting for a code owner review.
We've validated it with multiple fresh-server rounds under both 8k and 16k chunk configs; the fix consistently eliminates the performance regression caused by health-check traffic polluting the user routing state. The change is minimal, scoped only to data_parallel_controller.py, and all CPU unit tests pass.
Would appreciate a review when you have a moment – happy to address any feedback. Thanks!

[Phoenix3334]

Source-history check and extended runtime validation

I traced the bug back through the endpoint and dispatch history. This supports fixing the problem at the DPC dispatch boundary, rather than changing PrefillDelayer.

Historical source

• Before #8444, GET /health returned 200 without entering inference; only /health_generate created a HEALTH_CHECK_* request.
• #8444 (commit 6f9baf1, 2025-08-03) merged the routes, so ordinary /health polling began creating generation requests.
• At that revision, every TokenizedGenerateReqInput was first dispatched through the configured DataParallelController load balancer. The health-specific busy skip happened later in Scheduler.process_input_requests().
• Thus “ignored while busy” meant no model execution, but did not mean no mutation of user load-balancing state.
• #13320 later added SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION with default True; #13961 fixed its lookup but did not isolate dispatch state.

This history identifies the bug as an interaction between two individually reasonable behaviors:

generation-based health semantics
+
normal DPC routing before the scheduler-side health skip

DP Attention, long chunked prefill, and PrefillDelayer expose/amplify the resulting ownership skew; they are not the source of the health request mutating RR state.

Release-status correction

v0.5.14 (2026-06-26, 49e384ce...) and v0.5.17 (2026-08-08, 29481685...) both retain the harmful ordering.

This PR was opened on 2026-08-20 and is currently unmerged, so it was not included in v0.5.17. All v0.5.17 runtime numbers below mean v0.5.17 plus this patch applied locally.

16k effective-chunk follow-up

Generation health remained ON for all runs:

Run	Input TPS	avgRun	fill	HEALTH@DPC	Success	Fatal
r1	92,568.7	8.28	91.02%	43	160/160	0
r2	95,146.5	8.45	92.10%	40	160/160	0
r3	95,010.5	8.50	93.35%	39	160/160	0
r4	94,928.6	8.24	89.95%	36	160/160	0

Environment: 8x B300, DeepSeek-V4-Pro, TP8/DP8, DP Attention, MegaMoE, PrefillDelayer, effective 16k chunk, C16/N160, 60k input / 1 output, seed=1.

All four rounds were GOOD; mean Input TPS was 94,413.6. The nonzero HEALTH@DPC counts are important: the patch preserves generation-health traffic and removes only its mutation of user routing state.

Together with the existing patched 8k 4/4 validation, this shows that the invariant holds across both tested chunk configurations:

Health requests may still be dispatched and answered, but they do not advance or refresh the load-balancing state used by user requests.

[Phoenix3334]

@Ying1123 @merrymercy @xiezhq-hermann @hnyls2002

I checked the current CI gate configuration:

• Base and AMD ROCm 7.2 use the default gate and require run-ci.
• Extra explicitly requires both run-ci and run-ci-extra.
• The current failures all stop at call-gate / pr-gate; no actual test matrix ran.
• These gate-only failures do not count as consumed CI runs for the cooldown check.

Could an authorized maintainer please run:

/tag-and-rerun-ci extra

This adds both required labels and reruns the failed/skipped workflows for the current commit.

Validation already completed: 23/23 targeted CPU unit tests, pre-commit, and git diff --check passed; 8×B300 runtime validation passed 4/4 at both effective 8k and 16k prefill chunks, with 160/160 successful requests per run and no ownership holes, PrefillDelayer timeouts, or fatal errors.

[Phoenix3334]

Additional validation note: I also verified the same health-routing isolation semantics against both the v0.5.14 and v0.5.17 DataParallelController source shapes. The release backport preserves the same invariant as this PR: generation-health requests use separate routing state, do not refresh user DPBudget, explicit routed_dp_rank remains authoritative, and FOLLOW_BOOTSTRAP_ROOM keeps its existing PD/disaggregation routing contract. The v0.5.17 path also follows the existing active-worker/bounded-routing behavior. This was kept as downstream release-backport validation rather than adding image-build-specific patching logic to the upstream PR.

[Phoenix3334]

Additional 2×2 runtime validation: generation ON/OFF × patched/unpatched

I completed a same-image v0.5.17 2×2 follow-up to isolate the generation-health trigger from the DPC fix. External /health polling remained active in every arm.

Environment: 8×B300, DeepSeek-V4-Pro, TP8/DP8, DP Attention, MegaMoE, PrefillDelayer, effective 8k chunk, C16/N160, 60k input / 1 output, seed=1.

DPC	health generation	HEALTH@DPC	Result
unpatched	ON	48–61/run	1 severe LOW + 1 mild degraded + 6 GOOD
unpatched	OFF	0	4/4 GOOD, 94.78–94.85k TPS
patched	ON	49–50/run	4/4 GOOD, 94.56–94.70k TPS
patched	OFF	0	4/4 GOOD, 94.85–94.93k TPS

The severe unpatched+ON hit reached 68,403 Input TPS, avgRun 5.07 and p99 TTFT 32.1s. Its active-segment global scheduler running-request BS p50 fell to 48 (healthy runs typically 72–80), and user dispatch across the 8 DP ranks became uneven ([30,30,28,31,28,29,28,28], std 1.12).

The important control is patched+ON: generation-health traffic was still present at DPC (~50 requests/run), but all four runs stayed GOOD and user dispatch stayed balanced. This directly supports the PR invariant that the fix isolates mutable health dispatch state from user load-balancing state rather than disabling health generation.

One refinement from this follow-up: PrefillDelayer timeout counters alone are not a sufficient failure discriminator; patched+ON runs can remain healthy while timeout events are still observed. The severe failure signature is better characterized by user-dispatch skew together with scheduler running-request/avgRun collapse and TPS/tail-latency degradation.

So the runtime interaction is now:

generation health + unpatched shared DPC state -> probabilistic LOW
generation health + isolated DPC state          -> stable GOOD

This complements the existing 8k/16k patched validation with a controlled trigger-isolation experiment.

[Phoenix3334]

Canonical evidence update / correction

I updated the PR body to reflect the completed 2x2 evidence campaign and to tighten the causal chain around the DPC routing-state invariant.

One correction to my earlier CI/review comment: I previously summarized the patched runs as having no PrefillDelayer timeouts. The canonical data shows patched HEALTH-ON GOOD runs did record wait_timeout=12-24. Those timeouts did not correlate with LOW state, so wait_timeout > 0 is not used as a failure discriminator.

Likewise, IDLE-while-EXTEND was 0 across all 27 analyzed rounds, including severe LOW cases. The canonical runtime chain is therefore now stated as:

generation health
  -> shared user LB state mutated
  -> DPC user-dispatch skew
  -> per-rank effective-batch divergence / reduced running work
  -> TPS collapse + TTFT-tail inflation

The key isolation result remains unchanged and is now supported more directly: patched HEALTH-ON keeps HEALTH@DPC nonzero (49-50) while restoring user dispatch to exact balance (29.0+-0.00) and keeping all four runs GOOD.

[Phoenix3334]

Closed-loop benchmark interpretation

A small clarification on the runtime signals: the validation workload is fixed-concurrency closed-loop, so client refill is driven by request completion rather than by an independent offered QPS.

In the post-fix 60k/1 concurrency sweep, raw perf.json request throughput reaches ~1.57 req/s at C16 and stays essentially flat through C24/C32/C64, while median TTFT grows roughly with concurrency. This identifies C16 as the closed-loop saturation knee.

That benchmark model also explains why avg_queue_req does not need to spike in the pathological LOW state. When the routing bug lowers service capacity (~1.58 -> ~1.14 req/s in the canonical comparison), the client refill rate falls with completions, providing backpressure. Queue depth can therefore remain similar even while running width collapses and TTFT tail worsens.

This is only a signal-interpretation note; it does not change the PR's causal claim. The direct mechanism remains:

generation health
  -> shared DPC user LB state mutation
  -> user dispatch skew
  -> per-rank running-batch divergence / work-supply loss
  -> TPS and TTFT degradation

The ~1.57 req/s saturation value is not a production SLO-capacity claim; that would require a separate fixed-rate/open-loop QPS sweep.

[Phoenix3334]

Follow-up: total_requests does not replace health/user LB-state isolation

I ran a new control to test whether switching away from round-robin is sufficient to avoid the LOW state without this patch.

Environment was held fixed: v0.5.17 unpatched, generation health ON, TP8/DP8, DP Attention, DP LM Head, MegaMoE, PrefillDelayer, effective 8k/rank chunk, C16/N160, 60k input / 1 output. The only policy change was:

--load-balance-method total_requests

Results:

Run	Input TPS	median TTFT	p99 TTFT	HEALTH@DPC	Result
r01	94,511.6	10.30 s	11.21 s	52	GOOD
r02	90,597.0	10.42 s	14.43 s	55	LOW
r03	90,713.8	10.38 s	14.44 s	50	LOW
r04	88,454.8	9.59 s	18.07 s	54	LOW

So this arm was only 1/4 GOOD, with mean Input TPS 91.07k. All 640/640 requests completed and there were no server fatals.

The useful part is that coarse request-count balance was often already good. For example, rank-local admission reconstruction for r02 was:

[55, 55, 54, 54, 54, 54, 53, 53]  (432/432 observed)

while the run was still LOW at 90.6k TPS. r04 was similarly close to even at [55, 53, 53, 55, 54, 55, 53, 53] (431/432 observed) but was the slowest run at 88.5k TPS.

This refines the downstream mechanism: persistent aggregate request-count skew is not required. A better description is:

generation health
  -> mutates shared user LB state / dispatch dynamics
  -> perturbs rank-local arrival/refill timing and work supply
  -> per-rank effective running/batch formation diverges
  -> TPS drops and TTFT tail inflates

total_requests can reduce coarse ownership imbalance, but in the unpatched path health requests still participate in the same user load-budget refresh/update machinery. It therefore acts, at best, as an unstable partial mitigation; it does not restore the invariant that health traffic must not mutate user-routing state.

One instrumentation caveat: the current DPC trace does not expose selected_dp_rank for the total_requests path, so the ownership vectors above are reconstructed from rank-local admission traces rather than direct DPC selected-rank counters. r02 is the cleanest example because all 432 observed user admissions were reconstructed.

I will treat a future patched + total_requests run only as a load-balancing policy/performance comparison; the correctness requirement addressed by this PR remains independent of the selected user LB policy.

[xiezhq-hermann]

@alphabetc1 can you help take a look, seems related to a line of work you have done about idleness check

[alphabetc1]

/tag-and-rerun-ci

[alphabetc1]

Overall lgtm
non-blocking: can we simplify the test? I think there are too many lines of test code, and a lot of them are unnecessary.

[Phoenix3334]

Thanks @alphabetc1! Simplified the tests in 902559f (8 → 6, 28 fewer lines), retaining the key regression coverage. Could you take another look and help rerun CI if needed?