[faresobeid]

Why

The vllm-router health checker and circuit breaker use the router defaults: 5 s probe timeout, 3 failed probes, circuit breaker on.

On vLLM 0.31, /health only checks AsyncLLM.errored; it does not wait on the engine core. Long prefills and weight updates run in the engine-core process, so they do not delay it. What can delay it is a stalled API-server event loop (e.g. a gen-2 GC pause or a large JSON encode). #3900 removes the main cause of those stalls. These settings are a safety net on top of that: one slow probe or a burst of aborted or timed-out requests should not take a healthy engine out of rotation in the middle of a run.

What

New inference.router fields (vllm-router only):

field	default	router flag
health_check_timeout_secs	60	--health-check-timeout-secs
health_failure_threshold	10	--health-failure-threshold
circuit_breaker_failure_threshold	None = off	--cb-failure-threshold N or --disable-circuit-breaker

The probe interval stays at the router default (20 s). Tradeoff: with these defaults a dead worker can stay in rotation for up to about 10 × (20 + 60) s before the router drops it.

Both launch paths pass the flags: the local start_router and _launch_router.sh.j2 (SLURM, regular and P/D). The router docs list the new fields.

Checks

• pytest tests/unit/orchestrator tests/unit/test_configs.py -m "not gpu" and ruff check / ruff format --check pass.
• vllm-router 0.2.2 (the pinned wheel) accepts every flag used here.

[faresobeid]

Closing: with #3900 removing the API-server GC stalls, the router's default health checks no longer eject healthy engines (worst /health stall 3.7 s at GLM-4.5-Air scale on vLLM 0.31, under the 5 s timeout), so these extra settings aren't needed.