Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/pr-test-rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -904,6 +904,26 @@ jobs:
kv_backend: mooncake
timeout: 40
min_selected: 15 # floor ≈ 95% of measured effective (e2e audit 2026-08-21)
# No tokenspeed row yet. The classes carry the marks and the
# per-engine model, so `E2E_RUNTIME=tokenspeed pytest e2e_test/router`
# runs the suite by hand — but this lane cannot gate merge until the
# concurrency mismatch below is settled.
#
# Sequential PD works: on run 34173426995 both TestPDMessagesGrpc
# cases passed against Qwen3.5-9B on a 1p1d TokenSpeed pair. The
# MMLU case then drives 32 concurrent requests at the model spec's
# `--max-num-seqs 4` (the only such cap in e2e_test/, inherited from
# the 4-GPU EPD smoke lane). With an admission window that far below
# the offered concurrency the two legs admit disjoint subsets and
# each waits on the other, so the engine's own PD guards fire:
# prefill "Some requests timed out when bootstrapping ... fail to
# receive the cache manifest from the decode instance", decode "Some
# requests fail to receive KV Cache transfer done signal". 14 of 64

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: This is now the only recorded way to run the suite, and as written it does not reproduce what the lane did. E2E_RUNTIME only picks the engine binary; the marker filter in pytest_collection_modifyitems is a no-op unless E2E_ENGINE/E2E_GPU_TIER are set (hooks.py:361-380 — if any([engine, vendor, gpu_tier])). So E2E_RUNTIME=tokenspeed pytest e2e_test/router collects the whole directory under the TokenSpeed runtime, including the pd_http classes (no HTTP mode on TokenSpeed) and the 4-GPU classes, rather than the 11 cases the deleted row would have selected.

The lane's own env is E2E_RUNTIME=tokenspeed E2E_ENGINE=tokenspeed E2E_GPU_TIER=2; worth spelling that out. Also worth a word that Qwen/Qwen3.5-9B carries skip_tier_download: True (e2e_test/infra/model_specs.py:203) and was fetched via the now-removed extra_models input, so a by-hand run needs the weights already present.

# requests died that way; the eval took 1392s against ~16s on the
# sglang and vllm rows and scored 0.609 under the 0.65 threshold.
# Re-add this row once the window covers the suite's concurrency
# (`--max-num-seqs` >= the eval's num_threads), together with the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: The prescribed re-entry condition points at a knob that isn't lane-scoped. --max-num-seqs 4 lives in the shared Qwen/Qwen3.5-9B tokenspeed_args (e2e_test/infra/model_specs.py:189-191), which is also what the gating e2e-4gpu-epd job runs. Raising it to ≥ 32 there raises the engine's KV reservation for that lane too, at --gpu-memory-utilization 0.8 / --max-model-len 8192 on one H100 per role — so whoever acts on this comment can turn a currently-green gating lane red while trying to re-add a non-gating one.

Worth recording the alternatives next to it, since they don't have that blast radius: a role- or PD-scoped --max-num-seqs override in infra/worker.py, or an engine-aware num_threads in TestPDMMLUGrpc (test_pd_mmlu.py:80) — 32 is a suite-authored constant, not a property of what's under test, so lowering it for TokenSpeed keeps the eval's semantics intact.

# `extra_models` and `tokenspeed_prebuilt_image` inputs it needs.
uses: ./.github/workflows/e2e-gpu-job.yml
with:
engine: ${{ matrix.engine }}
Expand Down
10 changes: 4 additions & 6 deletions e2e_test/fixtures/hooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
get_zmq_engine_count,
)

from .markers import resolve_class_marker
from .markers import model_id_for_engine, resolve_class_marker

# Local wires a plain (non-PD/EPD) test case can be authored with; these are the
# only backends a ZMQ lane reuses (mapped onto ZMQ by the ``setup_backend``
Expand Down Expand Up @@ -131,7 +131,7 @@ def pytest_runtest_setup(item: pytest.Item) -> None:
# H100), so gpt-oss dies at model load on CI's H100 runners.
if current_runtime == "tokenspeed":
model_marker = resolve_class_marker(item, "model")
if model_marker and model_marker.args and "gpt-oss" in str(model_marker.args[0]):
if "gpt-oss" in str(model_id_for_engine(model_marker, current_runtime, default="")):
pytest.skip("tokenspeed cannot load gpt-oss on H100 (tokenspeed#1036)")


Expand Down Expand Up @@ -225,7 +225,7 @@ def _filter_tokenspeed_dp_items(
deselected: list[pytest.Item] = []
for item in items:
marker = resolve_class_marker(item, "model")
model = str(marker.args[0]) if marker is not None and marker.args else ""
model = str(model_id_for_engine(marker, "tokenspeed", default=""))
(deselected if model in _TOKENSPEED_DP_BROKEN_MODELS else kept).append(item)
return kept, deselected

Expand Down Expand Up @@ -424,9 +424,7 @@ def _pool_sort_key(item: pytest.Item) -> tuple:
backend = str(params.get("setup_backend", params.get("backend_router", "")))

model_marker = resolve_class_marker(item, "model")
model = ""
if model_marker is not None and model_marker.args:
model = str(model_marker.args[0])
model = str(model_id_for_engine(model_marker, get_runtime(), default=""))

return (backend, model, item.nodeid)

Expand Down
17 changes: 17 additions & 0 deletions e2e_test/fixtures/markers.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,20 @@ def get_marker_kwargs(
if marker is not None:
result.update(marker.kwargs)
return result


def model_id_for_engine(marker: pytest.Mark | None, engine: str | None, default: Any = None) -> Any:
"""Resolve the model id a ``@pytest.mark.model`` marker names for ``engine``.

``@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B")``
keeps one class portable across engines whose supported model sets differ:
the positional id is the default and a keyword named after an engine
overrides it for that engine only.
"""
if marker is None:
return default
if engine and engine in marker.kwargs:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: An unrecognized kwarg key falls through silently to the positional default. @pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeeed="Qwen/Qwen3.5-9B") (typo) or a rename of an engine id would quietly run the TokenSpeed lane on Llama — exactly the failure this helper exists to prevent, with no error anywhere: the lane goes green having tested the wrong model. Since the only legal kwarg keys are engine ids, consider validating against the known set (the same one @pytest.mark.engine accepts) and raising on anything else:

if marker.kwargs:
    unknown = set(marker.kwargs) - KNOWN_ENGINES
    if unknown:
        raise ValueError(f"@pytest.mark.model got non-engine kwargs {sorted(unknown)}")

This is cheap here because the marker is resolved at collection time, so a typo fails the whole run rather than one lane.

return marker.kwargs[engine]
if marker.args:
return marker.args[0]
return default
4 changes: 2 additions & 2 deletions e2e_test/fixtures/setup_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
from infra.worker import stop_workers
from infra.worker_pool import get_pool

from .markers import get_marker_kwargs, get_marker_value
from .markers import get_marker_kwargs, get_marker_value, model_id_for_engine, resolve_class_marker

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -214,7 +214,7 @@ def setup_backend(request: pytest.FixtureRequest):
if os.environ.get(ENV_SKIP_BACKEND_SETUP, "").lower() in ("1", "true", "yes"):
pytest.skip(f"{ENV_SKIP_BACKEND_SETUP} is set")

model_id = get_marker_value(request, "model")
model_id = model_id_for_engine(resolve_class_marker(request.node, "model"), get_runtime())
if model_id is None:
model_id = os.environ.get(ENV_MODEL, DEFAULT_MODEL)

Expand Down
4 changes: 2 additions & 2 deletions e2e_test/router/test_pd_messages.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,9 @@ class TestPDMessagesHttp(MessagesOverPD):
"""Messages API through the HTTP PD router's dual dispatch."""


@pytest.mark.engine("sglang", "vllm")
@pytest.mark.engine("sglang", "vllm", "tokenspeed")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: The module docstring above is now stale for this class. Lines 8-15 still read "pd_grpc": gRPC mode (both SGLang and vLLM) and list requirements for SGLang and vLLM only, while E2E_RUNTIME=tokenspeed is now a third supported runtime for TestPDMessagesGrpc (its requirement being TokenSpeed's Mooncake transfer + --disaggregation-mode, per _build_tokenspeed_grpc_cmd in e2e_test/infra/worker.py). Same drift in test_pd_mmlu.py (lines 6-14) and test_pd_responses.py (lines 7-14) — the header is the first thing someone reads when a lane fails, so it's worth a one-line update in all three.

@pytest.mark.gpu(2)
@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct")
@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B")
@pytest.mark.e2e
@pytest.mark.parametrize("setup_backend", ["pd_grpc"], indirect=True)
class TestPDMessagesGrpc(MessagesOverPD):
Expand Down
4 changes: 2 additions & 2 deletions e2e_test/router/test_pd_mmlu.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,9 @@ def test_pd_mmlu_basic(self, setup_backend):
logger.info("PD HTTP MMLU score: %.2f (threshold: 0.65)", metrics["score"])


@pytest.mark.engine("sglang", "vllm")
@pytest.mark.engine("sglang", "vllm", "tokenspeed")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: This makes PD-MMLU the only MMLU case TokenSpeed runs in CI, so a score < 0.65 failure will be ambiguous between a PD/KV-transfer bug and plain engine/model quality. test_mmlu.py::TestMMLUGrpc is already marked engine("sglang", "vllm", "tokenspeed"), but no lane collects it for TokenSpeed: e2e-1gpu-gateway runs e2e_test/router for sglang/vllm only, and the e2e-1gpu-chat (tokenspeed) entry narrows test_dirs to e2e_test/chat_completions e2e_test/router/test_admin_ops.py (pr-test-rust.yml:647). Adding e2e_test/router/test_mmlu.py to that lane's test_dirs (and bumping its min_selected) would give this threshold a single-worker TokenSpeed baseline to be diffed against.

@pytest.mark.gpu(2)
@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct")
@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B")
@pytest.mark.e2e
@pytest.mark.parametrize("setup_backend", ["pd_grpc"], indirect=True)
class TestPDMMLUGrpc:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: After this push there is no lane that collects this class for TokenSpeed, so the tokenspeed engine mark is now a claim with no verification behind it — and per the workflow comment added in pr-test-rust.yml, this is the one case measured as broken on TokenSpeed (14/64 requests killed by the engine's PD guards, 1392s, score 0.609). The two things that could go wrong from here both start in this file, not in the workflow:

  • The by-hand run the workflow comment advertises lands on a ~23-minute hang, and nothing here points at the reason.
  • Any future lane that runs e2e_test/router with E2E_ENGINE=tokenspeed at tier 2 silently re-adopts this case, because the mark says it's supported.

A one-liner above the mark — # tokenspeed: gRPC PD serves sequentially, but this eval's num_threads=32 against the model spec's --max-num-seqs 4 deadlocks the legs (see e2e-2gpu-pd in pr-test-rust.yml) — closes both. TestPDMessagesGrpc and TestPDResponsesGrpc did pass, so this note belongs only here, which is also the useful signal: the mark on those two means something different from the mark on this one.

Expand Down
4 changes: 2 additions & 2 deletions e2e_test/router/test_pd_responses.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,9 @@ def test_streaming_response(self, model, api_client):
assert completed_events[0].response.status == "completed"


@pytest.mark.engine("sglang", "vllm")
@pytest.mark.engine("sglang", "vllm", "tokenspeed")
@pytest.mark.gpu(2)
@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct")
@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B")
@pytest.mark.e2e
@pytest.mark.gateway(extra_args=["--history-backend", "memory"])
@pytest.mark.parametrize("setup_backend", ["pd_grpc"], indirect=True)
Expand Down
Loading