Repository navigation
ci(e2e): run the gRPC PD suite on TokenSpeed #2463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
1d01e00
dc292ab
4ae2a7e
e925ca6
1958275
bd98a60
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -904,6 +904,26 @@ jobs: | |
| kv_backend: mooncake | ||
| timeout: 40 | ||
| min_selected: 15 # floor ≈ 95% of measured effective (e2e audit 2026-08-21) | ||
| # No tokenspeed row yet. The classes carry the marks and the | ||
| # per-engine model, so `E2E_RUNTIME=tokenspeed pytest e2e_test/router` | ||
| # runs the suite by hand — but this lane cannot gate merge until the | ||
| # concurrency mismatch below is settled. | ||
| # | ||
| # Sequential PD works: on run 34173426995 both TestPDMessagesGrpc | ||
| # cases passed against Qwen3.5-9B on a 1p1d TokenSpeed pair. The | ||
| # MMLU case then drives 32 concurrent requests at the model spec's | ||
| # `--max-num-seqs 4` (the only such cap in e2e_test/, inherited from | ||
| # the 4-GPU EPD smoke lane). With an admission window that far below | ||
| # the offered concurrency the two legs admit disjoint subsets and | ||
| # each waits on the other, so the engine's own PD guards fire: | ||
| # prefill "Some requests timed out when bootstrapping ... fail to | ||
| # receive the cache manifest from the decode instance", decode "Some | ||
| # requests fail to receive KV Cache transfer done signal". 14 of 64 | ||
| # requests died that way; the eval took 1392s against ~16s on the | ||
| # sglang and vllm rows and scored 0.609 under the 0.65 threshold. | ||
| # Re-add this row once the window covers the suite's concurrency | ||
| # (`--max-num-seqs` >= the eval's num_threads), together with the | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: The prescribed re-entry condition points at a knob that isn't lane-scoped. Worth recording the alternatives next to it, since they don't have that blast radius: a role- or PD-scoped |
||
| # `extra_models` and `tokenspeed_prebuilt_image` inputs it needs. | ||
| uses: ./.github/workflows/e2e-gpu-job.yml | ||
| with: | ||
| engine: ${{ matrix.engine }} | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -83,3 +83,20 @@ def get_marker_kwargs( | |
| if marker is not None: | ||
| result.update(marker.kwargs) | ||
| return result | ||
|
|
||
|
|
||
| def model_id_for_engine(marker: pytest.Mark | None, engine: str | None, default: Any = None) -> Any: | ||
| """Resolve the model id a ``@pytest.mark.model`` marker names for ``engine``. | ||
|
|
||
| ``@pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B")`` | ||
| keeps one class portable across engines whose supported model sets differ: | ||
| the positional id is the default and a keyword named after an engine | ||
| overrides it for that engine only. | ||
| """ | ||
| if marker is None: | ||
| return default | ||
| if engine and engine in marker.kwargs: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: An unrecognized kwarg key falls through silently to the positional default. if marker.kwargs:
unknown = set(marker.kwargs) - KNOWN_ENGINES
if unknown:
raise ValueError(f"@pytest.mark.model got non-engine kwargs {sorted(unknown)}")This is cheap here because the marker is resolved at collection time, so a typo fails the whole run rather than one lane. |
||
| return marker.kwargs[engine] | ||
| if marker.args: | ||
| return marker.args[0] | ||
| return default | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -99,9 +99,9 @@ class TestPDMessagesHttp(MessagesOverPD): | |
| """Messages API through the HTTP PD router's dual dispatch.""" | ||
|
|
||
|
|
||
| @pytest.mark.engine("sglang", "vllm") | ||
| @pytest.mark.engine("sglang", "vllm", "tokenspeed") | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: The module docstring above is now stale for this class. Lines 8-15 still read |
||
| @pytest.mark.gpu(2) | ||
| @pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct") | ||
| @pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B") | ||
| @pytest.mark.e2e | ||
| @pytest.mark.parametrize("setup_backend", ["pd_grpc"], indirect=True) | ||
| class TestPDMessagesGrpc(MessagesOverPD): | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -60,9 +60,9 @@ def test_pd_mmlu_basic(self, setup_backend): | |
| logger.info("PD HTTP MMLU score: %.2f (threshold: 0.65)", metrics["score"]) | ||
|
|
||
|
|
||
| @pytest.mark.engine("sglang", "vllm") | ||
| @pytest.mark.engine("sglang", "vllm", "tokenspeed") | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: This makes PD-MMLU the only MMLU case TokenSpeed runs in CI, so a |
||
| @pytest.mark.gpu(2) | ||
| @pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct") | ||
| @pytest.mark.model("meta-llama/Llama-3.1-8B-Instruct", tokenspeed="Qwen/Qwen3.5-9B") | ||
| @pytest.mark.e2e | ||
| @pytest.mark.parametrize("setup_backend", ["pd_grpc"], indirect=True) | ||
| class TestPDMMLUGrpc: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: After this push there is no lane that collects this class for TokenSpeed, so the
A one-liner above the mark — |
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Nit: This is now the only recorded way to run the suite, and as written it does not reproduce what the lane did.
E2E_RUNTIMEonly picks the engine binary; the marker filter inpytest_collection_modifyitemsis a no-op unlessE2E_ENGINE/E2E_GPU_TIERare set (hooks.py:361-380—if any([engine, vendor, gpu_tier])). SoE2E_RUNTIME=tokenspeed pytest e2e_test/routercollects the whole directory under the TokenSpeed runtime, including thepd_httpclasses (no HTTP mode on TokenSpeed) and the 4-GPU classes, rather than the 11 cases the deleted row would have selected.The lane's own env is
E2E_RUNTIME=tokenspeed E2E_ENGINE=tokenspeed E2E_GPU_TIER=2; worth spelling that out. Also worth a word thatQwen/Qwen3.5-9Bcarriesskip_tier_download: True(e2e_test/infra/model_specs.py:203) and was fetched via the now-removedextra_modelsinput, so a by-hand run needs the weights already present.