[Apple Silicon] [CI] Move the MLX lane to the check-changes + pr-gate composite by jlee5814 · Pull Request #30121 · sgl-project/sglang

jlee5814 · 2026-07-04T15:21:28Z

Motivation

#29691 landed the MLX lane (pr-test-mlx.yml) with standalone gating. This PR began as a migration to the shared check-changes plus pr-gate composite used by the other hardware lanes. Implementations include: a separate sgl_kernel filter output following _pr-test-check-changes.yml, and a staged layout aligned with the other backends, wired for the smoke tests from #29440. All three are in this PR, so it is a gate migration plus a staged restructure. Per @yeahdongcn's review, the scope is inclusive of moving stage A onto run_suite.py suite registration, matching how every other hardware lane selects its tests instead of an explicit file list.

Modifications

Gate migration

A dorny/paths-filter check-changes job plus the reusable pr-gate workflow replace the inline label gate. pr-gate live fetches labels at runtime, so adding run-ci and rerunning failed jobs restores the suite. The old frozen
payload gate never could.
The workflow level permissions block is removed; it zeroed the pull-requests and actions scopes the paths filter and the gate API reads require. The vendor lanes running this same chain declare no block.

Filter split

main_package and sgl_kernel are separate filter outputs, following _pr-test-check-changes.yml; the sgl_kernel glob is copied verbatim from that file. pr-gate keys off a changes_exist aggregate, matching the MUSA and NPU lanes.
Stage A gates on main_package or sgl_kernel, matching the base stages of pr-test.yml, so the trigger surface is unchanged from the single key filter.

Staged layout and #29440 wiring

mlx-unit-test becomes stage-a-unit-test-mlx. Invocation, env, and runner are unchanged: model free unit tests, HF_HUB_OFFLINE=1, GitHub hosted macos-26.
A new stage-b-e2e-test-mlx job is defined for the [MLX] Add correctness tests for qwen2_moe and qwen3_moe #29440 smoke tests: served correctness for qwen2_moe and qwen3_moe plus the reference equivalence test. Those tests load 8 GB to 17 GB models and are tuned for 24 GB of unified memory; GitHub hosted macos-26 runners have 7 GB. Stage B therefore activates only through a workflow_dispatch target_stage input until a self hosted Apple Silicon runner registers. The mechanism is copied from the MUSA lane, with two corrections to make dispatch functional. Stage B currently has no scheduled execution; it fires only via the constrained dispatch input, pending the self hosted runner.
workflow_dispatch with a target_stage input is added for per stage dispatch; the input is a constrained type: choice following the AMD lane (a free text field, per the MUSA original, silently no-ops on a typo).
The finish job adopts the MUSA pattern: iterate over needs, fail on failure or cancellation, tolerate skipped.

Test registration

Stage A now runs via python3 run_suite.py --hw mlx --suite stage-a-unit-test-mlx instead of an explicit pytest file list, matching how the XPU lane invokes run_suite.py. This adds HWBackend.MLX and register_mlx_ci() to the shared ci_register.py, plus a stage-a-unit-test-mlx entry in run_suite.py's per-commit suites.
test_quantization.py mixed a model-free class (TestMlxQuantizationOverride, previously cherry-picked via pytest ::ClassName) with a model-downloading class (TestMlxQuantization) in one file. run_suite.py only runs whole files, so TestMlxQuantizationOverride moves to its own file, test_mlx_quantization_override.py, registered for both base-a-test-cpu and the new MLX suite. test_quantization.py keeps only the model-dependent class, now registered under stage-b-e2e-mlx (it performs real HF loads, so it belongs with the gated model requiring suite rather than stage-a).
Test registration is now structural: a new model-free MLX unit test file registers itself with register_mlx_ci(...), the same pattern every other backend uses, instead of a line in the workflow's explicit list.
A review pass surfaced three MLX test files whose assertions had never executed in any CI configuration (CPU registered with MLX skip guards: skipped on Linux, invisible to the MLX suite). test_mlx_pool_dtype.py and test_tp_worker_routing.py now run in stage-a (verified green under the lane's exact env locally); test_quantization.py reclassified to stage B as above.

Dependencies and deferrals

[MLX] Add correctness tests for qwen2_moe and qwen3_moe #29440 has been merged, the referenced files exist on main, and the stage now awaits only the self-hosted runner.
Shares the register_mlx_ci marker with [Apple Silicon] [MLX] Fuse MoE combine multiply-reduce into one Metal kernel #29804 (byte identical ci_register.py addition; merges cleanly in either order).

Accuracy Tests

CI only change; no model or kernel behavior affected. The FakeOverlapScheduler stub desync from #29217 (AttributeError on forward_ct) is fixed on main via #30125, and stage A is expected green against the merge commit.

Speed Tests and Profiling

N/A.

CI States

Latest PR Test (Base): ❌ Run #28996751813
Latest PR Test (Extra): ❌ Run #28996751681

gemini-code-assist · 2026-07-04T15:21:32Z

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8dfb833c69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jlee5814 · 2026-07-04T16:08:32Z

/tag-run-ci-label

yeahdongcn · 2026-07-06T00:39:32Z

-          # signature contracts, mocked Metal capture, dummy ServerArgs patching,
-          # and quant-config dicts.
          uv run python -m pytest -v \
            test/registered/unit/hardware_backend/mlx/test_runner_init_contract.py \


Since you're here, could we revisit the design of the multi-stage CI tests? It would be great to align them with the approach used by the other backends.

Also, could we incorporate the smoke tests from #29440 into the multi-stage CI as well?

Staged. Stage A: the existing macos-26 unit tests, unchanged. Stage B: wired the #29440 smoke tests, dispatch gated: they load 8 to 17 GB models while the hosted runners only have 7 GB. It activates once a self hosted runner exists and #29440 merges, which also fixes the stage A red on main (#29217).

Dispatch surfaced two MUSA bugs: input: for inputs:, so target_stage never registers, and missing base: main plus !cancelled().

check-changes → pr-gate → { stage-a, stage-b } → finish

jlee5814 · 2026-07-06T04:19:51Z

/tag-run-ci-label

yeahdongcn · 2026-07-07T01:36:23Z

+              - "test/**/!(*.md)"
+              - ".github/workflows/pr-test-mlx.yml"
+            sgl_kernel:
+              - "sgl-kernel/**/!(*.md|THIRDPARTYNOTICES.txt|LICENSE)"


Since this workflow now has a separate sgl_kernel filter output, I think .github/workflows/pr-test-mlx.yml should be listed under sgl_kernel as well, matching the MUSA/AMD split-filter pattern.

Added .github/workflows/pr-test-mlx.yml under the sgl_kernel filter.

yeahdongcn · 2026-07-07T01:37:27Z

+from sglang.test.ci.ci_register import register_cpu_ci, register_mlx_ci

 register_cpu_ci(est_time=1, suite="base-a-test-cpu")
+register_mlx_ci(est_time=1, suite="stage-a-unit-test-mlx")


Since this PR moves the MLX unit lane to run_suite.py, could we also tag the remaining model-free MLX unit tests with register_mlx_ci(..., suite="stage-a-unit-test-mlx")?

test/registered/unit/hardware_backend/mlx/test_mlx_runner_pool_contract.py

test/registered/unit/hardware_backend/mlx/test_scheduler_mixin.py

Both files are tagged into stage-a-unit-test-mlx (test_scheduler_mixin.py landed via #29217). This also establishes the pattern for new tests like #30097: a register_mlx_ci line in the test's own file.

yeahdongcn · 2026-07-07T01:43:25Z

-            test/registered/unit/hardware_backend/mlx/test_metal_profiler.py \
-            test/registered/unit/hardware_backend/mlx/test_attention_patching.py \
-            "test/registered/unit/hardware_backend/mlx/test_quantization.py::TestMlxQuantizationOverride"
+            test/registered/mlx/models_e2e/test_qwen2_moe_mlx_correctness.py \


PR #29440 hasn't been merged yet, and this still uses the list-based mechanism. Could we switch it to the run-suite style as well?

#29440 merged, unblocked. The two smoke tests plus test_mlx_reference_correctness.py (model) register under stage-b-e2e-mlx, added to run_suite.py's table; dispatch gate unchanged. test_metal_profiler fails on unmodified main (#30550).

… composite Replaces the inline mlx-label gate with the composite convention the other hardware lanes use: a dorny/paths-filter check-changes job carrying the sibling lanes' broad filter (python/sglang, test, sgl-kernel, the platform pyproject, and the workflow itself — the suite's contract tests import core srt, so narrow MLX-tree globs would miss the refactors they exist to catch), a pr-gate job calling the reusable run-ci gate, and mlx-unit-test hanging off both. The lane is now path triggered; 'labeled' leaves the trigger types since pr-gate live-fetches labels at runtime, so adding run-ci and re-running failed jobs un-skips the suite (the old frozen-payload gate never could). The workflow-level permissions block goes with the gate: an explicit block zeroes pull-requests and actions scopes, which the paths filter and the gate's API reads require; the vendor lanes running this same chain declare no block. Job names, env, steps, and the finish job are unchanged.

…ree tests into stage-a, register model-requiring tests under gated stage-b run-suite

yeahdongcn · 2026-07-09T01:19:09Z

 import sys
 from typing import Dict, List, Optional

 import tabulate


This workflow now calls test/run_suite.py, but the install step only installs python[srt_mps] and pytest. run_suite.py imports tabulate at module import time, and in python/pyproject_other.toml tabulate is only part of the test extra, not srt_mps.

Fixed, the install now covers the test extra (python[srt_mps,test]), which provides tabulate and subsumes the standalone pytest line. Applied to both stages since they share the install block.

yeahdongcn · 2026-07-09T01:19:47Z

+
+  stage-b-e2e-test-mlx:
+    needs: [check-changes, pr-gate]
+    # Inert until a self hosted Apple Silicon runner registers and #29440


Could you refresh this comment? #29440 is merged now, and this PR registers those smoke tests under stage-b-e2e-mlx. If stage B should remain manual-only because the self-hosted Apple Silicon runner is not generally available / hosted runners are too small, the comment should say that directly instead of saying it is waiting on #29440.

Fixed, it now states no available self-hosted Apple Silicon runner and that hosted runners too small for e2e.

…ting comment run_suite.py imports tabulate at module level, but the install step only covered python[srt_mps] plus a bare pytest; tabulate lives in the test extra of pyproject_other.toml. Extend both stage-a and stage-b install steps to python[srt_mps,test] (superset of the standalone pytest install). sgl-project#29440 merged, so the stage-b comment no longer describes the real blocker; restate it as manual-only pending a self hosted Apple Silicon runner.

…elong; constrain stage dispatch input to enum test_mlx_pool_dtype.py and test_tp_worker_routing.py were CPU-registered with MLX skip guards, so they never ran on Linux (skipped) or on the MLX lane (unregistered) -- verified green under the lane's exact env, now registered into stage-a-unit-test-mlx. test_quantization.py's TestMlxQuantization loads real HF models via MlxModelRunner; stage-a forbids downloads (HF_HUB_OFFLINE=1) and this was confirmed to fail with LocalEntryNotFoundError without a pre-warmed cache. Registered into stage-b-e2e-mlx instead, matching the models_e2e correctness tests' convention. target_stage moves from free-text to a choice enum (mirroring pr-test-amd.yml) so a typo can no longer silently no-op stage-b.

yeahdongcn · 2026-07-09T07:30:02Z

/tag-and-rerun-ci

…suites pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121. Register test_sliding_window_attention under stage-a-unit-test-mlx and test_gpt_oss_mlx_correctness under stage-b-e2e-mlx, mirroring the existing MLX test registrations.

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121.

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121. The macOS stage-a runner exercises the real MPS branch; elsewhere the test runs against the stub.

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121.

… composite (sgl-project#30121)

jlee5814 requested review from Fridge003, HaiShaw, Kangyan-Zhou, bingxche, ispobock and merrymercy as code owners July 4, 2026 15:21

github-actions Bot added the apple-silicon label Jul 4, 2026

chatgpt-codex-connector Bot reviewed Jul 4, 2026

View reviewed changes

Comment thread .github/workflows/pr-test-mlx.yml

jlee5814 force-pushed the mlx-lane-composite-gate branch from 8dfb833 to dfe0c49 Compare July 4, 2026 16:05

jlee5814 mentioned this pull request Jul 4, 2026

[Apple Silicon] [MLX] Fuse MoE combine multiply-reduce into one Metal kernel #29804

Open

5 tasks

LijuanTang94 mentioned this pull request Jul 4, 2026

[MLX] Fix FakeOverlapScheduler test stub broken by forward_ct accounting #30125

Merged

2 tasks

yeahdongcn reviewed Jul 6, 2026

View reviewed changes

yeahdongcn mentioned this pull request Jul 6, 2026

[MLX] Size the attention KV pool at the compute dtype for quantized models #30097

Merged

github-actions Bot added the quant LLM Quantization label Jul 6, 2026

yeahdongcn reviewed Jul 7, 2026

View reviewed changes

jlee5814 added 7 commits July 8, 2026 12:32

Split sgl_kernel into its own MLX lane filter output

08f6094

Restructure MLX lane into staged jobs with dispatch gating

229362d

Wire sgl-project#29440 smoke tests into a gated MLX stage b

3de9e40

Tighten MLX lane workflow comments

03cc88c

Run MLX stage A via run_suite.py registration

eba6068

ci(mlx): list workflow under sgl_kernel filter, tag remaining model-f…

0efacae

…ree tests into stage-a, register model-requiring tests under gated stage-b run-suite

jlee5814 force-pushed the mlx-lane-composite-gate branch from 2bf234a to 0efacae Compare July 8, 2026 16:47

jlee5814 mentioned this pull request Jul 8, 2026

[Bug] [MLX] SchedulerProfilerManager._start_profile returns success=False under mocked Metal capture #30550

Open

5 tasks

yeahdongcn reviewed Jul 9, 2026

View reviewed changes

jlee5814 added 2 commits July 8, 2026 23:42

yeahdongcn approved these changes Jul 9, 2026

View reviewed changes

github-actions Bot added the run-ci label Jul 9, 2026

Kangyan-Zhou merged commit a91c2e6 into sgl-project:main Jul 11, 2026
222 of 265 checks passed

chenkaiyue mentioned this pull request Jul 11, 2026

Graceful shutdown with SIGTERM for child processes #16484

Open

alisonshao mentioned this pull request Jul 16, 2026

ci: add MLX to coverage report backend display order #31442

Merged

LarrySimingDeng added a commit to LarrySimingDeng/sglang that referenced this pull request Jul 26, 2026

Register windowed KV cache unit test in the MLX stage-a CI suite

3b6d60e

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121.

LarrySimingDeng added a commit to LarrySimingDeng/sglang that referenced this pull request Jul 26, 2026

Register SWA radix pool unit test in the MLX stage-a CI suite

a32cfe1

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121.

LarrySimingDeng added a commit to LarrySimingDeng/sglang that referenced this pull request Jul 26, 2026

Register decode-KV pool sync unit test in the MLX stage-a CI suite

f0f0258

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121.

LarrySimingDeng added a commit to LarrySimingDeng/sglang that referenced this pull request Jul 26, 2026

Register MLX pool sizing unit test in the MLX stage-a CI suite

8b0f680

pr-test-mlx.yml selects stage A tests via run_suite.py suite registration since sgl-project#30121.

This was referenced Jul 26, 2026

[MLX] Support gpt-oss: sliding-window attention, attention sinks, sm_scale #30050

Merged

[Apple Silicon][MLX] Fix scheduler OOM on long prompts and concurrent requests #29439

Open

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[Apple Silicon] [CI] Move the MLX lane to the check-changes + pr-gate…

8df9345

… composite (sgl-project#30121)

jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026

[Apple Silicon] [CI] Move the MLX lane to the check-changes + pr-gate…

6d98770

… composite (sgl-project#30121)

Uh oh!

Conversation

jlee5814 commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

CI States

Uh oh!

gemini-code-assist Bot commented Jul 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

jlee5814 commented Jul 4, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jlee5814 Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jlee5814 commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeahdongcn commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jlee5814 commented Jul 4, 2026 •

edited

Loading

jlee5814 Jul 6, 2026 •

edited

Loading