Skip to content

[CI/Build] Pin Cython below 3.3 for arm64 tilelang sdist - #53358

Merged
khluu merged 1 commit into
mainfrom
fix/arm64-tilelang-cmake4
Aug 23, 2026
Merged

khluu merged 1 commit into
mainfrom
fix/arm64-tilelang-cmake4

Conversation

@khluu

@khluu khluu commented Aug 22, 2026

Copy link
Copy Markdown
Member

Purpose

Fix the deterministic arm64 image-build failure caused by Cython 3.3 while building tilelang==0.1.12 from source. No test runs: the image fails during dependency installation.

Exact reproductions on the restored immutable builder image:

Root cause

The aarch64 build base is manylinux 2_28 / glibc 2.28, so it cannot install tilelang's manylinux_2_34 aarch64 wheel and uv builds tilelang from sdist. Fresh isolated builds now resolve Cython 3.3, which rejects tilelang's hardcoded Py_LIMITED_API=0x03080000 because Cython 3.3 requires Python Limited API 3.9 or newer. The previous successful build used the same immutable builder before Cython 3.3 was released, so it resolved an older compatible Cython.

Change

On aarch64 only, set uv's isolated-build constraint to cython<3.3 before installing CUDA requirements. x86_64 continues to install tilelang's wheel and is unaffected. CMake is no longer constrained; the restored builder remains on its existing CMake 4.4.2.

Duplicate-work check

Searched open PRs for cython arm64 tilelang, cython 3.3, and tilelang build. This PR was already the only open match, so it was narrowed in place instead of opening a duplicate.

Test status

  • uvx --from pre-commit pre-commit run --files docker/Dockerfile: passed, including Dockerfile dependency-graph and versions.json validation.
  • Exact updated-head arm64 image #85232 at 66ad0efdceb4: passed exit 0 in 39m34s, including the tilelang sdist build and complete vLLM image build/push under CMake 4.4.2.
  • Merged as main e25c586b9030; post-merge main arm64 #85235 passed exit 0 in 39m21s, including the complete image build and registry push.
  • Post-merge GH200 #85236 at 7f17348fd67c (exact merge commit plus only the selector scaffold key: gh200-test on a non-main validation branch) completed the full wheel build and reached Docker build 17/17 with neither build-tool failure. It stopped only at the separately baseline-known wheel-size gate: 561.96 MB > 500 MB (baseline #84342: 552.17 MB > 500 MB). No retry.

Post-merge arm64 and GH200 both cleared the Cython/Limited-API and fatal CUDA imported-location signatures. This incident is resolved on main; the GH200 wheel-size gate remains separate baseline debt.

AI-assistance disclosure

This change was prepared with AI assistance and reviewed before submission.

@mergify mergify Bot added the ci/build label Aug 22, 2026
@khluu
khluu force-pushed the fix/arm64-tilelang-cmake4 branch from 436092f to b53b2f5 Compare August 22, 2026 07:04
@vllm-agent

Copy link
Copy Markdown
Contributor

Exact arm64 validation is terminal green at current head b53b2f5e293371940bd5be80ce3de79992b2e627.

This is proposed-fix validation only. Final resolution still requires merge plus a post-merge cold/main Build arm64 image pass.

@khluu

khluu commented Aug 22, 2026

Copy link
Copy Markdown
Member Author

New evidence after PR #53374 merged changes the minimum fix:

  • Restored CMake 4.4.2 image still failed exact head #85174 and post-merge main #85175.
  • The recorded failed Ninja subcommand is now the generated Cython wrapper: stable Cython 3.3.0 (uploaded today at 05:16 UTC) rejects tilelang's Py_LIMITED_API=3.8.
  • Historical green #84899 predates that Cython release, explaining why the same immutable image passed then.

This PR's broader cmake<4 + cython<3.3 combination remains exact-gate green in #85162, but after the image rollback the smallest follow-up should be rebase onto current main and narrow to only the aarch64 cython<3.3 isolated-build constraint, then run one fresh exact arm64 gate. Please do not merge the current draft as-is based solely on #85162.

@khluu

khluu commented Aug 22, 2026

Copy link
Copy Markdown
Member Author

Superseding my earlier Cython-only minimum-fix note with exact GH200 evidence:

  • Daily #85202 GH200 still fails tilelang configuration under CMake 4.4 because CUDA::cuda_driver / CUDA::nvml imported locations are unset. This proves the cmake<4 constraint remains necessary for GH200 even after PR [CI][ARM64] Restore known-good CUDA 13 builder image #53374 restored the builder image.
  • Valid isolated #85210 GH200 ran PR code b53b2f5e2933 plus only key: gh200-test as a selector scaffold. It cleared both CMake and Cython failures, built the aarch64 wheel, and reached the final size check.
  • The only terminal error was the pre-existing size gate: 553.33 MB > 500 MB, matching baseline #84342 at 552.17 MB > 500 MB. No test ran because this optional job is a build/wheel gate.

Together with literal-head arm64 #85162 passing, this validates the current paired cmake<4 + cython<3.3 fix across both affected aarch64 paths. The PR remains draft for human review. Post-merge confirmation should require the arm64 image to pass and GH200 to clear these exact build-tool signatures again; the separately known wheel-size debt may still leave GH200 red.

The aarch64 builder cannot install tilelang's manylinux_2_34 wheel, so uv builds tilelang from sdist. Cython 3.3 rejects tilelang's hardcoded Python Limited API 3.8 setting before tests can run.

Constrain uv's isolated aarch64 build environments to Cython below 3.3. x86_64 continues to install tilelang's wheel and is unaffected.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
@khluu
khluu force-pushed the fix/arm64-tilelang-cmake4 branch from b53b2f5 to 66ad0ef Compare August 23, 2026 07:29
@khluu khluu changed the title [CI/Build] Fix arm64 image build: force CMake 3.x for tilelang sdist [CI/Build] Pin Cython below 3.3 for arm64 tilelang sdist Aug 23, 2026
@khluu
khluu marked this pull request as ready for review August 23, 2026 08:09
@khluu
khluu requested a review from Harry-Chen as a code owner August 23, 2026 08:09

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@khluu
khluu merged commit e25c586 into main Aug 23, 2026
12 of 14 checks passed
@khluu
khluu deleted the fix/arm64-tilelang-cmake4 branch August 23, 2026 08:09
khluu added a commit that referenced this pull request Aug 24, 2026
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
(cherry picked from commit e25c586)
yonromai pushed a commit to marin-community/vllm that referenced this pull request Aug 26, 2026
…t#53358)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
(cherry picked from commit e25c586)
yonromai added a commit to marin-community/vllm that referenced this pull request Aug 28, 2026
* Add schema 2 GrugMoE inference

* [CI/Build] Pin Cython below 3.3 for arm64 tilelang sdist (vllm-project#53358)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
(cherry picked from commit e25c586)

* Deterministic MoE combine (reduce_scatterv) under VLLM_BATCH_INVARIANT

Signed-off-by: shijuzhao <758710341@qq.com>

Signed-off-by: shijuzhao <shijuzhao@tencent.com>

* Clarify Hero GrugMoE support

* Remove redundant GrugMoE state

---------

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Signed-off-by: shijuzhao <shijuzhao@tencent.com>
Co-authored-by: yoblin <268258002+yoblin@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: shijuzhao <shijuzhao@tencent.com>
am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
…t#53358)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
mikeshawcode pushed a commit to mikeshawcode/vllm that referenced this pull request Sep 1, 2026
…t#53358)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: mikeshawcode <michaelwshaw2@gmail.com>
yonromai pushed a commit to marin-community/vllm that referenced this pull request Sep 4, 2026
…t#53358)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
(cherry picked from commit e25c586)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants