Skip to content

[AMD] CI: stop installing transformers from git main in Qwen 3.5 and GLM-5 nightly tests#20748

Closed
michaelzhang-ai wants to merge 1 commit intosgl-project:mainfrom
michaelzhang-ai:fix/ci-remove-bleeding-edge-transformers
Closed

[AMD] CI: stop installing transformers from git main in Qwen 3.5 and GLM-5 nightly tests#20748
michaelzhang-ai wants to merge 1 commit intosgl-project:mainfrom
michaelzhang-ai:fix/ci-remove-bleeding-edge-transformers

Conversation

@michaelzhang-ai
Copy link
Copy Markdown
Collaborator

Summary

  • Remove pip install git+https://github.com/huggingface/transformers.git from all Qwen 3.5 and GLM-5 CI jobs across both nightly-test-amd.yml and nightly-test-amd-rocm720.yml
  • Use the stable transformers shipped in the docker image instead, matching all other passing nightly jobs (Grok2, DeepSeek-V3.2, etc.)
  • Keep mistral-common and lm-eval[api] for Qwen 3.5 tests that need them

Motivation

Transformers PR huggingface/transformers#41250 (merged Mar 16) converts PretrainedConfig subclasses to @dataclass via __init_subclass__, which breaks sglang's DeepseekVL2Config (non-default field vision_config follows default arguments). This crashes the server on import and prevents any test from running:

TypeError: non-default argument 'vision_config' follows default argument

Both the Qwen 3.5 MI35x and GLM-5 MI35x jobs failed in Nightly #633 for this reason, while Grok2 and DeepSeek-V3.2 (which don't install from git main) passed fine.

Installing bleeding-edge transformers from main is fragile — upstream breaking changes can silently break CI at any time. The stable version in the docker image is sufficient.

Test plan

  • Verify Qwen 3.5 nightly accuracy tests pass on MI30x and MI35x
  • Verify GLM-5 nightly accuracy tests pass on MI30x and MI35x
  • Verify ROCm 7.2 variants also pass

…GLM-5 nightly tests

Transformers PR huggingface/transformers#41250 (merged Mar 16) converts
PretrainedConfig subclasses to @DataClass via __init_subclass__, which
breaks sglang's DeepseekVL2Config (non-default field ordering) and
prevents the server from starting at all.

Remove `pip install git+https://github.com/huggingface/transformers.git`
from all Qwen 3.5 and GLM-5 CI jobs (MI30x, MI35x, ROCm 7.0 and 7.2).
Use the stable transformers shipped in the docker image instead, matching
all other nightly jobs (Grok2, DeepSeek-V3.2, etc.).

Keep mistral-common and lm-eval[api] for Qwen 3.5 tests that need them.
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant