upgrade to support latest transformers release#2984
Conversation
📝 Walkthrough""" WalkthroughThis update modifies dependency versions, workflow matrix entries, and dependency handling for compatibility between vllm and specific PyTorch versions. It adds a training argument, safeguards coverage upload in CI scripts, and adjusts imports for compatibility with transformers library changes. No exported/public API signatures are changed. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Suggested labels
Suggested reviewers
📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
.github/workflows/multi-gpu-e2e.yml(1 hunks)cicd/multigpu.sh(1 hunks)requirements.txt(2 hunks)setup.py(3 hunks)src/axolotl/core/builders/base.py(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
.github/workflows/multi-gpu-e2e.yml (1)
Learnt from: NanoCode012
PR: #2854
File: README.md:73-77
Timestamp: 2025-07-02T02:56:20.788Z
Learning: For Axolotl Docker commands, the --ipc=host flag should be included by default to prevent shared memory failures that commonly occur with PyTorch DataLoaders and multiprocessing during machine learning training workflows.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: test-axolotl-multigpu (126, 12.6.3, 3.11, 2.7.0, 2, true)
- GitHub Check: test-axolotl-multigpu (126, 12.6.3, 3.11, 2.6.0, 2, true)
- GitHub Check: PyTest (3.11, 2.7.0)
- GitHub Check: PyTest (3.11, 2.7.1)
- GitHub Check: PyTest (3.11, 2.6.0)
- GitHub Check: PyTest from Source Dist (3.11, 2.7.1)
- GitHub Check: PyTest from Source Dist (3.11, 2.7.0)
- GitHub Check: PyTest from Source Dist (3.11, 2.6.0)
🔇 Additional comments (8)
requirements.txt (3)
22-22: hf_xet 1.1.5 is backward-compatible and safe to upgradeBased on the official release notes, the hf_xet 1.1.5 bump:
- Introduces performance optimizations for faster Xet file downloads
- Remains compatible with Python 3.8+
- Maintains backward compatibility—no breaking changes or workflow updates required
You can proceed with the upgrade as specified in your requirements.txt.
71-71: mistral-common 1.8.3 upgrade is backward-compatible with existing text-only workflowsThe jump from 1.7.0 → 1.8.3 adds robust image/multimodal support (encoding images via PIL or URLs, token & image counting, Pixtral model compatibility) but retains all prior text-only tokenization APIs. No breaking changes are documented for existing text pipelines.
• File requiring review:
– requirements.txt (line 71:mistral-common==1.8.3)Recommendation:
- If you’re not leveraging any new image/multimodal features, no code changes are needed.
- Otherwise, verify any multimodal pipelines against the updated tokenizer API examples in the Mistral-Common 1.8.x docs.
16-16: Review transformers 4.54.0 upgrade for breaking changesThe bump from 4.53.2→4.54.0 introduces several silent breaking changes and emergent bugs. Please verify that your code and pipelines remain compatible:
- Hidden-state indexing has shifted by +1 for Llama (and some other) models when using
return_hidden_states=True(PR #39120). If you rely on a specific ordering of hidden states, update your post-processing accordingly.- ImportError:
deterministic_gis no longer available inmodeling_flash_attention_utils. Audit any flash-attention or custom attention integrations for this missing symbol.- Runtime errors in exaone4 models if
sliding_window_patternremainsNone(“LLLG”): add guards or update your model-setup code.- No CVEs or major security advisories reported for 4.54.0, but several user-filed issues have appeared within 24 hours of release—run your full test suite against 4.54.0 before deploying.
If you do not use Llama hidden states, flash-attention utils or exaone4 models, the risk may be minimal. Otherwise, ensure you’ve run integration tests and adjusted any affected code paths.
cicd/multigpu.sh (1)
22-25: Excellent defensive programming for CI robustness.The conditional check for
CODECOV_TOKENbefore attempting coverage upload is a good practice that prevents CI failures when the token is unavailable. The|| truefallback ensures the script continues even if upload fails.src/axolotl/core/builders/base.py (1)
503-503: Confirm distributed training stability with per‐device token averagingSetting
average_tokens_across_devices=Falsedisables global loss/gradient averaging across GPUs, causing each device to backpropagate based solely on its local token batch. This can lead to uneven gradient scales and may require tailored learning rates or batch sizes.• File: src/axolotl/core/builders/base.py (line 503)
• Verify this matches your intended distributed setup
• Ensure LR and batch sizes are tuned for local‐only scaling
• Confirm model convergence and training stability in multi‐GPU runs.github/workflows/multi-gpu-e2e.yml (1)
40-40: Matrix changes align with vllm compatibility updates.The swap of
axolotl_extrasvalues between PyTorch 2.7.0 and 2.7.1 correctly reflects the updated vllm compatibility handling insetup.py. This ensures the CI tests match the actual dependency constraints.Also applies to: 47-47
setup.py (2)
71-72: Confirm intentional removal of vllm extras due to incompatibilityWe’ve verified that vllm 0.9.x exhibits multiple breaking changes with Transformers 4.54.0—renamed internals, model‐loading failures (VLLM-001), and missing architecture support—making it incompatible out of the box. Removing the
"vllm"extras fromsetup.pyfor PyTorch 2.7.0 patch is therefore the correct approach. If continued vllm support is required, you’ll need to either:
- Pin Hugging Face Transformers to a known-compatible version (e.g. ≤ 4.47.1), or
- Upgrade to a newer vllm release with updated Transformers integration once available.
88-90: vllm extras removal safe for PyTorch ≥2.4
All references to vllm in the codebase are either:
- Deferred via runtime imports in the CLI (
do_vllm_serve),- Wrapped in test‐level guards (
is_vllm_installed/require_vllm), or- Limited to schema definitions that don’t import the vllm package at load time.
Removing the “vllm” extra for PyTorch 2.4.x and 2.5.x will not break existing functionality for users on those versions.
| ], | ||
| "vllm": [ | ||
| "vllm==0.7.2", | ||
| "vllm==0.10.0", |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify vllm 0.10.0 compatibility and breaking changes.
The significant version bump from vllm 0.7.2 to 0.10.0 may introduce breaking changes. Please verify compatibility with the current codebase.
🌐 Web query:
What are the major changes and breaking changes in vllm 0.10.0 compared to 0.7.2?
💡 Result:
vLLM 0.10.0 introduces major engine codebase cleanups and numerous breaking changes compared to v0.7.2, particularly the removal of legacy V0 backends and features. Several model and backend supports have been dropped, and new model families and architectural expansions have been added. Users upgrading from 0.7.2 should expect required changes to their deployments and code.
Major Breaking Changes from 0.7.2 to 0.10.0
1. Removal of V0 Backends and Features
The v0.10.0 release initiates a significant "cleanup of V0 engine codebase." The following components are removed from vLLM[1][4]:
- V0 CPU, XPU, TPU, and HPU backends ([#20412])
- Long context LoRA ([#21169])
- Prompt Adapters ([#20588])
- Phi3-Small & BlockSparse Attention ([#21217])
- Spec Decode workers ([#21152])
vLLM maintainers indicate ongoing plans to remove further unused/legacy code related to V0. This means users must migrate to the newer V1 backends and workflows. Code, scripts, or container setups relying on V0-specific backends will break and must be updated[1][4].
2. Model Format and Loader Changes
Some older model families (especially those dependent on dropped V0 features) may no longer be supported. Expanded support is added for new models (e.g., Llama 4 with EAGLE, EXAONE 4.0, additional transformers, etc.), but legacy models using V0-only features will require migration[1][4].
3. Dependency and Backend Compatibility
- Changed default backends for several architectures (e.g., ROCm is now V1 by default)[4].
- GGUF Format Dependency Issues: Users report that vLLM 0.10.0's required version of the
gguflibrary is not compatible withnumpy 2.xout of the box, leading to loading failures for some model types unless a newerggufversion is manually installed—altering this may break some models loading via vLLM[2]. This type of change was less significant in 0.7.x.
4. Architectural and API Expansions
- Support for new architectures (attention-free models, Mamba2 layers, hybrid SSM/attention, etc.)[1][4].
- Expanded model and task support, such as LlamaForSequenceClassification and VLM (Vision Language Model) improvements.
Additional Changes and Considerations
- Bug Fixes and Performance: Numerous bug fixes, performance improvements, and upgraded handling for certain models and architectures[4].
- Backward Compatibility: Some plugins and downstream tools (such as vLLM Ascend) introduce version logic to support both new and up to two recent vLLM versions, but expect plugin authors to note and flag breaking changes[3].
Upgrade Caveats
- If you use custom deployment scripts, Dockerfiles, or automation around V0-specific features, you will need substantial updates to move to 0.10.0.
- Plugins and projects wrapping vLLM (such as vLLM Ascend) are expected to adapt to these breakages, but users should check respective documentation for transition recommendations.
- You may need to pin or update dependencies like
ggufandnumpydue to compatibility changes in the 0.10.x line[2]. - Some features present in 0.7.2 may not have direct replacements (notably, Prompt Adapters and long-context LoRA for V0 have no official migration documented with this release).
In summary:
vLLM 0.10.0 is a major step forward, streamlining the codebase and dropping legacy backends and features. Most of the breaking changes result from the removal of the V0 engine and associated model support, necessitating a migration to newer workflows. Users upgrading from 0.7.2 will need to review all deployment scripts, dependencies, and model types to ensure continued support and should expect non-trivial migration work[1][4][2].
Citations:
- 1: https://github.com/vllm-project/vllm/releases
- 2: [Installation]: GGUF Limitation (newbyteorder) vllm-project/vllm#16134
- 3: https://vllm-ascend.readthedocs.io/en/latest/community/versioning_policy.html
- 4: https://newreleases.io/project/pypi/vllm/release/0.10.0
- 5: https://docs.llamaindex.ai/en/v0.10.19/changes/changelog.html
- 6: [Installation]: no version of pip install vllm works - Failed to initialize NumPy: No Module named 'numpy' vllm-project/vllm#11037
- 7: https://docs.llamaindex.ai/en/v0.10.17/changes/changelog.html
- 8: https://docs.vllm.ai/en/latest/contributing/
- 9: https://discuss.python.org/t/proposal-officially-deprecate-support-for-legacy-single-phase-init-extension-modules/89262
- 10: https://kserve.github.io/website/0.13/blog/articles/2023-10-08-KServe-0.11-release/
Action Required: Migrate to vLLM 0.10.0 Breaking Changes
Upgrading from vLLM 0.7.2 → 0.10.0 drops all V0 backends (CPU, XPU, TPU, HPU), long-context LoRA, Prompt Adapters, Phi3-Small/BlockSparse attention, Spec Decode workers, and more. You must:
• Migrate any V0-specific code, scripts or deployment containers to the new V1 backends.
• Review all models you load—legacy V0-only formats may no longer be supported.
• Pin or upgrade gguf to a version compatible with numpy 2.x (0.10.0’s default may fail otherwise).
• Update default backend settings (e.g. ROCm is now V1).
• Remove or replace any Prompt Adapter or long-context LoRA usage.
• Run integration tests against your target model families and workflows to confirm nothing breaks.
File: setup.py (line 157) – bumping to “vllm==0.10.0” is correct, but ensure the above migration steps are completed.
🤖 Prompt for AI Agents
In setup.py at line 157, you have updated the vLLM version to 0.10.0, which
introduces breaking changes. To fix this, migrate all V0-specific code, scripts,
and deployment containers to use the new V1 backends; review and update all
model loading code to ensure compatibility with supported formats; pin or
upgrade the gguf dependency to a version compatible with numpy 2.x; update
default backend settings to reflect changes like ROCm now being V1; remove or
replace any usage of Prompt Adapters or long-context LoRA; and run thorough
integration tests on your target models and workflows to verify nothing breaks
after the upgrade.
1 participant
Summary by CodeRabbit
New Features
Bug Fixes
Chores