Add Code World Model (CWM) #41199
Merged
Add Code World Model (CWM) #41199
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
zucchini-nlp
left a comment
zucchini-nlp
left a comment
There was a problem hiding this comment.
Thanks, left some comments to clean up. Btw, do we have converted weights already which we can use for the integration tests?
zucchini-nlp
approved these changes
Oct 1, 2025
Member
|
Can we update slow test ids and then I will trigger the slow CI? Overall lgtm and we can merge |
Member
|
run-slow: cwm |
Contributor
|
This comment contains run-slow, running the specified jobs: models: ['models/cwm'] |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
reviewed
Oct 2, 2025
Collaborator
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
LGTM I think you can completely inherit from Qwen2Model tho
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, cwm |
auto-merge was automatically disabled
October 9, 2025 15:30
Pull Request is not mergeable
AhnJoonSung
pushed a commit
to AhnJoonSung/transformers
that referenced
this pull request
Oct 12, 2025
* [wip][cwm] Code World Model stubs and setup in HF Transformers * [wip] Get other things working * [wip] Working * Tokenizer pad * fix: cwm window attn * temp remove test * temp remove test * Fixes * Temporarily add auto config remapping option until VLLM 0.11 is out * Fix model type and add layer validation * Lint, remove CwmForSequenceClassification * Lint, tests * Remove CwmForSequenceClassification * Lint * Remove intermediary layer expors/doc errorss, fix tests * Lint * run python utils/sort_auto_mappings.py --check_only * Remove Cwm processor mapping, get check_repo passing * Remove CwmTextConfig from test * Add docstring for CwmConfig * remove global_window and window_pattern params from config * Fix docstrings * Revert change to auto docstring util * lint * Fixes minus test improvements * Alter tests to simply check logits * lint * Have slow tests use repo, make CwmPretrainedModel passthrough * Remove decoder layer implementation, use Llama3Decoder + CwmAttetion * Use linear w/o bias for CwmAttention, add token-level integration test * Don't ignore config attention bias * Remove attention bias parameter entirely from config --------- Co-authored-by: galco <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the Code World Model (CWM) - https://ai.meta.com/research/publications/cwm-an-open-weights-llm-for-research-on-code-generation-with-world-models/
High-level implementation details:
The model repos are:
Note that for VLLM compatibility, model
config.jsonstill refer toLlama3ForCausalLMand allamamodel_type— see example. vllm-project/vllm#25611 adds support mappingCwmForCausalLMto the Llama3 model class in VLLM since VLLM supportsLlama3+layer_typeswith local/global attention - see docs. The model type in theconfig.jsonwill be updated on HF (and the special automapping condition removed) once this PR is merged and a Transformers release has happened containing theCwmForCausalLMmodel class.@ArthurZucker, @zucchini-nlp
Supersedes #41188 due to some fork misery