Repository navigation
Add GLM-5.3-Flash Miles FFT recipe - #645
Draft
zhouhelena1 wants to merge 3 commits into
Draft
zhouhelena1 wants to merge 3 commits into
zhouhelena1 wants to merge 3 commits into
Conversation
|
Documentation preview: https://modal-labs-training-gym--training-gym-previews-preview-r-78da35.modal.run/645/docs |
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a GLM-5.3-Flash full-parameter GRPO recipe for Miles, isolated from the bridge LoRA onboarding in #639. This follows upstream Miles (radixark/miles#2786), using its 32-GPU full-model configuration:
Recipe
zai-org/GLM-5.3-Flash, full-parameter training through Megatron and raw HF-to-torch_distconversion.TrainConfig; upstream uses DAPO math.1e-6, constant learning rate, weight decay 0.1, betas 0.9/0.98, clipping 0.2/0.28, and zero KL/entropy coefficients. Microbatch 1, static batching, full activation recomputation, and an 8,192-token per-GPU training budget.Differences from upstream Miles
glm53nextimage by digest and Miles tocc76e23915b2132ecfff65b97331fd83b02637e6.Validation
No completed FFT training validation run yet. This PR is a draft.
The attempted 10-step FFT run (
chewy-bag) failed while writing the converted checkpoint, before any training steps started (inline_container.cc: unexpected pos). End-to-end training correctness, stability, and performance remain unvalidated. Successful LoRA runs do not validate this FFT recipe.Local checks previously passed: 128 existing registry/Miles tests, recipe import and topology checks, compilation, lint, and models-table freshness. No new tests or model Markdown files are included.