chore: CI proxy for #2819 - #2820
Closed
HuiyingLi wants to merge 3 commits into
Closed
Conversation
Domino (sgl-project/SpecForge#571) extends the parallel DFlash draft backbone with a lightweight causal correction head: a GRU encodes a causal state from each block's previous tokens, and a low-rank projection of [backbone hidden | GRU state] adds a correction to the parallel base logits. Training jointly supervises the Domino-refined and backbone-only logits with a base-anchor curriculum (loss = (1-lambda_base)*final_loss + lambda_base*base_loss, lambda_base decaying to 0). The DFlash path is unchanged: the draft model gains an optional Domino head (projector_type='domino'), a new DominoTrainerModule reuses DFlash anchor sampling and the block mask, and TrainDominoRecipe subclasses the DFlash recipe via four behavior-preserving seams. Adds an example YAML and unit tests (domino core + recipe seams). Signed-off-by: khazic <khazzz1c@gmail.com>
model.to(bfloat16) rounded the rotary inv_freq buffer to bf16, but HF from_pretrained and the SGLang serving runtime keep it in fp32. That train/inference RoPE mismatch grows with absolute position and erodes draft acceptance. Recompute fresh fp32 frequencies from the rotary config after any low-precision cast (a bf16 round-trip cannot be undone by upcasting), covering the training build and reload paths. Signed-off-by: khazic <khazzz1c@gmail.com>
…nt mask clone /simplify cleanups (behavior unchanged): _compute_weighted_losses returns only the three losses instead of also re-returning flattened views (the caller recomputes them as views where needed), and the pre-decay eval mask references weight_mask directly since the decay step rebinds it out-of-place, avoiding a per-step clone. Signed-off-by: khazic <khazzz1c@gmail.com>
Contributor
Author
|
/ok to test 0c29737 |
Contributor
Author
|
Closing CI-only proxy because source PR #2819 is merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI-only proxy PR. Do not merge / do not review. Points at the exact head commit of #2819 (0c29737) so internal CI runs under the internal-contributor queue; results post back to #2819 via the shared SHA. Source of truth: #2819. Close once CI completes.