[Hotfix] final fixes for P2P Transfer - #22663
Merged
yueming-yuan merged 5 commits intoApr 13, 2026
Merged
Conversation
The stacked_params_mapping routes q_a_proj and kv_a_proj_with_mqa to ReplicatedLinear.weight_loader with a shard_id, but ReplicatedLinear does not support shard_id. Skip the stacked path for this param_name so weights fall through to the existing cached_a_proj path in do_load_weights(), which correctly caches both halves and torch.cats them before loading. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change lazy import from `sglang.srt.utils` to `sglang.srt.utils.network` to match the module where `get_local_ip_auto` is actually defined. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Glm4MoeGate._weight_fp32 is a FP32 cache of the bf16 gate weight. Runtime invalidation of this cache after weight update is not yet supported. Skip it in both _reset_tensors and _postprocess_tensors, same pattern as cos_sin_cache and inv_freq. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Already imported at module level (line 183). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
JD-ETH
requested review from
Fridge003,
Ying1123,
ch-wan,
fzyzcjy,
hnyls2002,
ispobock and
merrymercy
as code owners
April 13, 2026 05:15
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
This was referenced Apr 13, 2026
JensenFire
reviewed
Apr 13, 2026
| def _reset_tensors(self): | ||
| for name, param in self._model_state(): | ||
| if "cos_sin_cache" in name or "freqs_cis" in name: | ||
| if "cos_sin_cache" in name or "freqs_cis" in name or "_weight_fp32" in name: |
Contributor
There was a problem hiding this comment.
nit: maybe we could maintain a list where these keys could be skipped.
yueming-yuan
approved these changes
Apr 13, 2026
This was referenced May 6, 2026
fzyzcjy
pushed a commit
to fzyzcjy/sglang
that referenced
this pull request
May 6, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Letian Ruan <ruanlt420@gmail.com> Cherry-picked weight_checker.py changes only from the original PR. (cherry picked from commit 2b93c2f)
fzyzcjy
added a commit
that referenced
this pull request
May 6, 2026
…4534) Co-authored-by: JD <jaedon.guo@gmail.com>
LLThomas
pushed a commit
to LLThomas/sglang
that referenced
this pull request
May 8, 2026
…t#22663 (sgl-project#24534) Co-authored-by: JD <jaedon.guo@gmail.com>
yueming-yuan
pushed a commit
that referenced
this pull request
May 21, 2026
yueming-yuan
pushed a commit
that referenced
this pull request
Jun 17, 2026
nanjiangwill
pushed a commit
to nanjiangwill/sglang
that referenced
this pull request
Jul 7, 2026
…t#21278, sgl-project#22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Co-authored-by: JensenFire <elevenji@outlook.com>
nanjiangwill
pushed a commit
to nanjiangwill/sglang
that referenced
this pull request
Jul 7, 2026
…sgl-project#18565, sgl-project#22663, sgl-project#28001, sgl-project#29675) Combined weight-processing machinery, folding the general post-process plumbing from the INT4 QAT work (sgl-project#18565), the P2P post_load_weights extension (sgl-project#22663), the weight-processing refactor (sgl-project#28001), and the pause-aware weight locking fix (sgl-project#29675) into its final form: - Begin/EndWeightUpdate request chain: engine/http_server/ tokenizer_control_mixin (pause-aware locking) -> scheduler -> weight_updater sessions with {target,draft,all} runner selectors - loader.py: public post_load_weights, restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers) - model_runner begin/end_weight_update + receive_weights_from_distributed - iter_runners() on tp_worker and all spec v2 workers (incl. port to DFlashWorkerV2; v1 dflash/frozen-kv workers no longer exist) - CompressedTensorsFusedMoEMethod.restore_weights_before_loading dispatch - weight_checker ComparableWeight refactor + weight_checker_comparator Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
yueming-yuan
pushed a commit
that referenced
this pull request
Jul 14, 2026
…22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Co-authored-by: JensenFire <elevenji@outlook.com> Merged with v0.5.15's fused-indexer-wk loading in the deepseek weight loader; kept self.stacked_params_mapping refactor alongside it.
yueming-yuan
added a commit
that referenced
this pull request
Jul 14, 2026
…sessions + spec-draft weight check (#18565, #22663, #28001, #29675, #27750) Combined weight-processing machinery, folding the general post-process plumbing from the INT4 QAT work (#18565), the P2P post_load_weights extension (#22663), the weight-processing refactor (#28001), and the pause-aware weight locking fix (#29675) into its final form, plus the extension of the weight checker to speculative draft worker(s) (#27750): - Begin/EndWeightUpdate request chain: engine/http_server/ tokenizer_control_mixin (pause-aware locking) -> scheduler -> weight_updater sessions with {target,draft,all} runner selectors - loader.py: public post_load_weights, restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers) - model_runner begin/end_weight_update + receive_weights_from_distributed - iter_runners() on tp_worker and all spec v2 workers (incl. DFlashWorkerV2) - CompressedTensorsFusedMoEMethod.restore_weights_before_loading dispatch - weight checker: selector + skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner Kept v0.5.15's upstreamed ComparableWeight weight_checker_comparator (incl. the #29623 ue8m0 scale-unpack fix); adapted the weight_checker caller to its public compare_weights/CHUNK_NUMEL API. Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com> Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
yueming-yuan
pushed a commit
that referenced
this pull request
Jul 14, 2026
…22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Co-authored-by: JensenFire <elevenji@outlook.com> Merged with v0.5.15's fused-indexer-wk loading in the deepseek weight loader; kept self.stacked_params_mapping refactor alongside it.
yueming-yuan
added a commit
that referenced
this pull request
Jul 14, 2026
…sessions + spec-draft weight check (#18565, #22663, #28001, #29675, #27750) Combined weight-processing machinery, folding the general post-process plumbing from the INT4 QAT work (#18565), the P2P post_load_weights extension (#22663), the weight-processing refactor (#28001), and the pause-aware weight locking fix (#29675) into its final form, plus the extension of the weight checker to speculative draft worker(s) (#27750): - Begin/EndWeightUpdate request chain: engine/http_server/ tokenizer_control_mixin (pause-aware locking) -> scheduler -> weight_updater sessions with {target,draft,all} runner selectors - loader.py: public post_load_weights, restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers) - model_runner begin/end_weight_update + receive_weights_from_distributed - iter_runners() on tp_worker and all spec v2 workers (incl. DFlashWorkerV2) - CompressedTensorsFusedMoEMethod.restore_weights_before_loading dispatch - weight checker: selector + skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner Kept v0.5.15's upstreamed ComparableWeight weight_checker_comparator (incl. the #29623 ue8m0 scale-unpack fix); adapted the weight_checker caller to its public compare_weights/CHUNK_NUMEL API. Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com> Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
yueming-yuan
added a commit
that referenced
this pull request
Jul 14, 2026
…sessions + spec-draft weight check (#18565, #22663, #28001, #29675, #27750) Combined weight-processing machinery, folding the general post-process plumbing from the INT4 QAT work (#18565), the P2P post_load_weights extension (#22663), the weight-processing refactor (#28001), and the pause-aware weight locking fix (#29675) into its final form, plus the extension of the weight checker to speculative draft worker(s) (#27750): - Begin/EndWeightUpdate request chain: engine/http_server/ tokenizer_control_mixin (pause-aware locking) -> scheduler -> weight_updater sessions with {target,draft,all} runner selectors - loader.py: public post_load_weights, restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers) - model_runner begin/end_weight_update + receive_weights_from_distributed - iter_runners() on tp_worker and all spec v2 workers (incl. DFlashWorkerV2) - CompressedTensorsFusedMoEMethod.restore_weights_before_loading dispatch - weight checker: selector + skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner Kept v0.5.15's upstreamed ComparableWeight weight_checker_comparator (incl. the #29623 ue8m0 scale-unpack fix); adapted the weight_checker caller to its public compare_weights/CHUNK_NUMEL API. Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com> Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
yueming-yuan
added a commit
that referenced
this pull request
Jul 14, 2026
…sessions + spec-draft weight check (#18565, #22663, #28001, #29675, #27750) Combined weight-processing machinery, folding the general post-process plumbing from the INT4 QAT work (#18565), the P2P post_load_weights extension (#22663), the weight-processing refactor (#28001), and the pause-aware weight locking fix (#29675) into its final form, plus the extension of the weight checker to speculative draft worker(s) (#27750): - Begin/EndWeightUpdate request chain: engine/http_server/ tokenizer_control_mixin (pause-aware locking) -> scheduler -> weight_updater sessions with {target,draft,all} runner selectors - loader.py: public post_load_weights, restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers) - model_runner begin/end_weight_update + receive_weights_from_distributed - iter_runners() on tp_worker and all spec v2 workers (incl. DFlashWorkerV2) - CompressedTensorsFusedMoEMethod.restore_weights_before_loading dispatch - weight checker: selector + skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner Kept v0.5.15's upstreamed ComparableWeight weight_checker_comparator (incl. the #29623 ue8m0 scale-unpack fix); adapted the weight_checker caller to its public compare_weights/CHUNK_NUMEL API. Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com> Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
yueming-yuan
pushed a commit
that referenced
this pull request
Jul 25, 2026
…22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Rebased onto v0.5.16, which extracted the remote-instance transfer engine out of ModelRunner into RemoteInstanceWeightTransporter: the per-rank RankParallelismConfig is now built in `init_engine()` and published from `maybe_register_and_publish_weight_info()`, instead of the ModelRunner methods this commit originally added. The hoisted deepseek expert_params_mapping also picks up v0.5.16's broader `is_wint4afp8_or_wint4a16_config` predicate rather than the old `quant_config.get_name() == "w4afp8"` check. Co-authored-by: JensenFire <elevenji@outlook.com>
yueming-yuan
added a commit
that referenced
this pull request
Jul 25, 2026
… for spec draft worker(s) (#27749, #28575, #18565, #22663, #28001, #29675, #27750) Squash of the spec-draft distributed-update work and the weight-processing session work: the latter rewrites iter_draft_runners()/get_model_runners() the former introduces, so they cannot be applied independently. - Draft runners never join the update group, so the target receives the broadcast once and the weights are loaded into every selected runner. - Begin/EndWeightUpdate session chain: engine / http_server / tokenizer_control_mixin (pause-aware locking) -> scheduler -> SchedulerWeightUpdaterManager, with {target,draft,all} runner selectors. - loader.py: public post_load_weights plus restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers). - weight checker: skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner. Reimplemented onto v0.5.16, which had already extracted weight updating into ModelRunner.weight_updater (WeightUpdater): the receive/load split lands there as receive_weights_from_distributed() + load_weights() rather than on ModelRunner, and the scheduler fan-out drives runner.weight_updater.*. The worker-level update_weights_from_{distributed,tensor} entry points on TpModelWorker / EAGLEWorkerV2 / NGRAMWorker are dropped: the scheduler now owns the fan-out, and leaving them would be a second path that updates one runner only. iter_runners() replaces the ad-hoc draft-runner discovery on every spec v2 worker (incl. DFlashWorkerV2). The unit test moves to test/registered/rl/, since v0.5.16 retired test/srt/ from CI collection. Co-authored-by: Yueming Yuan <yym022502@gmail.com> Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
yueming-yuan
pushed a commit
that referenced
this pull request
Jul 25, 2026
…22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Rebased onto v0.5.16, which extracted the remote-instance transfer engine out of ModelRunner into RemoteInstanceWeightTransporter: the per-rank RankParallelismConfig is now built in `init_engine()` and published from `maybe_register_and_publish_weight_info()`, instead of the ModelRunner methods this commit originally added. The hoisted deepseek expert_params_mapping also picks up v0.5.16's broader `is_wint4afp8_or_wint4a16_config` predicate rather than the old `quant_config.get_name() == "w4afp8"` check. Co-authored-by: JensenFire <elevenji@outlook.com>
yueming-yuan
added a commit
that referenced
this pull request
Jul 25, 2026
… for spec draft worker(s) (#27749, #28575, #18565, #22663, #28001, #29675, #27750) Squash of the spec-draft distributed-update work and the weight-processing session work: the latter rewrites iter_draft_runners()/get_model_runners() the former introduces, so they cannot be applied independently. - Draft runners never join the update group, so the target receives the broadcast once and the weights are loaded into every selected runner. - Begin/EndWeightUpdate session chain: engine / http_server / tokenizer_control_mixin (pause-aware locking) -> scheduler -> SchedulerWeightUpdaterManager, with {target,draft,all} runner selectors. - loader.py: public post_load_weights plus restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers). - weight checker: skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner. Reimplemented onto v0.5.16, which had already extracted weight updating into ModelRunner.weight_updater (WeightUpdater): the receive/load split lands there as receive_weights_from_distributed() + load_weights() rather than on ModelRunner, and the scheduler fan-out drives runner.weight_updater.*. The worker-level update_weights_from_{distributed,tensor} entry points on TpModelWorker / EAGLEWorkerV2 / NGRAMWorker are dropped: the scheduler now owns the fan-out, and leaving them would be a second path that updates one runner only. iter_runners() replaces the ad-hoc draft-runner discovery on every spec v2 worker (incl. DFlashWorkerV2). The unit test moves to test/registered/rl/, since v0.5.16 retired test/srt/ from CI collection. Co-authored-by: Yueming Yuan <yym022502@gmail.com> Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
yueming-yuan
pushed a commit
that referenced
this pull request
Jul 25, 2026
…22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Rebased onto v0.5.16, which extracted the remote-instance transfer engine out of ModelRunner into RemoteInstanceWeightTransporter: the per-rank RankParallelismConfig is now built in `init_engine()` and published from `maybe_register_and_publish_weight_info()`, instead of the ModelRunner methods this commit originally added. The hoisted deepseek expert_params_mapping also picks up v0.5.16's broader `is_wint4afp8_or_wint4a16_config` predicate rather than the old `quant_config.get_name() == "w4afp8"` check. Co-authored-by: JensenFire <elevenji@outlook.com>
yueming-yuan
added a commit
that referenced
this pull request
Jul 25, 2026
… for spec draft worker(s) (#27749, #28575, #18565, #22663, #28001, #29675, #27750) Squash of the spec-draft distributed-update work and the weight-processing session work: the latter rewrites iter_draft_runners()/get_model_runners() the former introduces, so they cannot be applied independently. - Draft runners never join the update group, so the target receives the broadcast once and the weights are loaded into every selected runner. - Begin/EndWeightUpdate session chain: engine / http_server / tokenizer_control_mixin (pause-aware locking) -> scheduler -> SchedulerWeightUpdaterManager, with {target,draft,all} runner selectors. - loader.py: public post_load_weights plus restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers). - weight checker: skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner. Reimplemented onto v0.5.16, which had already extracted weight updating into ModelRunner.weight_updater (WeightUpdater): the receive/load split lands there as receive_weights_from_distributed() + load_weights() rather than on ModelRunner, and the scheduler fan-out drives runner.weight_updater.*. The worker-level update_weights_from_{distributed,tensor} entry points on TpModelWorker / EAGLEWorkerV2 / NGRAMWorker are dropped: the scheduler now owns the fan-out, and leaving them would be a second path that updates one runner only. iter_runners() replaces the ad-hoc draft-runner discovery on every spec v2 worker (incl. DFlashWorkerV2). The unit test moves to test/registered/rl/, since v0.5.16 retired test/srt/ from CI collection. Co-authored-by: Yueming Yuan <yym022502@gmail.com> Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
yueming-yuan
pushed a commit
that referenced
this pull request
Jul 25, 2026
…22663) Includes making Cohere2MoeConfig a dataclass so parameter mapping can introspect config fields for P2P weight update. Rebased onto v0.5.16, which extracted the remote-instance transfer engine out of ModelRunner into RemoteInstanceWeightTransporter: the per-rank RankParallelismConfig is now built in `init_engine()` and published from `maybe_register_and_publish_weight_info()`, instead of the ModelRunner methods this commit originally added. The hoisted deepseek expert_params_mapping also picks up v0.5.16's broader `is_wint4afp8_or_wint4a16_config` predicate rather than the old `quant_config.get_name() == "w4afp8"` check. Co-authored-by: JensenFire <elevenji@outlook.com>
yueming-yuan
added a commit
that referenced
this pull request
Jul 25, 2026
… for spec draft worker(s) (#27749, #28575, #18565, #22663, #28001, #29675, #27750) Squash of the spec-draft distributed-update work and the weight-processing session work: the latter rewrites iter_draft_runners()/get_model_runners() the former introduces, so they cannot be applied independently. - Draft runners never join the update group, so the target receives the broadcast once and the weights are loaded into every selected runner. - Begin/EndWeightUpdate session chain: engine / http_server / tokenizer_control_mixin (pause-aware locking) -> scheduler -> SchedulerWeightUpdaterManager, with {target,draft,all} runner selectors. - loader.py: public post_load_weights plus restore_weight/postprocess_weight via _apply_quant_method_hook (skips LoRA wrappers). - weight checker: skip_tensor_list + role-prefixed overall_checksum over get_model_runners(selector), replacing _get_draft_model_runner. Reimplemented onto v0.5.16, which had already extracted weight updating into ModelRunner.weight_updater (WeightUpdater): the receive/load split lands there as receive_weights_from_distributed() + load_weights() rather than on ModelRunner, and the scheduler fan-out drives runner.weight_updater.*. The worker-level update_weights_from_{distributed,tensor} entry points on TpModelWorker / EAGLEWorkerV2 / NGRAMWorker are dropped: the scheduler now owns the fan-out, and leaving them would be a second path that updates one runner only. iter_runners() replaces the ad-hoc draft-runner discovery on every spec v2 worker (incl. DFlashWorkerV2). The unit test moves to test/registered/rl/, since v0.5.16 retired test/srt/ from CI collection. Co-authored-by: Yueming Yuan <yym022502@gmail.com> Co-authored-by: JD-ETH <jaedon.guo@gmail.com> Co-authored-by: maocheng23 <35615230+maocheng23@users.noreply.github.com>
Chronostasys
pushed a commit
to MindLab-Research/sglang
that referenced
this pull request
Aug 24, 2026
…t#22663 (sgl-project#24534) Co-authored-by: JD <jaedon.guo@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
- Fix Qwen3 rope_parameters → use get_rope_config() helper instead of accessing config.rope_parameters dict directly
- (model_runner.py conflict resolved — redundant import removed)
- _reset_tensors(): skip _weight_fp32 buffers (don't randomize them)
- _postprocess_tensors(): add _weight_fp32 to non_persistent_buffer_patterns (don't fail on mismatch)
- Reason: Glm4MoeGate._weight_fp32 is a FP32 cache of the bf16 gate weight. Runtime invalidation after P2P weight update is not supported yet. Same skip pattern as cos_sin_cache / inv_freq.
Validated models (all ✅ with --check-weight-update-equal + p2p)