Skip to content

[PD] Transfer EAGLE draft probabilities for rejection sampling - #40953

Draft
roborluo wants to merge 3 commits into
sgl-project:mainfrom
roborluo:pd-eagle-rejection-sampling
Draft

roborluo wants to merge 3 commits into
sgl-project:mainfrom
roborluo:pd-eagle-rejection-sampling

Conversation

@roborluo

@roborluo roborluo commented Sep 23, 2026 •

Copy link
Copy Markdown

Motivation

The disaggregated EAGLE handoff needs the initial full draft proposal distribution when rejection sampling is enabled. Top-k candidates alone do not provide the proposal probabilities needed by the verifier.

Modifications

  • Transfer the FP32 draft distribution through prefill/decode metadata and reconstruct it in the decode draft input.
  • Validate the peers' rejection-sampling settings before transfer setup. Mixed/unknown prefill rank settings remain unknown rather than adopting a later rank's layout. Upgrade both peers together.
  • Restrict this handoff to single-layer EAGLE-family speculation; retain the disabled metadata layout.
  • Initialize fake-transfer warmup proposals consistently, without mutating reusable metadata slots.
  • Report metadata allocation size and document per-rank memory cost.
  • Preserve current upstream sampling-mask, DSA-seed, and optional KV-checksum fields.

Accuracy Tests

  • Passed: 7 source-isolated CPU tensor tests (plus 7 subtests), exercising proposal roundtrip, disabled layout, DSA/checksum ordering, missing/invalid proposals, and decode reconstruction.
  • Passed: changed Python files compile; focused Ruff checks; git diff --check.
  • Added bootstrap compatibility and Rust roundtrip/mixed-registration regressions.
  • Full SGLang unit-test collection is blocked on the local macOS environment by missing Triton. Rust tests were not run because the Rust toolchain is unavailable.
  • GPU PD transport, overlap/CUDA graphs, stochastic sampling quality and model accuracy have not been validated on this upstream port. This PR is Draft.

Reproducible CPU check with PyTorch installed:

python -m pytest test/registered/unit/disaggregation/test_pd_draft_metadata_cpu.py

In a complete SGLang development environment, also run:

python -m pytest test/registered/unit/disaggregation/test_disaggregation_wire.py test/registered/unit/disaggregation/test_register_to_bootstrap.py

Speed Tests and Profiling

No upstream speed results are claimed. Full-vocabulary FP32 metadata adds capacity-dependent memory and transfer overhead. No benchmark datasets, configs, results, or server logs are included.

Checklist

  • Apply the repository's Python import/lint/format tools to changed Python files.
  • Add focused regression tests and documentation.
  • Run full unit and Rust tests in a supported development environment.
  • Validate GPU transport, accuracy, and performance before merge.

CI States

Latest PR Test (Base): ❌ Run #35906832513
Latest PR Test (Extra): ❌ Run #35906831754
Latest PR Test (AMD ROCm 10): ❌ Run #35906832322

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant