Skip to content

Support async scheduling for prefill transitions - #24

Closed
lmcafee-nvidia wants to merge 1 commit into
defer-req-res-single-forwardfrom
defer-req-res-async-prefill
Closed

Support async scheduling for prefill transitions#24
lmcafee-nvidia wants to merge 1 commit into
defer-req-res-single-forwardfrom
defer-req-res-async-prefill

Conversation

@lmcafee-nvidia

Copy link
Copy Markdown
Owner

Summary

  • Run non-chunked prefill and mixed-batch transitions through the async scheduling path.
  • Drain a pending async forward before admitting newly ready work, preserving one model forward per controller step.
  • Transition surviving prefill requests into decode during resolution while preserving legacy request-row movement.
  • Keep chunked prefill outside the async scheduling path.

Architecture

The engine treats admission draining as a generic async scheduling transition. When new work is ready while an async forward is pending, the controller consumes and resolves that pending result without launching a successor forward. The next engine iteration can then admit the waiting request without mixing two forwards into one controller step.

For non-chunked prefill, the controller resolves the completed prompt forward, prepares the surviving requests for decode, and launches the next forward through the same async scheduling machinery used by steady-state decode. Request compaction follows the legacy row-movement order so request-local sampling state remains attached to the same logical requests.

Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
@lmcafee-nvidia

Copy link
Copy Markdown
Owner Author

Closing this archived implementation in favor of the active cumulative stack. Its MTP, EP, Mamba admission, primer, prefill, and step-order architecture is consolidated in #26 and #27.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant