Skip to content

Run async scheduling primers as standalone controller steps - #23

Closed
lmcafee-nvidia wants to merge 1 commit into
defer-req-res-overlapfrom
defer-req-res-single-forward
Closed

Run async scheduling primers as standalone controller steps#23
lmcafee-nvidia wants to merge 1 commit into
defer-req-res-overlapfrom
defer-req-res-single-forward

Conversation

@lmcafee-nvidia

@lmcafee-nvidia lmcafee-nvidia commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • Return an explicit controller-step result that distinguishes primer-only calls from sampled outputs.
  • End an async-scheduling controller call immediately after launching its forward primer.
  • Re-enter the controller inside the engine and synchronous wrapper without rescheduling requests or crossing the engine step boundary.

Architecture

Each controller invocation now launches at most one model forward. A missing logits state produces a primer-only result; the caller immediately invokes the controller again to consume those logits and run the normal prepare, sample, forward, and resolve flow. Request scheduling, engine bookkeeping, counters, and externally visible output remain attached to the completed sampled step rather than the internal primer transition.

Performance

DFW benchmark.py on the dense 3.58B model, using one real 3,675-token prompt, OSL 512, batch size 1, and full-iteration CUDA graphs. The table reports three independent recorded trials per branch; each trial followed one warmup run.

Branch Measured throughput (tok/s) Mean (tok/s)
Overlap base 104.0, 106.1, 104.7 104.9
Single-forward 105.8, 104.1, 101.6 103.8

The mean delta is -1.1%. The tested single-forward commit predates the preserving rebase but has the same stable patch ID as the current PR commit.

Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
@lmcafee-nvidia
lmcafee-nvidia force-pushed the defer-req-res-single-forward branch from b91846e to 8e8e278 Compare July 14, 2026 21:23
@lmcafee-nvidia

Copy link
Copy Markdown
Owner Author

Closing this archived implementation in favor of the active cumulative stack. Its MTP, EP, Mamba admission, primer, prefill, and step-order architecture is consolidated in #26 and #27.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant