Skip to content

[KV Connector][Don't merge] PoC for the async connector lookup functionality - #23622

Draft
ApostaC wants to merge 1 commit into
vllm-project:mainfrom
ApostaC:local-dev/poc-async-lookup
Draft

ApostaC wants to merge 1 commit into
vllm-project:mainfrom
ApostaC:local-dev/poc-async-lookup

Conversation

@ApostaC

@ApostaC ApostaC commented Aug 26, 2025 •

Copy link
Copy Markdown
Contributor

Purpose

This PR serves as a PoC of how we should use the new semantics introduced in #23620

With this "dumb PoC", we can already see the decrease in the overhead in the scheduler.

Test result

Timing the Scheduler.schedule() function:

Before this PR:

[scheduler.py:54] Average Scheduler.schedule time over last 60 runs: 2.3171 ms

After this PR:

[scheduler.py:54] Average Scheduler.schedule time over last 60 runs: 1.9629 ms

Nsys profile screenshot

Before this PR: 8.38 ms when scheduling the prefill requests

image

After this PR: 2.97 ms when scheduling the prefill requests

image
Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: ApostaC <yihua98@uchicago.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant