Skip to content

[Core] get_kv_cache_from_block within core ONLY#7052

Closed
KrishnaM251 wants to merge 2 commits intovllm-project:mainfrom
KrishnaM251:kv-trans-3-bmv2-only
Closed

[Core] get_kv_cache_from_block within core ONLY#7052
KrishnaM251 wants to merge 2 commits intovllm-project:mainfrom
KrishnaM251:kv-trans-3-bmv2-only

Conversation

@KrishnaM251
Copy link
Copy Markdown
Contributor

@KrishnaM251 KrishnaM251 commented Aug 1, 2024

See #6808 for full details

[v0.3.1]

  • implemented get_kv_from_block at the block_manager_v2.py level
  • get_kv_from_block simply calls lower level methods from within core to obtain the kv_cache without going to the worker
    • used the prepare_worker_input from worker.py as inspiration

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 1, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

  • Comment /ready on the PR
  • Add ready label to the PR
  • Enable auto-merge.

🚀

@KrishnaM251 KrishnaM251 marked this pull request as draft August 1, 2024 23:35
@hmellor hmellor closed this Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants