Repository navigation
[JJ-0926] 세션 기반 QuotaServe 및 LFU·FIFO KV 캐시 회수 정책 구현 - #2
Merged
Merged
Conversation
…ponses (vllm-project#40314) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
vllm-project#40245) Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>
…startup errors. (vllm-project#39977) Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
…project#39916) Signed-off-by: Julien Denize <julien.denize@mistral.ai> Signed-off-by: juliendenize <julien.denize@mistral.ai>
…atch (vllm-project#39478) Signed-off-by: velonica0 <like@mail.nankai.edu.cn>
…project#40324) Signed-off-by: San-Nguyen <san.nguyen@ibm.com>
…roject#40251) Signed-off-by: wuyingjun <wuyingjun_yewu@cmss.chinamobile.com>
Signed-off-by: milesial <milesial@users.noreply.github.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
…oject#38093) Signed-off-by: nemanjaudovic <nudovic@amd.com>
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Signed-off-by: ilmarkov <markovilya197@gmail.com> Signed-off-by: Markov Ilya <markovilya19@gmail.com>
… TransferTopology (vllm-project#39529) Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
…ding (vllm-project#39627) Signed-off-by: Yan Ma <yan.ma@intel.com>
…m-project#40125) Signed-off-by: Aleksandar Yanakiev <alexander.yanakiev@discretestack.com> Co-authored-by: Aleksandar Yanakiev <alexander.yanakiev@discretestack.com>
Signed-off-by: kse <kevin.sejourne@cloud-temple.com> Co-authored-by: kse <kevin.sejourne@cloud-temple.com>
Signed-off-by: Vasiliy Kuznetsov <vasiliy@meta.com>
…ject#40168) Signed-off-by: Sage Moore <sage@neuralmagic.com>
…3 speculative decoding on ROCm (vllm-project#39616) Signed-off-by: larryli2-amd <larryli2@amd.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
…vllm-project#39242) Signed-off-by: Rita Brugarolas Brufau <rita.brugarolasbrufau@amd.com>
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
…#40109) Signed-off-by: Yan Ma <yan.ma@intel.com>
…Base (vllm-project#35949) Signed-off-by: Bill Nell <bnell@redhat.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Signed-off-by: Sage Moore <sage@neuralmagic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
…ject#39329) Signed-off-by: Frederik Gossen <frgossen@meta.com>
Signed-off-by: bai <v@gor.io>
…ect#39733) Signed-off-by: Frederik Gossen <frgossen@meta.com>
…llm-project#33728) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Replace the slow/flaky add-apt-repository deadsnakes PPA with building Python from source. This avoids the long wait times and retry loops when Launchpad PPA servers are slow or unresponsive. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: khluu <khluu000@gmail.com>
…lm-project#41444) Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> (cherry picked from commit edd60ac)
This reverts commit 05ebca5.
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
…vllm-project#41526) Co-authored-by: Copilot <copilot@github.com>
Signed-off-by: zixi-qi <zixi@inferact.ai>
Assisted-by: OpenAI Codex
Assisted-by: OpenAI Codex
Assisted-by: Codex
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
Signed-off-by: zero-uuuuk <bike481227@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
주요 변경
vLLM V1에 세션별 KV 사용량과 후속 요청 대기 시간을 반영하는 QuotaServe를 구현하였다.
EVICTION_POLICY설정으로 Global LRU·LFU·FIFO·QuotaServe를 선택할 수 있다.QuotaServe 구현
application_id와session_id를 이용해 앱과 세션을 구분Quota는 회수 후보를 결정하는 기준이며, 고정 메모리 예약이나 블록 할당 상한으로 사용하지 않는다.
정책 비교 및 지표
GPU 실험 결과
별도 연구 저장소에서 A10G 24GB와 Llama-3.2-3B-Instruct로 180개 세션, 총 1,800개 요청을 비교하였다. 기록된 QuotaServe 결과는 Global LRU 대비 다음과 같다.
실행 조건과 결과 선택 기준, 원자료는 연구 저장소의
004_serving_experiments에 기록하였다.테스트 및 문서
docs/design/quota_serve.md에 문서화