[bug] Fix lora reloading from disk every step when using AsyncLLM by hao-aaron · Pull Request #41482 · vllm-project/vllm

hao-aaron · 2026-05-01T21:17:36Z

Purpose

When calling load_lora_adapter endpoint with load_inplace:true, lora reloads from disk every generation, leading to massive slowdowns. we set lora_request.load_inplace = False after first load to so we only load it once.

Reproduction:

VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 vllm serve meta-llama/Llama-3.2-3B-Instruct --enable-lora --max-loras 1 --max-lora-rank 64 --gpu-memory-utilization 0.5 --port 8765

# wait for vllm server startup, then in separate terminal
curl -s localhost:8765/v1/load_lora_adapter -H 'Content-Type: application/json' -d '{"lora_name":"l","lora_path":"jeeejeee/llama32-3b-text2sql-spider","load_inplace":true}'
curl -s localhost:8765/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"l","messages":[{"role":"user","content":"hi"}],"max_tokens":256}'

We observe that without the fix, lora reloads from disk on every request.

(APIServer pid=4022344) INFO:     127.0.0.1:38100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 73584.28it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
(APIServer pid=4022344) INFO 05-01 21:07:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 59.3%
Fetching 3 files: 100%|██████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 8612.53it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 20593.96it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 85598.04it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 10941.66it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 17403.75it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 26051.58it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 15887.52it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
Fetching 3 files: 100%|█████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11661.64it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                   | 0/3 [00:00<?, ?it/s]
(APIServer pid=4022344) INFO:     127.0.0.1:44558 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: hao-aaron <ahao@anyscale.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

gemini-code-assist

Code Review

This pull request updates the load_lora_adapter method in vllm/entrypoints/openai/models/serving.py to reset the load_inplace attribute of the lora_request object to False before caching it. This change prevents the flag from leaking into subsequent generation requests. I have no feedback to provide as there were no review comments to evaluate.

sfeng33 · 2026-05-05T15:26:07Z

PTAL @jeejeelee

Bug fixes found in #1616 While testing out the tinker CI, found a port collision bug. async def _run_server(self) -> None: """Internal method to run the HTTP server.""" # Release the port reservation right before vLLM rebinds. if self._port_reservation is not None: self._port_reservation.close() # ← step A self._port_reservation = None ... sock_addr = (self._cli_args.host, self._cli_args.port) sock = create_server_socket(sock_addr) There exists a window of time where we free port so vllm can take it, but other actors can also. Fixed this by allocating a specific, non-overlapping port range for each actor. Additionally, changed the disk reloading workaround #1606 created here to more closely mirror the upstream change vllm-project/vllm#41482, adding a new custom endpoint with the load_inplace override. --------- Signed-off-by: hao-aaron <ahao@anyscale.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

This PR introduces a tinker ci, closing #1553. It uses tinker_cookbook.recipes.math_rl.train. Currently the `REWARD_MIN_VALUE` is set to 0, but will be tightened once we have a couple of runs. Added both colocated and fully async tests. https://wandb.ai/sky-posttraining-uc-berkeley/gsm8k_tinker_ci?nw=nwuserahao https://wandb.ai/sky-posttraining-uc-berkeley/gsm8k_tinker_fully_async_ci?nw=nwuserahao While testing out the tinker CI, found a port collision bug. ``` async def _run_server(self) -> None: """Internal method to run the HTTP server.""" # Release the port reservation right before vLLM rebinds. if self._port_reservation is not None: self._port_reservation.close() # ← step A self._port_reservation = None ... sock_addr = (self._cli_args.host, self._cli_args.port) sock = create_server_socket(sock_addr) ``` There exists a window of time where we free port so vllm can take it, but other actors can also. Fixed this by allocating a specific, non-overlapping port range for each actor. Additionally, changed the disk reloading workaround #1606 created here to more closely mirror the upstream change vllm-project/vllm#41482, adding a new custom endpoint with the load_inplace override. --------- Signed-off-by: hao-aaron <ahao@anyscale.com> Signed-off-by: ahao-anyscale <ahao@anyscale.com>

github-actions · 2026-08-10T02:14:44Z

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

FurtherAI · 2026-08-14T03:27:59Z

I believe this is still an issue.
5.6 sol suggests

self.lora_requests[lora_name] = msgspec.structs.replace(
    lora_request,
    load_inplace=False,
)

as a fix for the openai path, but e.g.

llm.chat(..., lora_request=LoRARequest(..., load_inplace=True))

is not covered by that.
So maybe functionality like this:

await engine_client.add_lora(
    lora_request,
    load_inplace=True,
)

Otherwise maybe cover all the paths that use lora requests and make sure they don't persist load_inplace.

Why was it attached to load_inplace in the first place?

The original implementation actually used add_adapter(lora_request, force_load=True), essentially our proposal. Original commit (4f23aa5)
- A maintainer then requested putting the flag on LoRARequest so replacement could be initiated through offline LLM.chat(...), since LLM had no public add_lora method. Maintainer comment ([Feat] allow inplace loading lora #31326 (comment))
- The resulting tests checked that outputs changed from adapter A to B, but never counted disk loads across decode steps. Merged PR [Feat] allow inplace loading lora #31326 ([Feat] allow inplace loading lora #31326)

Didn't fact check everything.

mergify · 2026-10-22T02:56:10Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hao-aaron.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: aoshen02 <aoshen@inferact.ai>

aoshen02 · 2026-10-22T08:32:59Z

/ci run

github-actions · 2026-10-22T08:33:20Z

✅ Triggered Buildkite CI #96607 for commit 5f61df3377e0.

…lm-project#41482) Signed-off-by: hao-aaron <ahao@anyscale.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: aoshen02 <aoshen@inferact.ai>

…itself vllm-project/vllm#41482 resets load_inplace before caching the adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

x

c090f79

Signed-off-by: hao-aaron <ahao@anyscale.com>

hao-aaron requested review from DarkLight1337, aarnphm, chaunceyjiang and russellb as code owners May 1, 2026 21:17

claude Bot reviewed May 1, 2026

View reviewed changes

mergify Bot added frontend bug Something isn't working labels May 1, 2026

gemini-code-assist Bot reviewed May 1, 2026

View reviewed changes

DarkLight1337 requested a review from jeejeelee May 2, 2026 02:24

hao-aaron mentioned this pull request May 4, 2026

[CI] Add E2E CI for Tinker API with SkyRLTrainBackend NovaSky-AI/SkyRL#1616

Merged

hao-aaron mentioned this pull request May 6, 2026

[bug] Lora bug fixes + port collision fix NovaSky-AI/SkyRL#1625

Merged

SoluMilken mentioned this pull request May 10, 2026

[Bug]: Runtime LoRA same-name reload can reuse stale prefix-cache blocks from previous adapter version #42125

Closed

1 task done

Merge branch 'main' into lora-disk-bug

986cb5d

sladyn98 mentioned this pull request May 13, 2026

Fix LoRA prefix cache identity on reload #42495

Closed

5 tasks done

j316chuck mentioned this pull request Jul 5, 2026

Rotate lora_int_id on reload, bounded to {1,2} (stale sampler + VRAM leak) j316chuck/SkyRL#9

Open

github-actions Bot added the stale Over 90 days of inactivity label Aug 10, 2026

github-actions Bot added unstale Recieved activity after being labelled stale and removed stale Over 90 days of inactivity labels Aug 15, 2026

AmineDiro mentioned this pull request Sep 2, 2026

AsyncGRPO: PEFT/LoRA support with adapter-only vLLM sync huggingface/trl#7017

Merged

danielhanchen mentioned this pull request Oct 12, 2026

[Bug] When loading LoRA via load_lora_adapter, the inference becomes very slow with high CPU usage unslothai/unsloth#3981

Closed

aoshen02 approved these changes Oct 22, 2026

View reviewed changes

mergify Bot added the needs-rebase label Oct 22, 2026

Merge branch 'main' into lora-disk-bug

603f026

Signed-off-by: aoshen02 <aoshen@inferact.ai>

mergify Bot removed the needs-rebase label Oct 22, 2026

Merge branch 'main' into lora-disk-bug

5f61df3

aoshen02 enabled auto-merge (squash) October 22, 2026 08:33

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 22, 2026

aoshen02 merged commit ef5bcaa into vllm-project:main Oct 22, 2026
68 checks passed

richard-for-bugen Bot mentioned this pull request Oct 22, 2026

[Rust Frontend] Keep load_inplace off cached LoRA requests like Python BugenZhao/vllm#35

Closed

BugenZhao mentioned this pull request Oct 23, 2026

[Rust Frontend] Keep load_inplace off cached LoRA requests like Python #63316

Merged

6 tasks done

faresobeid mentioned this pull request Nov 2, 2026

chore: bump vLLM to 0.33.0 (torch 2.14.0) PrimeIntellect-ai/prime-rl#4365

Open

Navigation Menu

Repository navigation

[bug] Fix lora reloading from disk every step when using AsyncLLM - #41482

[bug] Fix lora reloading from disk every step when using AsyncLLM#41482
aoshen02 merged 4 commits into
vllm-project:mainfrom
hao-aaron:lora-disk-bug

hao-aaron commented May 1, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

sfeng33 commented May 5, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

FurtherAI commented Aug 14, 2026

Uh oh!

mergify Bot commented Oct 22, 2026

Uh oh!

aoshen02 commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

hao-aaron commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

sfeng33 commented May 5, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

FurtherAI commented Aug 14, 2026

Uh oh!

mergify Bot commented Oct 22, 2026

Uh oh!

aoshen02 commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hao-aaron commented May 1, 2026 •

edited

Loading