Repository navigation
[Bugfix][LoRA] Fix KeyError crash when a LoRA name is reloaded with requests in flight - #64616
Conversation
…equests in flight LoRARequest equality and hash used only lora_name, so the model runners' active-adapter sets merged requests that share a name but not a lora_int_id. Unloading and loading an adapter under the same name while requests for the old id are running produces exactly that batch; the merged-away id was never activated and convert_mapping raised KeyError, killing the engine core. Compare lora_name and lora_int_id instead. Fixes vllm-project#64570 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: heyemlee <heyemlee@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
GPU run of this PR on an A40 (SM86), covering MRV2 and the server workload from #64570. Image 1. Same name, two ids, one main, main, this PR, the same output with 2. The #64570 workload. 64 adapters main (MRV2) died about 10 s in. 68 is the id this PR, MRV2 for 600 s, then None of the three server logs has a traceback. The 404s are requests that reached a name between its unload and its load. 3. Tests (
I also grepped main for code that relies on repro_same_name.py# Offline repro for vllm-project/vllm#64570: two LoRA adapters that share a
# name but not an id, in one batch. Runner is chosen by VLLM_USE_V2_MODEL_RUNNER.
import os
import traceback
from huggingface_hub import snapshot_download
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
meow = snapshot_download("Jackmin108/Qwen3-0.6B-Meow-LoRA")
woof = snapshot_download("Jackmin108/Qwen3-0.6B-Woof-LoRA")
print(f"VLLM_USE_V2_MODEL_RUNNER={os.environ.get('VLLM_USE_V2_MODEL_RUNNER')}")
llm = LLM(
model="Qwen/Qwen3-0.6B",
enable_lora=True,
max_loras=2,
max_lora_rank=8,
max_model_len=512,
gpu_memory_utilization=0.5,
)
messages = [
{"role": "system", "content": "Follow the instructions to make animal noises"},
{"role": "user", "content": "Make your favorite animal noise."},
]
sp = SamplingParams(temperature=0, max_tokens=10)
def run(label, lora_requests):
print(f"--- {label}: {[(r.lora_name, r.lora_int_id) for r in lora_requests]}")
try:
outs = llm.chat([messages] * 2, sp, lora_request=lora_requests)
except Exception as e:
print(f"EXCEPTION {type(e).__name__}: {e}")
traceback.print_exc()
return False
for r, o in zip(lora_requests, outs):
weights = "meow" if r.lora_path == meow else "woof"
print(f" name={r.lora_name} id={r.lora_int_id} weights={weights}: "
f"{o.outputs[0].text!r}")
return True
ok = run("distinct names", [LoRARequest("meow", 1, meow), LoRARequest("woof", 2, woof)])
ok = ok and run(
"same name, two ids", [LoRARequest("animal", 3, meow), LoRARequest("animal", 4, woof)]
)
print("RESULT", "OK" if ok else "FAILED")stress_reload.py# Server stress repro for vllm-project/vllm#64570, following the issue:
# 64 adapters (t0..t63) registered at runtime (more than --max-cpu-loras),
# 16 concurrent clients sending /v1/completions (32 output tokens) to
# Zipf-chosen adapters, and about once a second one adapter re-registered
# under the same name (unload + load) without waiting for its requests.
# --mode mixed alternates unload + load with in-place loads (load_inplace).
import argparse
import asyncio
import json
import random
import time
import aiohttp
p = argparse.ArgumentParser()
p.add_argument("--base-url", default="http://127.0.0.1:8000")
p.add_argument("--adapters-dir", default="/root/adapters")
p.add_argument("--num-adapters", type=int, default=64)
p.add_argument("--concurrency", type=int, default=16)
p.add_argument("--duration", type=float, default=300)
p.add_argument("--reload-interval", type=float, default=1.0)
p.add_argument("--mode", choices=["unload_load", "mixed"], default="unload_load")
p.add_argument("--zipf", type=float, default=1.1)
p.add_argument("--seed", type=int, default=0)
args = p.parse_args()
rng = random.Random(args.seed)
names = [f"t{i}" for i in range(args.num_adapters)]
weights = [1 / (k + 1) ** args.zipf for k in range(args.num_adapters)]
stats = {"ok": 0, "404": 0, "other_4xx": 0, "5xx": 0, "conn_err": 0}
reloads = {"unload_load": 0, "inplace": 0, "failed": 0}
first_5xx = None
first_5xx_body = None
dead_at = None
t0 = time.monotonic()
def now():
return round(time.monotonic() - t0, 1)
def pick():
return rng.choices(names, weights)[0]
async def post(session, path, body):
async with session.post(args.base_url + path, json=body) as r:
return r.status, await r.text()
async def load(session, name, inplace=False):
body = {"lora_name": name, "lora_path": f"{args.adapters_dir}/{name}"}
if inplace:
body["load_inplace"] = True
return await post(session, "/v1/load_lora_adapter", body)
async def client(session, stop):
global first_5xx, first_5xx_body
while not stop.is_set():
body = {"model": pick(), "prompt": "Tell me a short story.", "max_tokens": 32}
try:
status, text = await post(session, "/v1/completions", body)
except aiohttp.ClientError:
stats["conn_err"] += 1
await asyncio.sleep(0.2)
continue
if status == 200:
stats["ok"] += 1
elif status == 404:
stats["404"] += 1
elif status >= 500:
stats["5xx"] += 1
if first_5xx is None:
first_5xx, first_5xx_body = now(), text[:300]
print(f"[{now()}s] first 5xx: {first_5xx_body}", flush=True)
else:
stats["other_4xx"] += 1
async def reloader(session, stop):
i = 0
while not stop.is_set():
await asyncio.sleep(args.reload_interval)
name = pick()
inplace = args.mode == "mixed" and i % 2 == 1
try:
if inplace:
s, _ = await load(session, name, inplace=True)
reloads["inplace" if s == 200 else "failed"] += 1
else:
s1, _ = await post(session, "/v1/unload_lora_adapter", {"lora_name": name})
s2, _ = await load(session, name)
reloads["unload_load" if s1 == 200 and s2 == 200 else "failed"] += 1
except aiohttp.ClientError:
reloads["failed"] += 1
i += 1
async def monitor(session, stop):
global dead_at
while not stop.is_set():
await asyncio.sleep(5)
try:
async with session.get(args.base_url + "/health") as r:
healthy = r.status == 200
except aiohttp.ClientError:
healthy = False
print(f"[{now()}s] health={healthy} stats={stats} reloads={reloads}", flush=True)
if not healthy:
dead_at = now()
stop.set()
elif now() >= args.duration:
stop.set()
async def main():
timeout = aiohttp.ClientTimeout(total=120)
async with aiohttp.ClientSession(timeout=timeout) as session:
for name in names:
s, text = await load(session, name)
assert s == 200, (name, s, text)
print(f"[{now()}s] registered {len(names)} adapters", flush=True)
stop = asyncio.Event()
tasks = [asyncio.create_task(client(session, stop)) for _ in range(args.concurrency)]
tasks.append(asyncio.create_task(reloader(session, stop)))
await monitor(session, stop)
await asyncio.wait(tasks, timeout=150)
summary = {
"mode": args.mode,
"duration_s": now(),
"engine_dead_at_s": dead_at,
"first_5xx_at_s": first_5xx,
"first_5xx_body": first_5xx_body,
"requests": stats,
"reloads": reloads,
}
print("SUMMARY " + json.dumps(summary), flush=True)
asyncio.run(main()) |
linitra24
left a comment
linitra24
left a comment
There was a problem hiding this comment.
Thanks for the fix! The change looks reasonable to me.
linitra24
added
the
verified
|
/ci run |
|
✅ Triggered Buildkite CI #98616 for commit |
Overview
Fixes #64570. This change compares both
lora_nameandlora_int_idinLoRARequest, so reloading an adapter under the same name while old requests are still running no longer kills the engine.Claims
KeyErrorinconvert_mapping→EngineDeadErrorreported in [Bug]: EngineCore dies with KeyError in punicaconvert_mappingwhen a LoRA name is unloaded and loaded again while its requests are in flight #64570.Validation
EngineDeadErroronmainand passes with this fix.main, the engine exits; with this fix, both requests complete with their respective adapters and/healthreturns 200.LoraState.get_activate_lorasreturns adapter IDs[1]onmainand[1, 2]with this fix. Local CPU testing used MRV1; the GPU CI job covers MRV2.tests/lora/test_worker.py::test_worker_apply_lorafails the same way onmainbecause it does not support CPU).pre-commitpassed.Commands and output
Regression test:
main:This PR:
Server flow:
vllm serve Qwen/Qwen3-0.6B --enable-lora --max-loras 2 --max-lora-rank 8 --max-model-len 1024 --enforce-eagerwithVLLM_ALLOW_RUNTIME_LORA_UPDATING=True. Loadpet(meow adapter), start a 200-token stream onpet, unloadpet, loadpetagain (woof adapter), load one more adapter so the newpetID is evicted from its slot, then requestpet.main:KeyError: np.int64(2)→EngineDeadError; both requests fail and the server exits.Meow Meow Meow Meow Meow…, the new request returnsWoof Woof Woof Woof Woof, and/healthreturns 200.Details
LoRARequestcurrently compares only adapter names. If an adapter is unloaded and reloaded under the same name while old requests are still running, the worker can treat the old and new IDs as one adapter. The missing ID then causesconvert_mappingto raiseKeyError, killing the engine.This does not change when unload takes effect or whether it waits for in-flight requests.
Not a duplicate: no open PR references #64570 or changes
LoRARequestequality. I searched open PRs by the issue number and byconvert_mapping,LoRARequest,unload_lora_adapterandmake_lora_inputs.AI assistance: Claude Code helped prepare this PR. I reviewed the changes and ran the tests above.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.