feat(vllm): add dynamic LoRA support to unified backend

[connorcarpenter15]

Overview:

Closes the unified-backend vLLM LoRA parity gap (DIS-2032). The legacy
(non-unified) vLLM path already supported dynamic LoRA; the unified vLLM engine
did not. This adds dynamic adapter load / unload / list, discovery (MDC)
publish / unpublish, and per-request LoRA routing to the unified vLLM
engine.

Scope: unified vLLM only. Legacy (worker_factory.py) vLLM, SGLang, and
TRT-LLM paths are untouched and inherit no LoRA controls.

Details:

• New engine_update / supported_updates surface (sibling to the existing
  engine_control / supported_controls), for engine-managed assets like LoRA.
  Both share the single engine_routes() registry and the single
  /engine/{*path} route; namespacing is by the registered key
  (control/<name> vs update/<name>). The engine receives the bare op name.
• Canonical unified LoRA API: POST /engine/update/load_lora,
  POST /engine/update/unload_lora, POST /engine/update/list_loras
  (gated on --enable-lora and DYN_LORA_ENABLED).
• on_endpoint_ready handoff: a new one-time optional trait/ABC method
  hands the Python engine its runtime Endpoint so it can publish its own
  discovery records (register_model/unregister_model). The Worker calls it
  before registering engine updates and treats failure as fatal to
  startup (closes the race where a route could fire before the engine stashed
  the endpoint).
• /v1/loras compatibility shim: /v1/loras routes are registered whenever
  DYN_LORA_ENABLED=true, regardless of backend. call_lora_endpoint now falls
  back to the engine_routes() registry (update/<name>) so the legacy HTTP
  surface keeps working for unified workers, returning a clean "LoRA
  management not available" error (not the opaque "local endpoint not found")
  when neither registry holds the name. Legacy /v1/loras HTTP semantics
  (status:"error" → HTTP 500) are preserved.

Testing:

• Python vLLM LoRA unit suite (components/src/dynamo/vllm/tests/test_vllm_lora.py,
  17 tests) — passed on a real-vLLM GPU session: update gating / dispatch,
  LoRA↔control disjointness, load/unload happy-path + rollback,
  on_endpoint_ready stash, generate() lora_request routing.
• Rust unit tests — cargo test -p dynamo-backend-common (handoff ordering /
  fatal-on-failure) and cargo test -p dynamo-runtime (call_lora_endpoint
  fallback + clean-error + error-status semantics). These are behind the
  integration feature and need live etcd+NATS → run in CI.
• E2E smoke (manual): unified vLLM with DYN_LORA_ENABLED=true --enable-lora;
  exercise /engine/update/{load_lora,unload_lora,list_loras} and the
  /v1/loras compat shim; model=<lora_name> completion; adapter appears in
  /v1/models; base-model requests unaffected.

Out of scope: SGLang / TRT-LLM LoRA; Worker-side (Rust) LoRA MDC publishing;
new LoRA storage/download/cache; multimodal LoRA.

Where should the reviewer start?

• lib/backend-common/src/worker.rs — the on_endpoint_ready handoff ordering
  (before update registration, fatal on failure) and engine_update registration.
• lib/runtime/src/system_status_server.rs — the /v1/loras compat shim in
  call_lora_endpoint (engine_routes() fallback + clean unsupported error).
• components/src/dynamo/vllm/llm_engine.py — LoRA load/unload/list,
  per-request routing in generate(), and the --enable-lora + DYN_LORA_ENABLED
  gating.
• lib/backend-common/src/engine.rs / components/src/dynamo/common/backend/engine.py
  — the new on_endpoint_ready and engine_update/supported_updates surface.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Closes DIS-2032

Summary by CodeRabbit

• New Features

  • Added dynamic LoRA (Low-Rank Adapter) loading and unloading during inference
  • Added capability to list all active LoRA adapters
  • Introduced new engine update interface for managing runtime features
  • Enhanced engine initialization with endpoint-ready callback mechanism

• Documentation

  • Updated backend documentation reflecting LoRA support status and implementation details

[coderabbitai]

Lost in the diff? Review this PR in Change Stack to follow the change map from intent to exact ranges.

Walkthrough

This PR implements a unified engine-update surface across Python/Rust backends and uses it to add dynamic-LoRA loading/unloading to vLLM. The changes establish engine discovery/dispatch hooks (supported_updates, engine_update), a handoff mechanism for endpoint-ready signaling, corresponding Python-Rust bindings, a complete vLLM LoRA lifecycle with discovery publishing and concurrency control, comprehensive tests, and updated LoRA resolution in the system status server.

Changes

Engine update surface and vLLM dynamic-LoRA

Layer / File(s)	Summary
Engine interface contracts components/src/dynamo/common/backend/engine.py	GenerateRequest adds required model: str field. LLMEngine abstract interface extends with supported_updates(), engine_update(), and on_endpoint_ready() methods.
Rust trait definition and documentation lib/backend-common/src/engine.rs, lib/backend-common/CLAUDE.md	Adds three new default trait methods to LLMEngine: supported_updates (returns empty update key list), engine_update (returns "unsupported" error by default), and on_endpoint_ready (no-op by default). Trait documentation expanded from 6 to 12 methods with detailed lifecycle specifications.
Rust worker infrastructure for engine updates lib/backend-common/Cargo.toml, lib/backend-common/src/worker.rs	Introduces integration feature for NATS-backed integration tests. Updates control registration to use control/<name> namespacing. Adds register_engine_updates() method for update/<name> routes with dedicated engine_update_callback. Implements pre-handoff: on_endpoint_ready() is awaited before control/update registration in serve_with_orchestrator(). Updates error messages to reference new namespaced paths. Includes feature-gated integration tests verifying handoff ordering and failure fatality.
Python-Rust FFI bridges lib/bindings/python/rust/backend.rs	Adds PyLLMEngine implementations for supported_updates() (collects Python iterable into Vec), engine_update() (converts body JSON bidirectionally, awaits coroutine), and on_endpoint_ready() (wraps endpoint with event-loop binding). All methods use spawn_blocking under GIL and map errors to DynamoError consistently.
vLLM dynamic-LoRA implementation components/src/dynamo/vllm/llm_engine.py	Extends VllmLLMEngine with dynamic-LoRA support: constructor accepts dyn_tool_call_parser and dyn_reasoning_parser for discovery metadata; initializes endpoint, loaded-adapter registry, per-adapter async locks, and lock-map guard. on_endpoint_ready() stashes endpoint for discovery publishing. generate() resolves optional LoRARequest from request's model name. _lora_enabled() gates updates based on config and manager availability. supported_updates() advertises load_lora, unload_lora, list_loras. engine_update() dispatches to handlers with full validation, concurrency, discovery registration/unregistration, and rollback on failure.
vLLM dynamic-LoRA tests components/src/dynamo/vllm/tests/test_vllm_lora.py	17 test cases with test helpers covering: update gating (advertised only when manager+LoRA enabled), dispatcher rejection, adapter resolution in generate(), load_lora lifecycle (happy path, idempotent reload, missing manager, registration rollback), unload_lora lifecycle (happy path, not-found, unregister rollback), list_loras response shape, and on_endpoint_ready handoff (stashing for discovery, fallback behavior when absent).
LoRA management resolution lib/runtime/src/system_status_server.rs	Rewrites call_lora_endpoint() to use two-stage resolution: direct local-registry calls (in-process), fallback to engine_routes using update/<endpoint_name> key. Returns explicit "LoRA management not available" error when neither matches. Includes integration tests for fallback resolution, error messaging, and status-error propagation.
Backend documentation components/src/dynamo/common/backend/README.md	Adds LoRA to WIP list. Moves dynamic-LoRA to "What works today" for vLLM with engine-update endpoints, /v1/loras compatibility, and gating details. Updates common-gaps row to indicate vLLM support. Removes vLLM-specific gap entry. Marks LoRA load/unload + MDC publishing as done for vLLM in migration order.

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding dynamic LoRA support to the unified vLLM backend, which is the primary objective of this pull request.
Description check	✅ Passed	The description provides a comprehensive overview, detailed implementation changes, reviewer guidance, and related issue references. It addresses all template sections effectively.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

components/src/dynamo/common/backend/engine.py (1)

33-50: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make model explicitly required in GenerateRequest.

GenerateRequest is TypedDict(total=False), so the plain model: str key is non-required by default. Mark it as Required[str] to keep the cross-layer request contract intact.

Proposed fix

 class GenerateRequest(TypedDict, total=False):
@@
-    model: str
+    model: Required[str]

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/common/backend/engine.py` around lines 33 - 50,
GenerateRequest currently uses TypedDict(total=False) and declares model: str
which is treated as optional; change the model field to be explicitly required
by using Required[str] (i.e. model: Required[str]) in the GenerateRequest
TypedDict and add the appropriate import for Required (from typing or
typing_extensions consistent with project usage) so the request contract is
enforced; update the GenerateRequest definition and imports only.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/vllm/llm_engine.py`:
- Around line 635-646: The current idempotency path in load_lora treats any
entry in self.loaded_loras as fully healthy and returns early, which can mask
partial failures when rollback/discovery reconciliation fails; update the logic
in load_lora (and the similar blocks around the current checks at the sections
handling lines ~635, ~724-733, and ~830-839) to distinguish between "loaded" vs
"published/reconciled" state—either by adding a separate registry (e.g.,
self.published_loras or a status flag on the loaded entry) or by checking a
published marker on the LoRA entry before short-circuiting; change the
early-return to only succeed when the LoRA is both loaded and
published/reconciled, and ensure that on rollback/discovery failure you either
remove the partial entry from self.loaded_loras or mark it as failed so
subsequent load_lora calls will retry discovery reconciliation.

In `@lib/backend-common/src/worker.rs`:
- Around line 1009-1020: The current engine_update_callback forwards any JSON
body to engine.engine_update(...) which can cause handler exceptions for
non-object bodies; update engine_update_callback to first validate the incoming
serde_json::Value (use body.is_object()) and if it's not an object return a
structured client error (e.g., an Err with a clear message like "invalid engine
update body: expected JSON object for <update_name>") instead of calling
engine.engine_update; only call engine.engine_update(update_name, body).await
when the body is an object to avoid HTTP 500s from scalar/array payloads.

In `@lib/runtime/src/system_status_server.rs`:
- Around line 588-593: Update the error message produced by the anyhow::bail
call so it is neutral about operator actions: locate the bail that references
endpoint_name in system_status_server.rs (the anyhow::bail(...) that currently
says "LoRA management not available: no '{}' handler is registered ... Ensure
the worker was started with LoRA support enabled."), and replace the message
with a neutral statement such as "LoRA management not available: no '{}' handler
is registered (neither a local LoRA endpoint nor an engine update); this backend
does not expose LoRA management." Keep the endpoint_name interpolation and
overall error type the same.

---

Outside diff comments:
In `@components/src/dynamo/common/backend/engine.py`:
- Around line 33-50: GenerateRequest currently uses TypedDict(total=False) and
declares model: str which is treated as optional; change the model field to be
explicitly required by using Required[str] (i.e. model: Required[str]) in the
GenerateRequest TypedDict and add the appropriate import for Required (from
typing or typing_extensions consistent with project usage) so the request
contract is enforced; update the GenerateRequest definition and imports only.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 61015477-d71d-4543-bf41-344adfe6b57d

📥 Commits

Reviewing files that changed from the base of the PR and between 0658f05 and aa9ab9d.

📒 Files selected for processing (10)

• components/src/dynamo/common/backend/README.md
• components/src/dynamo/common/backend/engine.py
• components/src/dynamo/vllm/llm_engine.py
• components/src/dynamo/vllm/tests/test_vllm_lora.py
• lib/backend-common/CLAUDE.md
• lib/backend-common/Cargo.toml
• lib/backend-common/src/engine.rs
• lib/backend-common/src/worker.rs
• lib/bindings/python/rust/backend.rs
• lib/runtime/src/system_status_server.rs

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-8acf4339-660a-4f99-b1df-4cc7e0222875.docs.buildwithfern.com/dynamo/dev

[biswapanda]

lgtm