[Metrics][LoRA] Publish LoRA load events and expose loaded adapters in metrics

[wseaton]

Stack

1. [Engine Core] Gather engine notifications from workers to frontends #51433 engine notification channel
2. [Metrics][LoRA] Publish LoRA load events and expose loaded adapters in metrics #54830 LoRA load events, doorbell, and Python Prometheus metrics (this PR)
3. [Rust][Metrics] Consume LoRA load events in the Rust frontend #54833 Rust frontend consumer for the same metrics

Each PR is based on main and includes the commits of the ones before it; review only the last commits of each.

End-to-end testing on the Rust frontend needs #54836 (--lora-modules support), tracked separately.

Note: the metrics report worker-side residency. /v1/unload_lora_adapter on the Python server only drops the frontend registry entry (#54839), so an unloaded adapter keeps its series until LRU eviction. This bug is now actually visible!

[mergify]

Documentation preview: https://vllm--54830.org.readthedocs.build/en/54830/

[atantawi]

Downstream consumer here — @wseaton pointed me at this stack from
llm-d/llm-d-router#1500. Putting the consumer-side case on the record, since "who needs
this metric" is what a metrics PR usually has to answer.

llm-d-router's LoRA scorer is broken today in two directions, and this PR fixes both.
It parses vllm:lora_requests_info into ActiveModels / WaitingModels and scores
endpoints in four tiers. Read at llm-d-router 7f1a611e against vLLM 67e86d1e6f:

• LoRARequestStates.update_scheduler_stats (vllm/v1/metrics/stats.py:637-639) is the
  only writer of either dict and writes every key of self.requests into both;
  loggers.py:1168-1172 renders each as ",".join(...keys()). The two label values are
  two renderings of one key set, so the router's "adapter queued to load" tier is dead
  code — waiting implies active, tested first.
• Its capacity tier tests len(active)+len(waiting) < max_lora, which is 2N for N
  active adapters and trips at N ≥ max_lora/2. Meanwhile an adapter resident with no
  traffic is invisible, so a full pod can read as empty. One expression, wrong in both
  directions.

Confirmed by running the router's own extractor and scorer, kept as mutation-tested Go
tests: with running = waiting = "target,other" and max_lora = 4, a queued target scores
1.0 instead of the intended 0.6, and a cold target scores 0.0 with two of four slots
free. Filed as llm-d/llm-d-router#2605 before I knew this stack existed.

vllm:num_gpu_loaded_lora_adapters against max_lora is exactly the comparison that
capacity tier fails to be, and vllm:lora_adapter_loaded{level="gpu"} is the residency its
tier names already claim to test. On cardinality, since that is the usual objection to a
per-adapter label: it is bounded by max_cpu_loras, not by the registered population, which
in our survey of adapter populations runs to 10²–10³ per base-model family with only a
handful resident.

I read the producer rather than the label names — misreading a label name is what put me
wrong on the llm-d thread to begin with — and the residency claim holds: activate_adapter
evicts only at len(_active_adapters) >= lora_slots and _apply_adapters only ever adds,
so an idle adapter stays in gpu_adapters. Three questions from that reading:

1. Running vs queued stays unavailable. The stack leaves the info-metric alone, so a
   consumer still cannot reconstruct scheduled_loras (scheduler.py:764-769, 814-824) —
   the quantity a capacity-aware scheduler actually wants. Is splitting the running set in
   scope here, or is the intent that consumers use residency plus activity and not try?
2. TP > 1 labelling. The gather flattens every rank and the labels carry no rank, so
   _record_lora_load_event is last-write-wins per engine_idx against
   _lora_loaded_series. Identical sets across ranks make that idempotent — guaranteed, or
   can a transient disagreement flap a series?
3. The idle-engine claim. docs/design/metrics.md says the gauges update "including on
   an idle engine with statically configured adapters". I could not confirm that
   --lora-modules puts the adapter in the worker's cache before the first request, as
   opposed to the frontend registry [Bug]: /v1/unload_lora_adapter never removes the adapter from the engine #54839 is about. Worth a test if scrape-before-traffic
   is load-bearing.

[atantawi]

Cross-linking one downstream consumer: @dmitripikus owns
llm-d-router#709 (dynamic LoRA
placement/balancing) and asked to be pointed at the relevant work. This stack is the first
per-adapter residency signal a placement- or eviction-aware router could use;
vllm:lora_requests_info reports in-flight activity instead, which is the failure mode
described in my review
above.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wseaton.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[atantawi]

Withdrawing my third question: it was already answered, on #54833, before I asked it.

Your test report there (comment) shows the startup scrape under --lora-modules alice=…, before any request:

vllm:num_gpu_loaded_lora_adapters{engine="0"} 1
vllm:lora_adapter_loaded{engine="0",adapter_name="alice",level="gpu",pinned="false"} 1

with the Python frontend giving the same numbers. That is exactly the scrape-before-traffic case I said I could not confirm, and it is worker-side rather than a frontend registry — the event's lists come from LoRAModelManager.get_loaded_state() reading _active_adapters / _registered_adapters. I reviewed one PR of a stack without reading its sibling's test report; sorry for the noise.

Questions 1 and 2 stand, and 1 is the one that matters downstream: with the info-metric untouched, a consumer still cannot separate running from queued, so llm-d-router's "adapter queued to load" tier stays dead code even after this lands, and scheduled_loras stays unreconstructible. If splitting the running set is out of scope for this stack, that is a perfectly good answer — I would just rather know, so the router-side consumer can be written against what this will actually emit.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[wseaton]

Marking as ready for review (have end to end tests now in the llm-d router), but still needs it's first PR in the stack reviewed first.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wseaton.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork