Expose vLLM Metrics to serve.llm API by eicherseiji · Pull Request #52719 · ray-project/ray

eicherseiji · 2025-05-01T16:32:04Z

Why are these changes needed?

This change provides visibility into Ray Serve LLM deployments, including vLLM-specific statistics.

Dashboard panels:

Docs:

Related issue number

JR-1864

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Tested following steps on https://docs.ray.io/en/latest/cluster/metrics.html

kouroshHakha

Just some V0 vs. V1 stuff. Could you also ask from observability team to review as well??

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py

python/ray/llm/_internal/serve/configs/server_models.py

python/ray/dashboard/modules/metrics/dashboards/common.py

python/ray/dashboard/modules/metrics/dashboards/serve_dashboard_panels.py

python/ray/dashboard/modules/metrics/dashboards/serve_llm_dashboard_panels.py

kouroshHakha

The changes to server_models and vllm_engine looks good to me. Thanks a ton.

kouroshHakha

Could You create docs for logging?

Basically you want to cover:

How to enable logging?
What does logging give you: i.e engine emitted metrics like vllm metrics about cache hit rate, spec decoding hit rate, etc + service level metrics like number of input tokens served, output tokens, etc
Maybe with some nice screenshots.

You don't need to create an extensive list of all metrics.

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_loggers.py

kouroshHakha

some minor change requests:

doc/source/serve/llm/serving-llms.rst

kouroshHakha

LGTM

dstrodtman

Some suggestions, mostly for clarity and to improve readability and SEO.

doc/source/serve/llm/serving-llms.rst

eicherseiji · 2025-05-13T14:40:48Z

Thanks @dstrodtman for comments!

angelinalg

Just some nits. Thanks for doing the tech writer review, Douglas and the quick resolutions, @eicherseiji!

doc/source/serve/llm/serving-llms.rst

eicherseiji · 2025-05-13T16:54:18Z

Thanks @angelinalg!

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

…1 only Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>

Adding back some default panel configurations that were accidentally removed in a prior PR #52719 Signed-off-by: Alan Guo <aguo@anyscale.com>

eicherseiji self-assigned this May 1, 2025

eicherseiji force-pushed the JR_1864 branch 2 times, most recently from 33cca0b to 56e7858 Compare May 7, 2025 02:52

hainesmichaelc added the community-contribution Contributed by the community label May 7, 2025

eicherseiji requested review from GeneDer and kouroshHakha May 8, 2025 00:05

kouroshHakha reviewed May 8, 2025

View reviewed changes

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py Outdated Show resolved Hide resolved

python/ray/llm/_internal/serve/configs/server_models.py Outdated Show resolved Hide resolved

kouroshHakha requested a review from alanwguo May 8, 2025 16:58

kouroshHakha removed the community-contribution Contributed by the community label May 8, 2025

hainesmichaelc added the community-contribution Contributed by the community label May 8, 2025

alanwguo reviewed May 8, 2025

View reviewed changes

eicherseiji marked this pull request as ready for review May 9, 2025 00:46

eicherseiji requested a review from a team as a code owner May 9, 2025 00:46

kouroshHakha approved these changes May 9, 2025

View reviewed changes

alanwguo approved these changes May 9, 2025

View reviewed changes

eicherseiji force-pushed the JR_1864 branch from e370115 to 92e324e Compare May 9, 2025 20:55

kouroshHakha approved these changes May 9, 2025

View reviewed changes

eicherseiji force-pushed the JR_1864 branch from 92e324e to e4aee09 Compare May 9, 2025 23:59

eicherseiji added the go add ONLY when ready to merge, run all tests label May 10, 2025

eicherseiji mentioned this pull request May 12, 2025

[Misc] Add Ray Prometheus logger to V1 vllm-project/vllm#17925

Merged

kouroshHakha reviewed May 12, 2025

View reviewed changes

python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_loggers.py Outdated Show resolved Hide resolved

eicherseiji requested review from a team, akshay-anyscale, edoakes and zcin as code owners May 12, 2025 20:48

kouroshHakha reviewed May 12, 2025

View reviewed changes

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

kouroshHakha approved these changes May 12, 2025

View reviewed changes

kouroshHakha enabled auto-merge (squash) May 12, 2025 22:16

mascharkh added serve Ray Serve Related Issue usability labels May 12, 2025

dstrodtman reviewed May 13, 2025

View reviewed changes

angelinalg approved these changes May 13, 2025

View reviewed changes

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

doc/source/serve/llm/serving-llms.rst Outdated Show resolved Hide resolved

eicherseiji and others added 12 commits May 13, 2025 18:17

Expose vLLM Metrics to serve.llm API

cba9ec9

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Add configuration to LLMConfig

85e934f

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Add stat logger to MQLLMEngine

8caaed1

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Initial LLM Serve panel

24d61cf

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Initial Grafana panels

6386887

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Progress porting metric dashboards

655bb6d

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Finalized dashboards, respond to comments, support metrics for vLLM V…

586ec34

…1 only Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Bring RayPrometheusStatLogger to vLLM V1

6e0fd17

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Add docs

dd2d730

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Change rayllm -> serve_llm

dd45b4b

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Docs revisions

a204da5

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Apply suggestions from code review

3e0a60d

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>

eicherseiji force-pushed the JR_1864 branch from 1cb2343 to 3e0a60d Compare May 13, 2025 18:17

kouroshHakha merged commit 881cd91 into ray-project:master May 13, 2025
5 checks passed

alanwguo mentioned this pull request May 20, 2025

fix MAX chart not being a dotted line #53184

Merged

8 tasks

matthewdeng pushed a commit that referenced this pull request May 20, 2025

fix MAX chart not being a dotted line (#53184)

bc87b73

Adding back some default panel configurations that were accidentally removed in a prior PR #52719 Signed-off-by: Alan Guo <aguo@anyscale.com>

hainesmichaelc added the community-backlog label May 22, 2025

Conversation

eicherseiji commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

dstrodtman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eicherseiji commented May 13, 2025

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eicherseiji commented May 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

eicherseiji commented May 1, 2025 •

edited

Loading