Skip to content

feat(selection): Add Prometheus metrics for model selection evolution tracking#1124

Merged
rootfs merged 1 commit intovllm-project:mainfrom
asaadbalum:feat/issue-1093-prometheus-metrics-selection
Jan 21, 2026
Merged

feat(selection): Add Prometheus metrics for model selection evolution tracking#1124
rootfs merged 1 commit intovllm-project:mainfrom
asaadbalum:feat/issue-1093-prometheus-metrics-selection

Conversation

@asaadbalum
Copy link
Copy Markdown
Collaborator

@asaadbalum asaadbalum commented Jan 20, 2026 •
edited
Loading

Summary

Implements issue #1093 - adds comprehensive Prometheus metrics for explainability and traceability of ALL model selection methods evolution.

This is a follow-up to #987 (Advanced Model Selection Methods) as suggested by @rootfs in PR #1089.

Dashboard Demo

image

Key Features Shown:

  • Elo Rating Over Time: Live evolution of model ratings based on user feedback
  • Method Filter: Dropdown with ALL 5 methods (elo, automix, hybrid, router_dc, static)
  • Current Elo Ratings: Real-time ratings per model and category
  • Category Filter: Filter by decision category (tech, finance, etc.)

New Metrics

General Selection Metrics (Auto-recorded for ALL methods)

Metric Type Description
llm_model_selection_total Counter Total selections by method/model
llm_model_selection_history Counter Selection count over time
llm_model_selection_score Histogram Score distribution
llm_model_selection_confidence Histogram Confidence distribution

Elo-specific Metrics

Metric Type Description
llm_model_elo_rating Gauge Current Elo rating per model/category
llm_model_feedback_total Counter Feedback events (wins/losses/ties)
llm_model_rating_change Histogram Rating change distribution
llm_model_win_rate Gauge Win rate per model

AutoMix-specific Metrics

Metric Type Description
llm_model_automix_verification_prob Gauge Learned verification probability
llm_model_automix_quality Gauge Learned quality score
llm_model_automix_success_rate Gauge Query success rate

RouterDC-specific Metrics

Metric Type Description
llm_model_routerdc_similarity Histogram Query-model similarity distribution
llm_model_routerdc_affinity Gauge Learned affinity from feedback

Hybrid-specific Metrics

Metric Type Description
llm_model_selection_component_agreement Histogram Component agreement ratio

Architecture

  • Basic metrics are auto-recorded centrally in req_filter_classification.go for ALL methods
  • Method-specific evolution metrics are recorded within each selector
  • New methods automatically get basic metrics - no code needed
  • Evolution tracking requires method-specific code if the method has unique learning data

Files Changed

File Change
pkg/selection/metrics.go New metrics definitions and recording functions
pkg/selection/automix.go Record AutoMix evolution metrics
pkg/selection/router_dc.go Record similarity/affinity metrics
pkg/selection/hybrid.go Record component agreement
pkg/selection/elo.go Record Elo evolution metrics
pkg/selection/metrics_test.go Comprehensive tests for all methods
pkg/selection/METRICS.md Documentation
deploy/docker-compose/addons/model-selection-dashboard.json Grafana dashboard

Testing

All tests pass including comprehensive evolution tests for each method:

=== Testing Evolution Metrics for ALL Selection Methods ===
✅ Elo evolution works
✅ AutoMix evolution works  
✅ RouterDC evolution works
✅ Component agreement metrics recorded

Acceptance Criteria

  • Prometheus metrics exposed at /metrics endpoint
  • Grafana dashboard template for visualizing evolution
  • Documentation for metrics interpretation
  • Metrics for ALL selection methods

Fixes: #1093

@netlify
Copy link
Copy Markdown

netlify Bot commented Jan 20, 2026 •
edited
Loading

✅ Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit cda215e
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/6970e50912bbe500077014c3
😎 Deploy Preview https://deploy-preview-1124--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jan 20, 2026 •
edited
Loading

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 deploy

Owners: @rootfs, @Xunzhuo
Files changed:

  • deploy/docker-compose/addons/model-selection-dashboard.json

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

  • src/semantic-router/pkg/extproc/router.go
  • src/semantic-router/pkg/selection/METRICS.md
  • src/semantic-router/pkg/selection/automix.go
  • src/semantic-router/pkg/selection/elo.go
  • src/semantic-router/pkg/selection/factory.go
  • src/semantic-router/pkg/selection/hybrid.go
  • src/semantic-router/pkg/selection/metrics.go
  • src/semantic-router/pkg/selection/metrics_test.go
  • src/semantic-router/pkg/selection/router_dc.go
vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

@asaadbalum asaadbalum marked this pull request as draft January 20, 2026 08:47
@asaadbalum asaadbalum closed this Jan 20, 2026
@asaadbalum asaadbalum reopened this Jan 20, 2026
@asaadbalum asaadbalum force-pushed the feat/issue-1093-prometheus-metrics-selection branch 7 times, most recently from 9572e03 to 629a686 Compare January 21, 2026 11:38
@asaadbalum asaadbalum marked this pull request as ready for review January 21, 2026 11:39
@asaadbalum asaadbalum marked this pull request as draft January 21, 2026 11:40
@asaadbalum asaadbalum force-pushed the feat/issue-1093-prometheus-metrics-selection branch 9 times, most recently from 1deff25 to c94cdc8 Compare January 21, 2026 14:24
@asaadbalum
feat(selection): Add Prometheus metrics for model selection evolution…
cda215e 
… tracking

Implements comprehensive Prometheus metrics for explainability and
traceability of ALL model selection methods evolution.

## New Metrics

### Elo-specific
- llm_model_elo_rating: Current Elo rating per model/category
- llm_model_feedback_total: Feedback events (wins/losses/ties)
- llm_model_rating_change: Distribution of rating changes
- llm_model_comparisons_total: Total comparisons per model
- llm_model_win_rate: Win rate percentage

### AutoMix-specific
- llm_model_automix_verification_prob: Learned verification probability
- llm_model_automix_quality: Learned quality score
- llm_model_automix_success_rate: Query success rate

### RouterDC-specific
- llm_model_routerdc_similarity: Query-model similarity distribution
- llm_model_routerdc_affinity: Learned affinity scores

### Hybrid-specific
- llm_model_component_agreement: Agreement ratio between component selectors

## Other Changes
- Pre-initialize all methods (elo, router_dc, automix, hybrid, static)
- Fix GlobalRegistry initialization in router.go
- Fix AutoMix deadlock in updateValueFunction
- Add Grafana dashboard with method-specific panels
- Add comprehensive METRICS.md documentation
- Add unit tests for all new metrics

Signed-off-by: asaadbalum <asaad.balum@gmail.com>
@asaadbalum asaadbalum marked this pull request as ready for review January 21, 2026 14:33
@asaadbalum asaadbalum force-pushed the feat/issue-1093-prometheus-metrics-selection branch from c94cdc8 to cda215e Compare January 21, 2026 14:39
@asaadbalum asaadbalum requested a review from rootfs January 21, 2026 14:43
@rootfs rootfs merged commit 398b928 into vllm-project:main Jan 21, 2026
32 checks passed
@rootfs
Copy link
Copy Markdown
Collaborator

rootfs commented Jan 21, 2026

@asaadbalum this is cool! can you record a video of the dashboard over time? we can add to our youtube channel. Thanks

@asaadbalum
Copy link
Copy Markdown
Collaborator Author

asaadbalum commented Jan 21, 2026

@asaadbalum this is cool! can you record a video of the dashboard over time? we can add to our youtube channel. Thanks

Sure will do

henschwartz pushed a commit to henschwartz/semantic-router that referenced this pull request Feb 18, 2026
@asaadbalum
feat(selection): Add Prometheus metrics for model selection evolution…
be71d73 
… tracking (vllm-project#1124)

Implements comprehensive Prometheus metrics for explainability and
traceability of ALL model selection methods evolution.

## New Metrics

### Elo-specific
- llm_model_elo_rating: Current Elo rating per model/category
- llm_model_feedback_total: Feedback events (wins/losses/ties)
- llm_model_rating_change: Distribution of rating changes
- llm_model_comparisons_total: Total comparisons per model
- llm_model_win_rate: Win rate percentage

### AutoMix-specific
- llm_model_automix_verification_prob: Learned verification probability
- llm_model_automix_quality: Learned quality score
- llm_model_automix_success_rate: Query success rate

### RouterDC-specific
- llm_model_routerdc_similarity: Query-model similarity distribution
- llm_model_routerdc_affinity: Learned affinity scores

### Hybrid-specific
- llm_model_component_agreement: Agreement ratio between component selectors

## Other Changes
- Pre-initialize all methods (elo, router_dc, automix, hybrid, static)
- Fix GlobalRegistry initialization in router.go
- Fix AutoMix deadlock in updateValueFunction
- Add Grafana dashboard with method-specific panels
- Add comprehensive METRICS.md documentation
- Add unit tests for all new metrics

Signed-off-by: asaadbalum <asaad.balum@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rootfs rootfs rootfs approved these changes

Assignees

@rootfs rootfs

@wangchen615 wangchen615

@Xunzhuo Xunzhuo

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[v0.2-Athena]: Add Prometheus metrics for model selection evolution tracking

4 participants

@asaadbalum @rootfs @wangchen615 @Xunzhuo