fix(metrics): preserve colons in /engine_metrics metric names

[chromecast56]

PR stack (merge bottom-up; each PR is based on main and includes the PRs below it)

1. fix(metrics): preserve colons in /engine_metrics metric names #2636 ← this PR
2. fix(router): forward client JSON numbers exactly on HTTP and PD paths #2637
3. feat(router): route /v1/messages/count_tokens and forward Anthropic headers #2638
4. perf(pd): commit the client stream on decode's 2xx, not the prefill body #2639

Bottom of the stack: review and merge this first.

---

Description

Problem

/engine_metrics renames every engine metric that contains a colon. openmetrics_parser rejects colons in metric names, so aggregate_metrics replaced every : in the scraped text with _. As a result:

• SGLang's sglang:* and vLLM's vllm:* metrics come out as sglang_* / vllm_*, so a query that works against an engine's /metrics doesn't work against the gateway.
• Colons inside label values and HELP text are rewritten too, e.g. model_name="org/model:v1" becomes org/model_v1.
• Distinct families a:b and a_b are merged into one.

Solution

Before parsing, swap each colon for an escape sequence, and swap it back in the rendered output. The escape is xsmgcolon{n}z, using the first n that appears nowhere in the scraped text or the extra labels. That way a literal copy of the escape in the input is never rewritten. Only the first character is x, so two adjacent escapes can't combine into a false match.

Ported from sgl-project/sglang#37985.

Changes

• model_gateway/src/worker/metrics_aggregator.rs: escape and restore colons instead of replacing them with _.
• model_gateway/tests/metrics_aggregator_test.rs: test_real now expects sglang: names. New tests cover colons in names, HELP text and label values; a:b vs a_b twins plus leading (:x) and doubled (a::b) colons; and a literal escape in the input.

No dashboards, docs or examples in this repo query sglang_* engine metrics, so nothing else needs updating.

Test Plan

cargo test -p smg --test metrics_aggregator_test

test test_aggregate_multiple_metrics ... ok
test test_colon_and_underscore_twins_stay_distinct ... ok
test test_colons_in_names_and_label_values_are_preserved ... ok
test test_empty_input ... ok
test test_invalid_metrics_are_skipped ... ok
test test_mismatched_labels_pd_mode ... ok
test test_literal_colon_escape_is_not_rewritten ... ok
test test_aggregate_simple ... ok
test test_real ... ok
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

On main (only the aggregator source reverted), the new and updated tests fail:

test test_aggregate_simple ... ok
test test_colon_and_underscore_twins_stay_distinct ... FAILED
test test_aggregate_multiple_metrics ... ok
test test_empty_input ... ok
test test_invalid_metrics_are_skipped ... ok
test test_colons_in_names_and_label_values_are_preserved ... FAILED
test test_mismatched_labels_pd_mode ... ok
test test_literal_colon_escape_is_not_rewritten ... FAILED
test test_real ... FAILED
thread 'test_colon_and_underscore_twins_stay_distinct' (2170376) panicked at model_gateway/tests/metrics_aggregator_test.rs:396:5:
thread 'test_colons_in_names_and_label_values_are_preserved' (2170377) panicked at model_gateway/tests/metrics_aggregator_test.rs:93:5:
thread 'test_literal_colon_escape_is_not_rewritten' (2170380) panicked at model_gateway/tests/metrics_aggregator_test.rs:149:5:
thread 'test_real' (2170382) panicked at model_gateway/tests/metrics_aggregator_test.rs:396:5:
test result: FAILED. 5 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes (run without --all-features / OpenCV: cargo clippy --workspace --all-targets -- -D warnings)
• (Optional) Documentation updated
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 6dcfb25e-2f77-4890-bfa7-4556373dc2f1

📥 Commits

Reviewing files that changed from the base of the PR and between 2e61389 and ef6b046.

📒 Files selected for processing (2)

• model_gateway/src/worker/metrics_aggregator.rs
• model_gateway/tests/metrics_aggregator_test.rs

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Summary

Summary by CodeRabbit

• Bug Fixes
  • Prometheus metric names containing colons are now preserved during aggregation, including names that differ only by colons versus underscores.
  • Colons in label values and literal text resembling escape markers remain unchanged.
  • Aggregated time series now include a source label.

Walkthrough

Metric aggregation now preserves colons in metric names and label values by using an unused escape string during parsing and restoring colons in the output. Regression tests cover colon and underscore name collisions, literal escape strings, and expected aggregated series.

Changes

Metric Aggregation

Layer / File(s)	Summary
Colon-preserving aggregation and regression coverage model_gateway/src/worker/metrics_aggregator.rs, model_gateway/tests/metrics_aggregator_test.rs	aggregate_metrics selects an escape string not found in input metric text or labels, substitutes it for colons before parsing, then restores colons in the output. Tests cover colon-containing names and values, distinct colon/underscore metric families, literal escape strings, and expected series labels.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to ef6b0

The /engine_metrics endpoint now reports engine metric names with their original colons (for example sglang:* and vllm:*), as intended. Internal load monitoring reads each worker directly and is unaffected. No merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 88.89% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 2 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The pull request title 'fix(metrics): preserve colons in /engine_metrics metric names' accurately describes the primary change. The title clearly specifies that the fix affects how colons are handled …
Description check	✅ Passed	The pull request description is directly related to the changeset. It explains the problem (colons being replaced with underscores), the solution (using an escape-and-restore approach with xsmgcolon{n…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}