feat(global-frontend): routing metrics for affinity by jthomson04 · Pull Request #17553 · ai-dynamo/dynamo

jthomson04 · 2026-10-29T00:01:09Z

Overview

With --router-policy affinity, the Global Frontend's /metrics now shows what the selector weighs and what it decides:

each pool's occupancy;
the round trip to each region;
what became of each request's session pin.

Without these, the only way to see why sessions stay or move is the debug logs.

Stacked on #17525, which adds the Global Frontend's traffic metrics. This PR's base is that branch; it will be retargeted to main once #17525 merges.

Key Changes

About 130 core lines.

affinity.rs: AffinitySelector counts each placed request by what became of its session's pin, when it pins the session, and exposes the counts as selections(). It uses four relaxed atomics, so lib/global-router gains no Prometheus dependency.

metrics.rs: Metrics::with_routing(signals, selector) adds four families, all read at scrape time.

Family	Labels	What
`dynamo_global_frontend_pool_occupancy_ratio`	pool, region, resource	`PoolSignals::occupancy`: `slots` and `kv`, 1 being full. A pool has no series while its load is unknown, which the selector treats as full.
`dynamo_global_frontend_region_round_trip_seconds`	region	The round trip the selector orders regions by.
`dynamo_global_frontend_affinity_selections_total`	outcome	`stayed`, `moved`, `new` (no live pin) or `no_session`. The share of turns kept on their pool is `stayed / (stayed + moved)`.
`dynamo_global_frontend_affinity_pinned_sessions`	none	Sessions pinned in this replica; an expired pin counts until the next sweep, at most a minute later.

Series: 2 per pool, 1 per region, 4 outcomes and 1 more gauge. For 8 pools in 4 regions, that is about 25.
Per request: one relaxed atomic increment in the selector; everything else is computed at scrape time.

main.rs: the affinity policy passes its signals and selector to the metrics.
README and Metrics Catalog: document the four families.

Validation

Behaviour	Test
A session's first request is `new`, a repeat on its pool is `stayed`, a move for more room is `moved`; a request without a session is `no_session`	`counts_what_became_of_each_sessions_pin`
`/metrics` shows slots and KV occupancy for a pool with a report and nothing for one without, the region round trip, outcomes and pinned sessions; counters catch up with the selector at each scrape	`shows_what_the_affinity_selector_weighs_and_decides`

cargo clippy --all-targets -D warnings and cargo test pass for dynamo-global-common, dynamo-global-router and dynamo-global-frontend.

Issues and Ongoing Work

Part of feat(global-frontend): forward requests to statically configured pools #17487. Stacked on feat(global-frontend): serve /metrics #17525.

Pull Request Checklist

I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- The Global Frontend now provides a Prometheus /metrics endpoint with request counts, response errors, latency, in-flight requests, and routing and affinity measurements.
- Metrics include relevant pool, region, status, and routing details; some experimental series appear only under applicable conditions.
Documentation
- Added guidance on the endpoint, available metrics, labels, and measurement conditions.

jthomson04 · 2026-10-29T00:01:19Z

/ok to test d92d393

github-actions · 2026-10-29T00:08:01Z

🌿 Fern Docs Preview: https://nvidia-preview-01a188f7-e807-7445-ba69-d41a22c094a0.docs.buildwithfern.com/dynamo/dev

dynamo-review-agent · 2026-10-29T00:11:16Z

        }
    }

+    /// Adds what the affinity selector `affinity` weighs, from `signals`, and


This doc comment only paraphrases with_routing(signals, affinity) and the metrics constructed immediately below; it records no constraint or behavior beyond the code.

🤖 AI Fix

Remove this two-line doc comment.

sachalmalick · 2026-10-29T01:35:43Z

Do you want to raise this against main John?

devin-ai-integration · 2026-10-29T02:07:27Z

✅ Dynamo PR CI passed — run 40931422553 (attempt 1) on `3f7a1d36b9`

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 40931422553. Updated on every full-CI run of this PR.}

jthomson04 · 2026-10-29T04:12:46Z

/ok to test 94a133c

jthomson04 · 2026-10-29T05:17:55Z

/ok to test eca25d3

Serve the Global Frontend's own Prometheus families, dynamo_global_frontend_*: requests per pool by response status (with a local label for the in-region share, and cancelled for a client that leaves before the pool answers), requests answered without a pool, time to first byte and duration per pool, responses that broke off, and requests in flight per pool. Series are labelled with the pool's name and region, never with the request's model. Pools now carry a name for people, their <cluster>/<namespace>/<DGD name>. Inter-token latency and token counts need the stream parsed and stay with the pool frontends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

jthomson04 · 2026-10-29T17:41:33Z

/ok to test e8b7f09

A Dynamo pool frontend sends its response headers once a worker has the request, so a client that gives up while a worker queues its request counts under the pool's status, not as cancelled. Also note that an SSE keep-alive before the first token counts as the first byte, and list no_response in the README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

With --router-policy affinity, /metrics also shows what the selector weighs and decides, read at scrape time: each pool's occupancy by slots and KV (absent while unknown), the round trip to each region, each request's outcome for its session's pin (stayed, moved, new, no_session), and the sessions pinned. AffinitySelector counts outcomes in relaxed atomics, so lib/global-router gains no Prometheus dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>

jthomson04 · 2026-10-29T18:30:08Z

/ok to test 3f7a1d3

coderabbitai · 2026-10-29T19:52:41Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: ai-dynamo/dynamo
Branch: jthomson04/global-frontend-routing-metrics
File: .coderabbit.yaml
Commit: 3f7a1d3
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: 8a51cebc-e8e1-4b0c-b9ba-3a9c72e58510

Commits

Reviewing files that changed from the base of the PR and between 05442be and 3f7a1d3.

Files ignored due to path filters: 1

Cargo.lock is excluded by !**/*.lock

Files selected for processing: 10

docs/fern/pages/reference/observability/metrics-catalog.mdx
lib/global-frontend/Cargo.toml
lib/global-frontend/README.md
lib/global-frontend/src/lib.rs
lib/global-frontend/src/main.rs
lib/global-frontend/src/metrics.rs
lib/global-frontend/src/proxy.rs
lib/global-frontend/tests/forwarding.rs
lib/global-router/src/affinity.rs
lib/global-router/src/pool.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The Global Frontend now exposes Prometheus metrics at /metrics. The metrics cover request outcomes, response timing, in-flight requests, and affinity routing. The proxy, affinity selector, pool data, tests, and documentation are updated to support and describe these metrics.

Changes

Global Frontend metrics

Layer / File(s)	Summary
Affinity outcomes and pool labels `lib/global-router/src/affinity.rs`, `lib/global-router/src/pool.rs`	`AffinitySelector` counts `stayed`, `moved`, `new`, and `no_session` outcomes. `Pool` exposes a name built from its cluster, namespace, and DGD name.
Metric registration and refresh `lib/global-frontend/Cargo.toml`, `lib/global-frontend/src/lib.rs`, `lib/global-frontend/src/metrics.rs`	`Metrics` registers request, response, timing, in-flight, and optional routing metrics. Scrapes refresh gauges from pool snapshots and encode the Prometheus registry.
Proxy instrumentation and endpoint `lib/global-frontend/src/main.rs`, `lib/global-frontend/src/proxy.rs`, `lib/global-frontend/tests/forwarding.rs`, `lib/global-frontend/README.md`, `docs/fern/pages/reference/observability/metrics-catalog.mdx`	The proxy records request outcomes and response timing, and serves metrics at `/metrics`. The frontend configures metrics, tests check request and routing measurements, and the documentation describes the endpoint and metric families.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Minimal · up to 3f7a1

No identified issue blocks merging after normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the main change: adding affinity routing metrics to the Global Frontend.
Description check		The description includes all required template sections and provides clear details about the changes, metrics, validation, and ongoing work.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 21.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 14 functions across 7 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

⚔️ Resolve merge conflicts: 💡

Resolve merge conflict in branch jthomson04/global-frontend-routing-metrics

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

jthomson04 requested review from a team as code owners October 29, 2026 00:01

pull-request-size Bot added the size/L label Oct 29, 2026

github-actions Bot added the feat label Oct 29, 2026

github-actions Bot added the documentation Improvements or additions to documentation label Oct 29, 2026

jthomson04 mentioned this pull request Oct 29, 2026

feat(global-frontend): serve /metrics #17525

Merged

1 task done

dynamo-review-agent Bot reviewed Oct 29, 2026

View reviewed changes

jthomson04 force-pushed the jthomson04/global-frontend-passthrough branch from b4dbaa9 to f088c63 Compare October 29, 2026 04:10

jthomson04 requested review from a team as code owners October 29, 2026 04:10

jthomson04 force-pushed the jthomson04/global-frontend-routing-metrics branch from d92d393 to 94a133c Compare October 29, 2026 04:10

jthomson04 mentioned this pull request Oct 29, 2026

feat(global-router): Global Frontend with load- and session-aware pool selection #17524

Merged

1 task done

jthomson04 force-pushed the jthomson04/global-frontend-passthrough branch from f088c63 to 7c924bc Compare October 29, 2026 05:17

jthomson04 force-pushed the jthomson04/global-frontend-routing-metrics branch from 94a133c to eca25d3 Compare October 29, 2026 05:17

jthomson04 force-pushed the jthomson04/global-frontend-passthrough branch from 7c924bc to eb4f370 Compare October 29, 2026 16:57

jthomson04 force-pushed the jthomson04/global-frontend-routing-metrics branch from eca25d3 to e8b7f09 Compare October 29, 2026 17:41

jthomson04 and others added 2 commits October 29, 2026 11:28

jthomson04 force-pushed the jthomson04/global-frontend-routing-metrics branch from e8b7f09 to 3f7a1d3 Compare October 29, 2026 18:30

Base automatically changed from jthomson04/global-frontend-passthrough to main October 29, 2026 19:45

pull-request-size Bot removed the size/L label Oct 29, 2026

pull-request-size Bot added the size/XL label Oct 29, 2026

sachalmalick mentioned this pull request Oct 30, 2026

feat(global-router): add DynamoDB-backed session affinity #17675

Open

1 task done

jthomson04 mentioned this pull request Oct 30, 2026

perf(global-frontend): serve with per-connection shutdown signals #17717

Merged

1 task done

jh-nv mentioned this pull request Nov 3, 2026

docs(observability): use one table format and mark stability in the Metrics Catalog #17922

Open

1 task done

Navigation Menu

Repository navigation

feat(global-frontend): routing metrics for affinity - #17553

feat(global-frontend): routing metrics for affinity#17553
jthomson04 wants to merge 3 commits into
mainfrom
jthomson04/global-frontend-routing-metrics

jthomson04 commented Oct 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

dynamo-review-agent Bot Oct 29, 2026

Uh oh!

sachalmalick commented Oct 29, 2026

Uh oh!

devin-ai-integration Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

coderabbitai Bot commented Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jthomson04 commented Oct 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Key Changes

Validation

Issues and Ongoing Work

Summary by CodeRabbit

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dynamo-review-agent Bot Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

sachalmalick commented Oct 29, 2026

Uh oh!

devin-ai-integration Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Dynamo PR CI passed — run 40931422553 (attempt 1) on 3f7a1d36b9

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

jthomson04 commented Oct 29, 2026

Uh oh!

coderabbitai Bot commented Oct 29, 2026

Walkthrough

Changes

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jthomson04 commented Oct 29, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented Oct 29, 2026 •

edited

Loading

devin-ai-integration Bot commented Oct 29, 2026 •

edited

Loading

✅ Dynamo PR CI passed — run 40931422553 (attempt 1) on `3f7a1d36b9`