feat(routing): latency-aware target selection (least_latency strategy)

[jarvis9443]

Adds a least_latency routing strategy: a routing model ranks its targets fastest-first by a moving average of recent observed upstream latency and falls forward down the ranked order. Targets with no samples yet rank first so they get probed (explore, then exploit).

Why

No latency-aware selection existed — a gap vs LiteLLM (lowest_latency) and Kong (lowest-latency). Second of the smart-routing strategies from the routing gap list (#873); aligns with #895 Phase 2 (the latency objective).

How

Reuses the metric-ordering framework introduced by least_cost (#681): the ranking runs in the shared resolve_attempt_models and inherits the existing retry / cooldown / health-filter / fallback machinery.

• Signal: an EWMA (alpha 0.3, ~LiteLLM's last-10 moving average) on the per-target RuntimeEntry in the runtime-status tracker, next to the existing health/cooldown state. Fed on each successful attempt from the routing dispatch loops of /v1/chat/completions (round-trip for non-streaming, time-to-first-response for streaming), /v1/messages, and /v1/responses.
• Cold start: an unmeasured target sorts first (−∞) so it gets sampled; once it has an EWMA it ranks by that. Ties keep declaration order (stable sort).

/v1/count_tokens is intentionally excluded from feeding the EWMA — its latency is a token-count round-trip, not representative of inference latency, and would skew the signal. It still gets the ranking via the shared path.

Tests

• Core + proxy unit tests: enum parsing, is_metric_based, fastest-first ordering, unmeasured-targets-probed-first, EWMA math.
• DP E2E (latency-aware-routing-e2e.test.ts): a fast + a deliberately-slow (responseDelayMs) upstream declared slow-first; after warmup the strategy sends every request to the fast target.

Notes

Regenerated the committed resource schemas (schemas/resources/{routing,model}.schema.json). User-facing docs will ship as one consolidated api7/docs routing-strategies page for the whole smart-routing family.

Fixes api7/AISIX-Cloud#924
EOF

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 19 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f6f94314-ff15-4726-b0dd-d02cd5f73e5a

📥 Commits

Reviewing files that changed from the base of the PR and between 35f5afe and 5abd3e9.

📒 Files selected for processing (10)

• crates/aisix-admin/src/openapi.rs
• crates/aisix-core/src/models/routing.rs
• crates/aisix-proxy/src/chat.rs
• crates/aisix-proxy/src/health.rs
• crates/aisix-proxy/src/messages.rs
• crates/aisix-proxy/src/responses.rs
• crates/aisix-proxy/src/routing.rs
• schemas/resources/model.schema.json
• schemas/resources/routing.schema.json
• tests/e2e/src/cases/latency-aware-routing-e2e.test.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/routing-latency-aware

---

_{Comment @coderabbitai help to get the list of available commands.}