feat: multi-dimension rate limiting (model, team, member) - #267
Merged
Conversation
Extend rate limiting beyond per-API-key to support model, team, and member dimensions. All layers use AND logic — every active layer must pass or the request is rejected with 429. Changes: - ApiKey struct: add team_id, owner_id, team_rate_limit, owner_rate_limit - MultiReservation: wraps N reservations, commit_tokens/drop applies to all - quota.rs: multi-layer enforce (api_key → model → team → member) - chat.rs: refactored to use quota::enforce_rate_limit, streaming path uses multi-key post-stream token accounting - All non-chat endpoints updated to pass model rate limit to enforce() - Unit tests for MultiReservation (commit, drop, keys, partial failure) Part of api7/AISIX-Cloud#269
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughRate limiting moves from single API-key reservations to multi-layer enforcement (API key, model, team, owner). Adds ModelRateLimit and MultiReservation, refactors quota.enforce/enforce_rate_limit, extends ApiKey with optional team/owner fields, and updates proxy endpoints to pass model-specific limits. ChangesMulti-layer Rate Limiting
Sequence DiagramsequenceDiagram
participant Endpoint as Proxy Endpoint
participant Enforcer as quota::enforce / enforce_rate_limit
participant ApiKeyLimiter as API Key Limiter
participant ModelLimiter as Model RateLimit
participant TeamLimiter as Team RateLimit
participant OwnerLimiter as Owner RateLimit
participant MultiRes as MultiReservation
Endpoint->>Endpoint: derive ModelRateLimit (optional)
Endpoint->>Enforcer: enforce(state, auth, model_rl) / enforce_rate_limit(state, auth, model_rl)
Enforcer->>ApiKeyLimiter: reserve API-key layer
Enforcer->>ModelLimiter: reserve model layer (if present)
Enforcer->>TeamLimiter: reserve team layer (if present)
Enforcer->>OwnerLimiter: reserve owner layer (if present)
Enforcer->>MultiRes: bundle reservations
MultiRes-->>Enforcer: return MultiReservation
Enforcer-->>Endpoint: return MultiReservation
Endpoint->>MultiRes: commit_tokens(token_count) / commit on post-stream
MultiRes->>ApiKeyLimiter: commit tokens
MultiRes->>ModelLimiter: commit tokens
MultiRes->>TeamLimiter: commit tokens
MultiRes->>OwnerLimiter: commit tokens
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/aisix-proxy/src/quota.rs (1)
57-105: ⚡ Quick winExtract shared layer-reservation flow into a single helper.
enforceandenforce_rate_limitduplicate the same 4-layer pre-commit sequence. Centralizing this avoids policy drift between paths (especially on future layer/order changes).Also applies to: 115-159
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/aisix-proxy/src/quota.rs` around lines 57 - 105, Extract the duplicated 4-layer pre-commit sequence into a single helper (e.g., reserve_layers or collect_reservations) that accepts the state.limiter, auth, and model_rl and returns a Vec of reservations or a MultiReservation; move the existing logic that checks key_limits (auth.key().rate_limit), model_rl.limits (and model_rl.name), team_id/team_rate_limit, and owner_id/owner_rate_limit into that helper and call limiter.pre_commit(...) there (preserving ProxyError mapping), then replace the inlined blocks in both enforce and enforce_rate_limit with a call to this helper and construct MultiReservation::new(reservations) from its result so both paths share identical layer/order behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/aisix-proxy/src/quota.rs`:
- Around line 57-105: Extract the duplicated 4-layer pre-commit sequence into a
single helper (e.g., reserve_layers or collect_reservations) that accepts the
state.limiter, auth, and model_rl and returns a Vec of reservations or a
MultiReservation; move the existing logic that checks key_limits
(auth.key().rate_limit), model_rl.limits (and model_rl.name),
team_id/team_rate_limit, and owner_id/owner_rate_limit into that helper and call
limiter.pre_commit(...) there (preserving ProxyError mapping), then replace the
inlined blocks in both enforce and enforce_rate_limit with a call to this helper
and construct MultiReservation::new(reservations) from its result so both paths
share identical layer/order behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 0086f992-aa4c-46fa-8aff-5d5fd8945935
📒 Files selected for processing (13)
crates/aisix-core/src/models/apikey.rscrates/aisix-proxy/src/audio.rscrates/aisix-proxy/src/chat.rscrates/aisix-proxy/src/completions.rscrates/aisix-proxy/src/embeddings.rscrates/aisix-proxy/src/images.rscrates/aisix-proxy/src/messages.rscrates/aisix-proxy/src/passthrough.rscrates/aisix-proxy/src/quota.rscrates/aisix-proxy/src/rerank.rscrates/aisix-proxy/src/responses.rscrates/aisix-ratelimit/src/lib.rscrates/aisix-ratelimit/src/limiter.rs
There was a problem hiding this comment.
Pull request overview
This PR extends the gateway’s rate limiting from a single API-key bucket to multi-dimension enforcement across API key, model, team, and member (owner), using AND logic so any denied layer returns 429 while releasing held concurrency permits.
Changes:
- Added
MultiReservationto atomically manage commit/drop behavior across multiple rate-limit layer reservations. - Reworked
aisix-proxyquota enforcement to optionally apply model/team/member limits in addition to API-key limits, and wired model rate-limit resolution through all LLM endpoints. - Extended
ApiKeywith optionalteam_id/owner_idand corresponding inherited rate-limit fields, plus serde tests.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/aisix-ratelimit/src/limiter.rs | Introduces MultiReservation and adds unit tests for multi-layer reservation behavior. |
| crates/aisix-ratelimit/src/lib.rs | Re-exports MultiReservation for downstream use. |
| crates/aisix-proxy/src/quota.rs | Implements multi-layer rate-limit enforcement (API key + model + team + member) and adds ModelRateLimit helper. |
| crates/aisix-proxy/src/chat.rs | Refactors chat rate-limit reservation to use multi-layer enforcement and applies multi-key post-stream token accounting. |
| crates/aisix-proxy/src/responses.rs | Passes resolved model rate-limit info into quota enforcement. |
| crates/aisix-proxy/src/rerank.rs | Passes resolved model rate-limit info into quota enforcement. |
| crates/aisix-proxy/src/passthrough.rs | Updates enforcement call site for endpoints without model resolution (None model rate limit). |
| crates/aisix-proxy/src/messages.rs | Passes resolved model rate-limit info into quota enforcement. |
| crates/aisix-proxy/src/images.rs | Passes resolved model rate-limit info into quota enforcement. |
| crates/aisix-proxy/src/embeddings.rs | Passes resolved model rate-limit info into quota enforcement (and commits tokens across layers when known). |
| crates/aisix-proxy/src/completions.rs | Passes resolved model rate-limit info into quota enforcement. |
| crates/aisix-proxy/src/audio.rs | Passes resolved model rate-limit info into quota enforcement for both multipart and speech dispatch. |
| crates/aisix-core/src/models/apikey.rs | Adds optional team/member identifiers and rate-limit fields to ApiKey, with serde tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Deduplicate the 4-layer pre-commit sequence between enforce() and enforce_rate_limit() into a single reserve_layers() function.
- Rewrite multi_reservation_partial_failure test to actually exercise MultiReservation with multiple acquired layers - Fix from_model docstring to mention unrestricted case
This was referenced Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends rate limiting beyond per-API-key to support model, team, and member dimensions. All layers use AND logic — every active layer must pass or the request gets a 429.
What changed
ApiKey struct (
aisix-core): addedteam_id,owner_id,team_rate_limit,owner_rate_limitfields (all optional, serde defaults).MultiReservation (
aisix-ratelimit): wraps N reservations socommit_tokensandDropapply to all layers at once.keys()returns owned key list for post-stream token accounting.quota.rs (
aisix-proxy): rewritten to enforce 4 layers in order:model:<name>, usesmodel.rate_limitteam:<id>, usesapikey.team_rate_limitmember:<id>, usesapikey.owner_rate_limitIf any layer denies, earlier reservations are dropped (releasing concurrency permits).
chat.rs: refactored to use
quota::enforce_rate_limit, streaming path usesMultiReservation::keys()for multi-key post-stream accounting.All other endpoints (messages, embeddings, completions, audio, images, rerank, responses, passthrough): updated to pass model rate limit to
enforce().Design decisions
RateLimit::is_unrestricted()skips no-op layersTests
What's next (CP PR)
The control-plane PR will populate these fields via schema changes + CRUD + kine sync. Until then, all new fields default to
Noneso behavior is unchanged.Part of api7/AISIX-Cloud#269
Summary by CodeRabbit
New Features
Refactor
Tests