Skip to content

docs(resilience): Quota-Share Concurrency Control (max_concurrent + serialization + cooldown-wait) #4980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
diegosouzapw merged 1 commit into from
Jun 25, 2026
Merged

docs(resilience): Quota-Share Concurrency Control (max_concurrent + serialization + cooldown-wait) #4980

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/architecture/RESILIENCE_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,46 @@ lockout *state* is ephemeral.

---

## 4. Quota-Share Concurrency Control (v3.8.36)

Subscription accounts (GLM, MiniMax, etc.) often accept only ~1–3 concurrent
requests; exceeding that triggers 429s and cooldowns. This is acute under
**quota-share** (`qtSd/…`) combos, where several API keys share one upstream
account. Three layers keep a shared account from being flooded.

### Per-connection concurrency cap (`max_concurrent`)

Each provider connection can declare a `max_concurrent` ceiling
(`provider_connections.max_concurrent`, set in the connection modal / API / DB).
Leave it empty for no limit. This is the single knob that drives the serialization
layer below — set it to the account's real concurrency (e.g. GLM ~1, MiniMax ~2).

### Quota-share request serialization

When a quota-share dispatch targets a connection that declares a positive
`max_concurrent`, concurrent requests to that **account** are serialized through a
per-connection semaphore (key `qsconn:<connectionId>`): excess requests **wait in
the queue** instead of flooding the account. It is **fail-open** — a saturated
queue or timeout proceeds without a slot rather than ever rejecting a dispatchable
request. Toggle in **Settings → Resilience → Quota-share per-connection
concurrency** (`resilienceSettings.quotaShareConcurrencyLimit.enabled`, default
on). Without a `max_concurrent` cap the behavior is unchanged.

> The quota-share routing gate (`selectQuotaShareTarget`, DRR + P2C) is itself
> fail-open and only *deprioritizes* an at-cap connection — with a
> single-connection pool it cannot hard-limit, so this semaphore is what actually
> contains the flood.

### Combo cooldown-aware retry

For quota-share combos only, a request that would crystallize a 429 for a SHORT
transient cooldown waits it out and re-dispatches instead of returning the 429.
Bounded by `comboCooldownWait` (`enabled`, `maxWaitMs` 5s, `maxAttempts` 2,
`budgetMs` 8s) in **Settings → Resilience**. It never waits on `quota_exhausted`
(locked until midnight) or auth/not-found reasons.

---

## Other Resilience Features

- **14 routing strategies** (priority, weighted, round-robin, context-relay, fill-first, p2c, random, least-used, cost-optimized, reset-aware, strict-random, auto, lkgp, context-optimized) — see [AUTO-COMBO.md](../routing/AUTO-COMBO.md).
Expand Down
Loading