Skip to content

feat: provider/account-level concurrency cap enforcement - #1524

Merged
diegosouzapw merged 4 commits into
diegosouzapw:release/v3.7.0from
kang-heewon:provider-account-concurrency-cap
Apr 23, 2026
Merged

diegosouzapw merged 4 commits into
diegosouzapw:release/v3.7.0from
kang-heewon:provider-account-concurrency-cap

Conversation

@kang-heewon

Copy link
Copy Markdown
Contributor

Summary

  • DB schema & migration: Added provider_connections.max_concurrent column (migration 028) for declarative per-account concurrency limits.
  • Account semaphore: New open-sse/services/accountSemaphore.ts — in-memory FIFO queue keyed by provider:model:account with acquire()/release()/timeout/block hooks.
  • Request pipeline integration: chatCore.ts acquires account semaphore slot before dispatching upstream. Requests beyond the cap wait in queue or timeout with SEMAPHORE_TIMEOUT; fallback routes remain active.
  • Provider DB module: round-trip maxConcurrent through createConnection/getProviderConnectionById/updateProviderConnection. Added cleanNulls() fix to preserve explicit null values.
  • REST API: Provider limits route (/api/v1/providers/[provider]/limits) expanded to GET/PUT maxConcurrent.
  • Dashboard UI: Provider detail page now exposes account-native cap input with descriptive hint. ResilienceTab combo concurrency label clarified to distinguish combo-level from account-level.
  • i18n: EN/KO translations for new labels and hints.
  • Schema validation: maxConcurrent validates non-negative integers, accepts null/undefined as unlimited.
  • Tests: 6 semaphore unit tests + 6 DB round-trip/validation tests, all passing.

Motivation

Some providers (e.g. Alibaba) enforce per-account native concurrency caps (e.g. max 3 concurrent requests → 429 on excess). Previously there was no declarative way to enforce these caps before upstream rejection. This adds an account-level semaphore layer that sits alongside the existing model-level semaphore, providing two-tier concurrency control.

Changed Files

File Type Description
src/lib/db/migrations/028_provider_connection_max_concurrent.sql New DB migration
open-sse/services/accountSemaphore.ts New Account-level semaphore
open-sse/services/__tests__/accountSemaphore.test.ts New Semaphore tests (6)
tests/unit/account-concurrency-cap.test.ts New DB round-trip + validation tests (6)
open-sse/handlers/chatCore.ts Modified Semaphore acquire/enforcement
src/lib/db/providers.ts Modified maxConcurrent round-trip + cleanNulls
src/app/api/providers/[id]/route.ts Modified maxConcurrent in limits API
src/shared/validation/schemas.ts Modified maxConcurrent Zod validation
src/app/(dashboard)/dashboard/providers/[id]/page.tsx Modified Account cap UI input + hint
src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx Modified Combo concurrency label clarification
src/app/(dashboard)/dashboard/combos/page.tsx Modified Reference to combo concurrency
src/i18n/messages/en.json Modified EN translations
src/i18n/messages/ko.json Modified KO translations

QA

  • ✅ npm run lint — 0 errors
  • ✅ npm run typecheck:core — PASS
  • ✅ npm run build — PASS
  • ✅ accountSemaphore.test.ts — 6/6 PASS
  • ✅ account-concurrency-cap.test.ts — 6/6 PASS

Breaking Changes

None. maxConcurrent defaults to null (unlimited) — existing accounts unaffected.

- DB 마이그레이션 028: provider_connections.max_concurrent 컬럼 추가
- AccountSemaphore: 계정별 FIFO 세마포어 (acquire/release/timeout/block)
- chatCore.ts: 요청 파이프라인에 선제적 cap enforcement 통합
- providers.ts: maxConround-trip round-trip 저장/조회, cleanNulls() 보정
- API: provider limits route에서 maxConcurrent GET/PUT 지원
- UI: provider 연결 상세 페이지 account native cap 입력 필드 + hint
- UI: ResilienceTab combo concurrency 라벨 구분 (combo vs account)
- i18n: en/ko 번역 키 추가
- schemas.ts: maxConcurrent 음수 검증 + null 허용
- 테스트: semaphore 6개, DB round-trip + validation 6개

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an account-level concurrency semaphore to manage and limit simultaneous requests per provider account, addressing issues where providers enforce strict native caps. It includes a new accountSemaphore service, database migrations to store max_concurrent settings, and UI updates to configure these limits. However, the current implementation has a critical flaw where the semaphore is released prematurely for streaming requests, potentially bypassing the concurrency limit. Additionally, the semaphore key includes the model name, which contradicts the goal of enforcing a total account-level limit across all models. Finally, the new test file violates the project's style guide regarding file placement.

Comment thread open-sse/handlers/chatCore.ts Outdated
Comment thread open-sse/handlers/chatCore.ts Outdated
Comment thread tests/unit/accountSemaphore.test.ts
…, test location

- Stream release: wrap stream body with TransformStream to release
  account semaphore only when stream is fully consumed (Thread 1)
- Key scope: remove model from semaphore key — was per-account-per-model,
  now truly per-account to match PR motivation (Thread 2)
- Test location: move accountSemaphore.test.ts to tests/unit/ per style guide (Thread 3)
- Fix flaky timestamp assertions in semaphore tests
@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.7.0 April 23, 2026 11:09
@diegosouzapw
diegosouzapw merged commit 34bdd1e into diegosouzapw:release/v3.7.0 Apr 23, 2026
1 check was pending
@diegosouzapw diegosouzapw mentioned this pull request Apr 23, 2026
Poid-ZA pushed a commit to Poid-ZA/OmniRoute that referenced this pull request Aug 5, 2026
…t-concurrency-cap

feat: provider/account-level concurrency cap enforcement
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…t-concurrency-cap

feat: provider/account-level concurrency cap enforcement
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants