feat: provider/account-level concurrency cap enforcement - #1524
diegosouzapw merged 4 commits into
Conversation
- DB 마이그레이션 028: provider_connections.max_concurrent 컬럼 추가 - AccountSemaphore: 계정별 FIFO 세마포어 (acquire/release/timeout/block) - chatCore.ts: 요청 파이프라인에 선제적 cap enforcement 통합 - providers.ts: maxConround-trip round-trip 저장/조회, cleanNulls() 보정 - API: provider limits route에서 maxConcurrent GET/PUT 지원 - UI: provider 연결 상세 페이지 account native cap 입력 필드 + hint - UI: ResilienceTab combo concurrency 라벨 구분 (combo vs account) - i18n: en/ko 번역 키 추가 - schemas.ts: maxConcurrent 음수 검증 + null 허용 - 테스트: semaphore 6개, DB round-trip + validation 6개
There was a problem hiding this comment.
Code Review
This pull request introduces an account-level concurrency semaphore to manage and limit simultaneous requests per provider account, addressing issues where providers enforce strict native caps. It includes a new accountSemaphore service, database migrations to store max_concurrent settings, and UI updates to configure these limits. However, the current implementation has a critical flaw where the semaphore is released prematurely for streaming requests, potentially bypassing the concurrency limit. Additionally, the semaphore key includes the model name, which contradicts the goal of enforcing a total account-level limit across all models. Finally, the new test file violates the project's style guide regarding file placement.
…, test location - Stream release: wrap stream body with TransformStream to release account semaphore only when stream is fully consumed (Thread 1) - Key scope: remove model from semaphore key — was per-account-per-model, now truly per-account to match PR motivation (Thread 2) - Test location: move accountSemaphore.test.ts to tests/unit/ per style guide (Thread 3) - Fix flaky timestamp assertions in semaphore tests
34bdd1e
into
diegosouzapw:release/v3.7.0
…t-concurrency-cap feat: provider/account-level concurrency cap enforcement
…t-concurrency-cap feat: provider/account-level concurrency cap enforcement
Summary
provider_connections.max_concurrentcolumn (migration 028) for declarative per-account concurrency limits.open-sse/services/accountSemaphore.ts— in-memory FIFO queue keyed byprovider:model:accountwithacquire()/release()/timeout/blockhooks.chatCore.tsacquires account semaphore slot before dispatching upstream. Requests beyond the cap wait in queue or timeout withSEMAPHORE_TIMEOUT; fallback routes remain active.maxConcurrentthroughcreateConnection/getProviderConnectionById/updateProviderConnection. AddedcleanNulls()fix to preserve explicitnullvalues./api/v1/providers/[provider]/limits) expanded to GET/PUTmaxConcurrent.maxConcurrentvalidates non-negative integers, acceptsnull/undefinedas unlimited.Motivation
Some providers (e.g. Alibaba) enforce per-account native concurrency caps (e.g. max 3 concurrent requests → 429 on excess). Previously there was no declarative way to enforce these caps before upstream rejection. This adds an account-level semaphore layer that sits alongside the existing model-level semaphore, providing two-tier concurrency control.
Changed Files
src/lib/db/migrations/028_provider_connection_max_concurrent.sqlopen-sse/services/accountSemaphore.tsopen-sse/services/__tests__/accountSemaphore.test.tstests/unit/account-concurrency-cap.test.tsopen-sse/handlers/chatCore.tssrc/lib/db/providers.tssrc/app/api/providers/[id]/route.tssrc/shared/validation/schemas.tssrc/app/(dashboard)/dashboard/providers/[id]/page.tsxsrc/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsxsrc/app/(dashboard)/dashboard/combos/page.tsxsrc/i18n/messages/en.jsonsrc/i18n/messages/ko.jsonQA
npm run lint— 0 errorsnpm run typecheck:core— PASSnpm run build— PASSaccountSemaphore.test.ts— 6/6 PASSaccount-concurrency-cap.test.ts— 6/6 PASSBreaking Changes
None.
maxConcurrentdefaults tonull(unlimited) — existing accounts unaffected.