Repository navigation
fix(api): scale pool usage snapshot limits by pool member count (summed budget) - #10253
Conversation
…r-inference#189, cheaper-inference#190) Bumps: nanoid ^3.3.17 (was transitive, now overridden), dompurify ^3.4.13 (with monaco-editor scoped override). Closes Dependabot cheaper-inference#189, cheaper-inference#190. Remaining cheaper-inference#182-cheaper-inference#188 (js-yaml + mermaid) already closed by cheaper-inference#9651 merge — awaiting Dependabot re-scan. npm audit → 0 vulnerabilities.
cheaper-inference#190 Closes Dependabot cheaper-inference#189 (dompurify 3.4.13) and cheaper-inference#190 (nanoid 3.3.17). npm audit → 0.
_tasks is a SEPARATE nested git repo (gitignored). The pattern _tasks/ (trailing slash) ignores only a directory, not a SYMLINK named _tasks. A self-referential _tasks symlink can slip in via git add -A and, once pulled, checkout materializes it over the real _tasks repo (destroying plans/specs/hands-off). Anchored /_tasks ignores the symlink too, preventing re-capture.
…ference#10026) Mirror the request-time exclusion rule (provider_specific_data.excludedModels) in the unified catalog builder: a model is hidden when its provider has connections but none of them is eligible for it. Applied across the PROVIDER_MODELS, synced, custom, alias-backed, and managed-fallback loops so ghost models no longer appear as available. Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>
…per-inference#10055) * fix(models): memoize getModelsDevPricing for /v1/models catalog resolveCatalogPricing called getModelsDevPricing once per model while building GET /v1/models. Each call re-scanned models_dev_pricing and JSON.parsed every row (~10k SQL scans + multi-GB parse work), pegging the event loop so even /healthz timed out (cheaper-inference#9685, cheaper-inference#10052). Memoize the parsed map until saveModelsDevPricing / clearModelsDevPricing and add a unit test for invalidation. Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> * fix(db): invalidate modelsDevPricing cache on DB reset (cheaper-inference#10055) Copilot review fixes: 1. Register invalidateModelsDevPricingCache() with DB state reset system so resetDbInstance() clears the process-local memo, preventing stale pricing data from surviving across DB reset/restore operations. 2. Add test assertion verifying DB reset bypasses the memo (Copilot cheaper-inference#10055). The process-local memo at modelsDevSync.ts:204 caches getModelsDevPricing() results until saveModelsDevPricing()/clearModelsDevPricing() to avoid re-scanning all pricing rows on every /v1/models request. Without this hook, backup restore and test DB resets would serve stale cached data from the previous connection. Tests: npm run test:unit:serial -- tests/unit/modelsDevSync-extended.test.ts --------- Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|
hi @diegosouzapw , would be great if u can look on it , thanks! |
|
Correct, production-reproduced bug fix — |
75d4ddd to
40c1204
Compare
|
Retargeted to main and resolved the conflict. The branch was originally cut from the release lineage, so rather than rebase 13 commits I rebuilt it from main and cherry-picked the fix commit, which applied cleanly. The conflict was all release-branch history, not the change itself. Re-verified against main: enforce.ts there still applies the same accountCount multiply, the 4 quota test files pass (23/23), and eslint is clean on both changed files. Should be good to merge now. |
|
Correct, production-reproduced bug fix — |
c899d3f to
40c1204
Compare
|
Retargeted to Tests still pass on that tree: The old head One thing I want to raise before you merge, because it might change what you want here. |
…pool-usage-summed-budget Resolves drift conflicts in .gitignore, package.json/package-lock.json, and src/lib/modelsDevSync.ts (superseded process-local pricing memo, already replaced by release's version-tracked pricingMemo mechanism). The PR's own change (pool usage snapshot limits scaled by member count) merged cleanly. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
ac2439b
into
cheaper-inference:release/v3.8.50
…ed budget) (cheaper-inference#10253) * fix(deps): bump nanoid, dompurify for 2 new Dependabot alerts (cheaper-inference#189, cheaper-inference#190) Bumps: nanoid ^3.3.17 (was transitive, now overridden), dompurify ^3.4.13 (with monaco-editor scoped override). Closes Dependabot cheaper-inference#189, cheaper-inference#190. Remaining cheaper-inference#182-cheaper-inference#188 (js-yaml + mermaid) already closed by cheaper-inference#9651 merge — awaiting Dependabot re-scan. npm audit → 0 vulnerabilities. * fix(repo): harden .gitignore to also ignore a _tasks symlink (/_tasks) _tasks is a SEPARATE nested git repo (gitignored). The pattern _tasks/ (trailing slash) ignores only a directory, not a SYMLINK named _tasks. A self-referential _tasks symlink can slip in via git add -A and, once pulled, checkout materializes it over the real _tasks repo (destroying plans/specs/hands-off). Anchored /_tasks ignores the symlink too, preventing re-capture. * Hide health-check excluded models from /v1/models catalog (cheaper-inference#10026) Mirror the request-time exclusion rule (provider_specific_data.excludedModels) in the unified catalog builder: a model is hidden when its provider has connections but none of them is eligible for it. Applied across the PROVIDER_MODELS, synced, custom, alias-backed, and managed-fallback loops so ghost models no longer appear as available. Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com> * fix(models): memoize getModelsDevPricing (event loop / healthz) (cheaper-inference#10055) * fix(models): memoize getModelsDevPricing for /v1/models catalog resolveCatalogPricing called getModelsDevPricing once per model while building GET /v1/models. Each call re-scanned models_dev_pricing and JSON.parsed every row (~10k SQL scans + multi-GB parse work), pegging the event loop so even /healthz timed out (cheaper-inference#9685, cheaper-inference#10052). Memoize the parsed map until saveModelsDevPricing / clearModelsDevPricing and add a unit test for invalidation. Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> * fix(db): invalidate modelsDevPricing cache on DB reset (cheaper-inference#10055) Copilot review fixes: 1. Register invalidateModelsDevPricingCache() with DB state reset system so resetDbInstance() clears the process-local memo, preventing stale pricing data from surviving across DB reset/restore operations. 2. Add test assertion verifying DB reset bypasses the memo (Copilot cheaper-inference#10055). The process-local memo at modelsDevSync.ts:204 caches getModelsDevPricing() results until saveModelsDevPricing()/clearModelsDevPricing() to avoid re-scanning all pricing rows on every /v1/models request. Without this hook, backup restore and test DB resets would serve stale cached data from the previous connection. Tests: npm run test:unit:serial -- tests/unit/modelsDevSync-extended.test.ts --------- Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> * fix(api): scale pool usage snapshot limits by member count (summed budget) --------- Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com> Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@outlook.com> Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> Co-authored-by: ritheshcn25 <rithesh.chandran@snb.ca> Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com> Co-authored-by: Ravi Tharuma <25951435+RaviTharuma@users.noreply.github.com> Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Problem
GET /api/quota/pools/[id]/usagepasses the provider plan's per-account dimensions straight topoolUsageWithDimensions, whileenforce.tsmultiplies every dimension limit by the pool's member-connection count before deciding anything (the summed budget). For a multi-connection pool the snapshot therefore reports limits and fair shares N times smaller than what enforcement actually uses.I hit this in production today: a 27-connection pool with a
usd/dailyplan limit of 66.67 (enforced budget 1800.09) had consumed 54.09. The Quota Share page showed 81% utilisation; real utilisation was 3%. The per-keyborrowingflags have the same problem: they trip once a key crosses weight% of the per-account limit instead of weight% of the summed budget, so on that pool they fire 27x too early.Fix
Scale the resolved plan dimensions by
accountCountin the usage route before calling the store, using the same computation and legacy fallback asenforce.ts(connectionIds.length, falling back to 1 when the array is missing or empty). Snapshotlimit,fairShare,deficitandborrowingthen all describe the budget that is enforced. Single-connection and legacy pools are unchanged (N=1), and the{ usage: snapshot }wrapper contract is untouched.Scaling at the route level covers both store drivers, since sqlite and redis both receive their dimensions from this one caller.
Tests
Added
tests/unit/quota-pool-usage-summed-budget.test.ts, following the pattern ofquota-summed-budget.test.ts(which pins the same multiply on the enforcement side) andquota-pool-usage-shape.test.ts: structural assertions that the route applies the multiply with the right fallback, plus a logic-level check that the scaled limit correctsfairShareandborrowingfor the 27-connection case above.Ran locally:
npm run lint(scoped to the two changed files) is clean.