Skip to content

fix(api): scale pool usage snapshot limits by pool member count (summed budget) - #10253

Merged
diegosouzapw merged 7 commits into
cheaper-inference:release/v3.8.50from
dpozimski:fix/quota-pool-usage-summed-budget
Aug 18, 2026
Merged

diegosouzapw merged 7 commits into
cheaper-inference:release/v3.8.50from
dpozimski:fix/quota-pool-usage-summed-budget

Conversation

@dpozimski

Copy link
Copy Markdown
Contributor

Problem

GET /api/quota/pools/[id]/usage passes the provider plan's per-account dimensions straight to poolUsageWithDimensions, while enforce.ts multiplies every dimension limit by the pool's member-connection count before deciding anything (the summed budget). For a multi-connection pool the snapshot therefore reports limits and fair shares N times smaller than what enforcement actually uses.

I hit this in production today: a 27-connection pool with a usd/daily plan limit of 66.67 (enforced budget 1800.09) had consumed 54.09. The Quota Share page showed 81% utilisation; real utilisation was 3%. The per-key borrowing flags have the same problem: they trip once a key crosses weight% of the per-account limit instead of weight% of the summed budget, so on that pool they fire 27x too early.

Fix

Scale the resolved plan dimensions by accountCount in the usage route before calling the store, using the same computation and legacy fallback as enforce.ts (connectionIds.length, falling back to 1 when the array is missing or empty). Snapshot limit, fairShare, deficit and borrowing then all describe the budget that is enforced. Single-connection and legacy pools are unchanged (N=1), and the { usage: snapshot } wrapper contract is untouched.

Scaling at the route level covers both store drivers, since sqlite and redis both receive their dimensions from this one caller.

Tests

Added tests/unit/quota-pool-usage-summed-budget.test.ts, following the pattern of quota-summed-budget.test.ts (which pins the same multiply on the enforcement side) and quota-pool-usage-shape.test.ts: structural assertions that the route applies the multiply with the right fallback, plus a logic-level check that the scaled limit corrects fairShare and borrowing for the 27-connection case above.

Ran locally:

node --import tsx/esm --test tests/unit/quota-pool-usage-summed-budget.test.ts \
  tests/unit/quota-pool-usage-shape.test.ts \
  tests/unit/quota-summed-budget.test.ts \
  tests/unit/quota-sharing-fixes.test.ts
# 23 pass, 0 fail

npm run lint (scoped to the two changed files) is clean.

diegosouzapw and others added 5 commits August 8, 2026 00:08
…r-inference#189, cheaper-inference#190)

Bumps: nanoid ^3.3.17 (was transitive, now overridden), dompurify ^3.4.13
(with monaco-editor scoped override). Closes Dependabot cheaper-inference#189, cheaper-inference#190.

Remaining cheaper-inference#182-cheaper-inference#188 (js-yaml + mermaid) already closed by cheaper-inference#9651 merge —
awaiting Dependabot re-scan.

npm audit → 0 vulnerabilities.
_tasks is a SEPARATE nested git repo (gitignored). The pattern _tasks/ (trailing
slash) ignores only a directory, not a SYMLINK named _tasks. A self-referential
_tasks symlink can slip in via git add -A and, once pulled, checkout materializes
it over the real _tasks repo (destroying plans/specs/hands-off). Anchored /_tasks
ignores the symlink too, preventing re-capture.
…ference#10026)

Mirror the request-time exclusion rule (provider_specific_data.excludedModels)
in the unified catalog builder: a model is hidden when its provider has
connections but none of them is eligible for it. Applied across the
PROVIDER_MODELS, synced, custom, alias-backed, and managed-fallback loops
so ghost models no longer appear as available.

Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>
…per-inference#10055)

* fix(models): memoize getModelsDevPricing for /v1/models catalog

resolveCatalogPricing called getModelsDevPricing once per model while
building GET /v1/models. Each call re-scanned models_dev_pricing and
JSON.parsed every row (~10k SQL scans + multi-GB parse work), pegging
the event loop so even /healthz timed out (cheaper-inference#9685, cheaper-inference#10052).

Memoize the parsed map until saveModelsDevPricing / clearModelsDevPricing
and add a unit test for invalidation.

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

* fix(db): invalidate modelsDevPricing cache on DB reset (cheaper-inference#10055)

Copilot review fixes:
1. Register invalidateModelsDevPricingCache() with DB state reset system
   so resetDbInstance() clears the process-local memo, preventing stale
   pricing data from surviving across DB reset/restore operations.
2. Add test assertion verifying DB reset bypasses the memo (Copilot cheaper-inference#10055).

The process-local memo at modelsDevSync.ts:204 caches getModelsDevPricing()
results until saveModelsDevPricing()/clearModelsDevPricing() to avoid
re-scanning all pricing rows on every /v1/models request. Without this hook,
backup restore and test DB resets would serve stale cached data from the
previous connection.

Tests: npm run test:unit:serial -- tests/unit/modelsDevSync-extended.test.ts

---------

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
@dpozimski

Copy link
Copy Markdown
Contributor Author

hi @diegosouzapw , would be great if u can look on it , thanks!

@diegosouzapw
diegosouzapw changed the base branch from release/v3.8.49 to release/v3.8.50 August 13, 2026 13:49
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Correct, production-reproduced bug fix — GET /api/quota/pools/[id]/usage now multiplies each plan dimension limit by accountCount. However, the PR targets the dead release/v3.8.49 and is BLOCKED. Please retarget to main and resolve the merge conflict — then it's good to merge.

@dpozimski
dpozimski force-pushed the fix/quota-pool-usage-summed-budget branch from 75d4ddd to 40c1204 Compare August 13, 2026 15:39
@dpozimski
dpozimski changed the base branch from release/v3.8.50 to main August 13, 2026 15:39
@dpozimski

Copy link
Copy Markdown
Contributor Author

Retargeted to main and resolved the conflict. The branch was originally cut from the release lineage, so rather than rebase 13 commits I rebuilt it from main and cherry-picked the fix commit, which applied cleanly. The conflict was all release-branch history, not the change itself. Re-verified against main: enforce.ts there still applies the same accountCount multiply, the 4 quota test files pass (23/23), and eslint is clean on both changed files. Should be good to merge now.

@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.8.50 August 15, 2026 11:37
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Correct, production-reproduced bug fix — GET /api/quota/pools/[id]/usage now multiplies each plan dimension limit by accountCount. However, the PR targets the dead release/v3.8.50 and is BLOCKED. Please retarget to main and resolve the merge conflict — then it's good to merge.

@dpozimski
dpozimski force-pushed the fix/quota-pool-usage-summed-budget branch from c899d3f to 40c1204 Compare August 17, 2026 06:21
@dpozimski
dpozimski changed the base branch from release/v3.8.50 to main August 17, 2026 06:22
@dpozimski

Copy link
Copy Markdown
Contributor Author

Retargeted to main and force-pushed. Head is now 40c1204b9: one commit, the same two files, merges into main clean. There wasn't a conflict to resolve, main is the base this commit was written on in the first place.

Tests still pass on that tree:

node --import tsx/esm --test tests/unit/quota-pool-usage-summed-budget.test.ts \
  tests/unit/quota-pool-usage-shape.test.ts \
  tests/unit/quota-summed-budget.test.ts \
  tests/unit/quota-sharing-fixes.test.ts
# 23 pass, 0 fail

The old head c899d3f93 was a merge of this same commit into release/v3.8.50, and both changed files were byte-identical across the two, so the rewrite didn't drop anything. It's parked on my fork as backup/quota-pool-usage-summed-budget-merge if you want it back.

One thing I want to raise before you merge, because it might change what you want here. release/v3.8.50 still looks like the branch everything lands on: it's the repo's default branch, the last 30 merged PRs all targeted it (#10520 went in yesterday), and 29 of the 30 currently open PRs target it. main hasn't moved since Aug 13 and is 2282 commits behind. Merging this into main fixes main, but the 27-connection pool I hit is served off the release line, so it keeps reporting 81% utilisation for a pool that's actually at 3%. Happy to open a second PR against release/v3.8.50 (the backup commit is exactly that, and it merges clean), or you can cherry-pick once this lands. Whichever you prefer.

@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.8.50 August 17, 2026 08:46
…pool-usage-summed-budget

Resolves drift conflicts in .gitignore, package.json/package-lock.json,
and src/lib/modelsDevSync.ts (superseded process-local pricing memo,
already replaced by release's version-tracked pricingMemo mechanism).
The PR's own change (pool usage snapshot limits scaled by member count)
merged cleanly.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@diegosouzapw
diegosouzapw merged commit ac2439b into cheaper-inference:release/v3.8.50 Aug 18, 2026
3 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ed budget) (cheaper-inference#10253)

* fix(deps): bump nanoid, dompurify for 2 new Dependabot alerts (cheaper-inference#189, cheaper-inference#190)

Bumps: nanoid ^3.3.17 (was transitive, now overridden), dompurify ^3.4.13
(with monaco-editor scoped override). Closes Dependabot cheaper-inference#189, cheaper-inference#190.

Remaining cheaper-inference#182-cheaper-inference#188 (js-yaml + mermaid) already closed by cheaper-inference#9651 merge —
awaiting Dependabot re-scan.

npm audit → 0 vulnerabilities.

* fix(repo): harden .gitignore to also ignore a _tasks symlink (/_tasks)

_tasks is a SEPARATE nested git repo (gitignored). The pattern _tasks/ (trailing
slash) ignores only a directory, not a SYMLINK named _tasks. A self-referential
_tasks symlink can slip in via git add -A and, once pulled, checkout materializes
it over the real _tasks repo (destroying plans/specs/hands-off). Anchored /_tasks
ignores the symlink too, preventing re-capture.

* Hide health-check excluded models from /v1/models catalog (cheaper-inference#10026)

Mirror the request-time exclusion rule (provider_specific_data.excludedModels)
in the unified catalog builder: a model is hidden when its provider has
connections but none of them is eligible for it. Applied across the
PROVIDER_MODELS, synced, custom, alias-backed, and managed-fallback loops
so ghost models no longer appear as available.

Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>

* fix(models): memoize getModelsDevPricing (event loop / healthz) (cheaper-inference#10055)

* fix(models): memoize getModelsDevPricing for /v1/models catalog

resolveCatalogPricing called getModelsDevPricing once per model while
building GET /v1/models. Each call re-scanned models_dev_pricing and
JSON.parsed every row (~10k SQL scans + multi-GB parse work), pegging
the event loop so even /healthz timed out (cheaper-inference#9685, cheaper-inference#10052).

Memoize the parsed map until saveModelsDevPricing / clearModelsDevPricing
and add a unit test for invalidation.

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

* fix(db): invalidate modelsDevPricing cache on DB reset (cheaper-inference#10055)

Copilot review fixes:
1. Register invalidateModelsDevPricingCache() with DB state reset system
   so resetDbInstance() clears the process-local memo, preventing stale
   pricing data from surviving across DB reset/restore operations.
2. Add test assertion verifying DB reset bypasses the memo (Copilot cheaper-inference#10055).

The process-local memo at modelsDevSync.ts:204 caches getModelsDevPricing()
results until saveModelsDevPricing()/clearModelsDevPricing() to avoid
re-scanning all pricing rows on every /v1/models request. Without this hook,
backup restore and test DB resets would serve stale cached data from the
previous connection.

Tests: npm run test:unit:serial -- tests/unit/modelsDevSync-extended.test.ts

---------

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix(api): scale pool usage snapshot limits by member count (summed budget)

---------

Signed-off-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <diegosouzapw@users.noreply.github.com>
Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@outlook.com>
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Co-authored-by: ritheshcn25 <rithesh.chandran@snb.ca>
Co-authored-by: ritheshcn25 <ritheshcn25@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <25951435+RaviTharuma@users.noreply.github.com>
Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants