Skip to content

fix(cache): share one in-flight load across concurrent cache misses - #15101

Merged
diegosouzapw merged 3 commits into
diegosouzapw:release/v3.8.51from
cryptiklemur:fix/pricing-cache-single-flight
Sep 29, 2026
Merged

diegosouzapw merged 3 commits into
diegosouzapw:release/v3.8.51from
cryptiklemur:fix/pricing-cache-single-flight

Conversation

@cryptiklemur

Copy link
Copy Markdown
Contributor

Summary

  • TTLCache in src/lib/db/readCache.ts gets a load(key, loader) method. Callers that miss the same key at the same time now share one pending load instead of each starting their own.
  • every cached wrapper (settings, pricing, connections, raw connections, connection by id, nodes, LKGP) uses it.
  • invalidate() also drops the pending load, so a caller that arrives after an invalidation starts a fresh read and never gets the old one.
  • why: the nightly usage_history rollup (src/lib/usage/aggregateHistory.ts) prices thousands of groups in one Promise.all. it often runs right after a models.dev sync clears the pricing cache. every group missed, and each one loaded about 1.5 MB of pricing. that ran a 20 GB heap out of memory.

Related Issues

  • none filed. seen in production as an OOM during the nightly rollup.

Validation

  • Change type: DB
  • Focused tests and category gates from the golden path
  • npm run lint: no errors. it exits 2 only because release/v3.8.51 has 5 stale suppressions in unrelated files (openai-to-cursor.ts, SystemStorageTab.tsx, both vscode combos/route.ts, ghe-copilot.ts). the same happens on the base, and --pass-on-unpruned-suppressions exits 0.
  • Reconciled with the current active release base; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR
check result
new test on release/v3.8.51 without the fix 2 of 3 fail (150 and 600 pricing reads, expected 3)
new test with the fix 3 of 3 pass
db-read-cache, usage-history-*, combo-cache-invalidation, combo-delete-lkgp-cleanup-12326, db-synced-model-catalog-invalidation-8728 pass
eslint + prettier on touched files clean

Tests Added Or Updated

  • tests/unit/pricing-cache-single-flight.test.ts (new)

Coverage Notes

  • the new test covers 50 concurrent getCachedPricing() misses, an invalidation while a load is pending, and a full rollupUsageHistoryBeforeDate() over 200 distinct rows. the other wrappers share the same load() path.

Reviewer Notes

  • a failed load is not cached. the next caller retries, same as before.
  • usage-history-provider-alias-13459.test.ts fails about 1 run in 4 when run next to other DB tests. it does this on the base branch too, without this change.

@diegosouzapw
diegosouzapw merged commit 4796624 into diegosouzapw:release/v3.8.51 Sep 29, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants