Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
225 changes: 225 additions & 0 deletions src/lib/quota/accountBuckets.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
/**
* accountBuckets.ts — Saturating per-connection, per-window buckets.
*
* Tracks whether a connection has hit 100 % of a quota window (5h / 7d /
* per-model 7d). Each bucket is lazily reset: on read, if now >= resetsAtMs
* the entry is cleared and the connection is eligible again. No cron needed —
* the reset is probed on the read path.
*
* Fail-open: missing entry → not saturated. All time input is injectable
* (the `nowMs` param) so unit tests drive the clock deterministically — the
* tested path never calls Date.now() implicitly.
*
* Complementary to connectionRecovery.ts: that module recovers DB-backed
* request-error cooldowns (testStatus 'unavailable' + rateLimitedUntil); this
* module tracks in-process plan-window utilization. Orthogonal concerns.
*
* Part of: Quota Sharing Engine — Phase 3 (#3 multi-window buckets).
*/

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

/** In-process state for one (connectionId, windowKey) pair. */
interface BucketEntry {
saturated: boolean;
resetsAtMs: number; // 0 when the reset instant is unknown
}

/**
* Minimal shape of a parsed UsageQuota entry, as produced by getClaudeUsage in
* open-sse/services/usage.ts. `used` = % consumed (0..100); `total` = 100 for
* percent-based windows; `resetAt` = ISO 8601 string or null (already
* normalized upstream by parseResetTime).
*/
export interface UsageQuotaSlim {
used: number;
total: number;
resetAt: string | null;
}

/**
* Partial shape of the getClaudeUsage() return value this module needs. Only
* `quotas` is consumed; other fields (plan, extraUsage, bootstrap) are ignored.
*/
export interface ClaudeUsageResult {
quotas?: Record<string, UsageQuotaSlim | undefined>;
}

// ---------------------------------------------------------------------------
// Constants
// ---------------------------------------------------------------------------

/**
* Utilization threshold (0..100) at/above which a window is considered
* saturated. 100 = exhausted. Named so it can be tuned globally if Anthropic
* ever soft-throttles below the hard cap.
*/
export const SATURATION_THRESHOLD_PCT = 100;

// ---------------------------------------------------------------------------
// In-process store
// ---------------------------------------------------------------------------

/** Key: `${connectionId}::${windowKey}`. */
const _buckets = new Map<string, BucketEntry>();

function storeKey(connectionId: string, windowKey: string): string {
return `${connectionId}::${windowKey}`;
}
Comment on lines +65 to +70

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Memory Leak & Stale Saturation Issues

Using a flat Map with a string key (${connectionId}::${windowKey}) introduces two significant issues:

  1. Memory Leak: Every time a new connectionId is used, entries are added to _buckets. If a connection is deleted or becomes inactive, its entries are never removed from _buckets because there is no background cleanup or TTL sweep. Over time, this will cause unbounded memory growth.
  2. Stale Model Saturation: In updateAccountBuckets, we only process the quotas returned in the current usageResult.quotas payload. If a model (e.g., 7d:opus) was previously saturated, but is omitted from the new quotas payload (which can happen if the model is no longer used or returned by the upstream API), processQuotaEntry is never called for it. If its resetsAtMs was 0 (unknown reset time), it will remain saturated in _buckets forever!

Solution

Refactor _buckets to use a nested Map structure: Map<string, Map<string, BucketEntry>> where the outer key is connectionId and the inner key is windowKey. This allows us to:

  • Delete the inner Map for a connectionId once it becomes empty (connBuckets.size === 0), preventing memory leaks.
  • Easily track which keys were updated in the current payload, and safely delete any other keys for that connection that were not present in the update.
const _buckets = new Map<string, Map<string, BucketEntry>>();

function getOrCreateConnectionBuckets(connectionId: string): Map<string, BucketEntry> {
  let connBuckets = _buckets.get(connectionId);
  if (!connBuckets) {
    connBuckets = new Map<string, BucketEntry>();
    _buckets.set(connectionId, connBuckets);
  }
  return connBuckets;
}


/**
* Parse an ISO 8601 `resetAt` string to epoch ms. Returns 0 on any failure
* (unknown reset time → the lazy reset cannot fire for that bucket).
*/
function parseResetAtMs(resetAt: string | null | undefined): number {
if (!resetAt) return 0;
const ms = Date.parse(resetAt);
return Number.isFinite(ms) && ms > 0 ? ms : 0;
}

// ---------------------------------------------------------------------------
// Public API
// ---------------------------------------------------------------------------

/**
* Check whether the bucket for (connectionId, windowKey) is currently
* saturated.
*
* Lazy reset: if `nowMs >= entry.resetsAtMs` (and resetsAtMs > 0) the entry is
* cleared and `false` is returned — the connection is eligible again without
* any background sweep.
*
* Fail-open: a missing entry returns false (not saturated).
*
* @param connectionId Opaque string identifier for the connection.
* @param windowKey One of: "5h", "7d", "7d:<modelName>" (e.g. "7d:designer").
* @param nowMs Current epoch ms; defaults to Date.now() (off-path only).
*/
export function isBucketSaturated(
connectionId: string,
windowKey: string,
nowMs: number = Date.now()
): boolean {
if (!connectionId || !windowKey) return false; // fail-open
const key = storeKey(connectionId, windowKey);
const entry = _buckets.get(key);
if (!entry) return false; // fail-open

// Lazy reset: the window rolled over → the saturation is stale.
if (entry.resetsAtMs > 0 && nowMs >= entry.resetsAtMs) {
_buckets.delete(key);
return false;
}

return entry.saturated;
}
Comment on lines +100 to +117

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Update isBucketSaturated to use the nested Map structure. If the inner Map becomes empty after a lazy reset, delete the outer connection entry to prevent memory leaks.

export function isBucketSaturated(
  connectionId: string,
  windowKey: string,
  nowMs: number = Date.now()
): boolean {
  if (!connectionId || !windowKey) return false; // fail-open
  const connBuckets = _buckets.get(connectionId);
  if (!connBuckets) return false; // fail-open
  const entry = connBuckets.get(windowKey);
  if (!entry) return false; // fail-open

  // Lazy reset: the window rolled over → the saturation is stale.
  if (entry.resetsAtMs > 0 && nowMs >= entry.resetsAtMs) {
    connBuckets.delete(windowKey);
    if (connBuckets.size === 0) {
      _buckets.delete(connectionId);
    }
    return false;
  }

  return entry.saturated;
}


/**
* Record a usage observation for one (connectionId, windowKey) pair.
*
* Marks the bucket saturated when `usedPct >= SATURATION_THRESHOLD_PCT` and the
* window has NOT already rolled over. When the observation is below the
* threshold (or already past its reset), any existing entry is cleared so the
* connection's eligibility is restored promptly. When `resetAt` is a valid ISO
* string, the reset epoch is stored so lazy reset can fire later.
*
* @param connectionId Opaque connection identifier.
* @param windowKey "5h", "7d", or "7d:<modelName>".
* @param usedPct Utilization percentage (0..100).
* @param resetAt ISO 8601 string (or null) for when this window resets.
* @param nowMs Current epoch ms; defaults to Date.now() (off-path only).
*/
export function recordUsage(
connectionId: string,
windowKey: string,
usedPct: number,
resetAt: string | null,
nowMs: number = Date.now()
): void {
if (!connectionId || !windowKey) return;
const key = storeKey(connectionId, windowKey);

const resetsAtMs = parseResetAtMs(resetAt);

// Stale signal: the window already reset — discard any state and bail.
if (resetsAtMs > 0 && nowMs >= resetsAtMs) {
_buckets.delete(key);
return;
}

const saturated = Number.isFinite(usedPct) && usedPct >= SATURATION_THRESHOLD_PCT;
if (!saturated) {
// Below threshold → clear any stale saturation so the bucket is eligible.
_buckets.delete(key);
return;
}

_buckets.set(key, { saturated: true, resetsAtMs });
}
Comment on lines +134 to +160

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Update recordUsage to use the nested Map structure. Ensure that we clean up the outer connection entry if the inner Map becomes empty after deleting a key.

export function recordUsage(
  connectionId: string,
  windowKey: string,
  usedPct: number,
  resetAt: string | null,
  nowMs: number = Date.now()
): void {
  if (!connectionId || !windowKey) return;
  const connBuckets = getOrCreateConnectionBuckets(connectionId);

  const resetsAtMs = parseResetAtMs(resetAt);

  // Stale signal: the window already reset — discard any state and bail.
  if (resetsAtMs > 0 && nowMs >= resetsAtMs) {
    connBuckets.delete(windowKey);
    if (connBuckets.size === 0) {
      _buckets.delete(connectionId);
    }
    return;
  }

  const saturated = Number.isFinite(usedPct) && usedPct >= SATURATION_THRESHOLD_PCT;
  if (!saturated) {
    // Below threshold → clear any stale saturation so the bucket is eligible.
    connBuckets.delete(windowKey);
    if (connBuckets.size === 0) {
      _buckets.delete(connectionId);
    }
    return;
  }

  connBuckets.set(windowKey, { saturated: true, resetsAtMs });
}


/**
* Parse the `quotas` map from a getClaudeUsage() result and record each known
* window into its bucket.
*
* Window key mapping:
* "session (5h)" → "5h"
* "weekly (7d)" → "7d"
* "weekly <model> (7d)" → "7d:<model>" (e.g. "weekly designer (7d)" → "7d:designer")
*
* Fail-open: a null/undefined result, a missing `quotas` map, or any malformed
* quota entry is silently skipped.
*
* @param connectionId Connection identifier.
* @param usageResult getClaudeUsage() return value (only `quotas` is read).
* @param nowMs Current epoch ms; defaults to Date.now() (off-path only).
*/
export function updateAccountBuckets(
connectionId: string,
usageResult: ClaudeUsageResult | null | undefined,
nowMs: number = Date.now()
): void {
if (!connectionId || !usageResult?.quotas) return;
const { quotas } = usageResult;

// Fixed windows.
processQuotaEntry(connectionId, "5h", quotas["session (5h)"], nowMs);
processQuotaEntry(connectionId, "7d", quotas["weekly (7d)"], nowMs);

// Per-model weekly windows: "weekly <model> (7d)" → "7d:<model>".
for (const [key, entry] of Object.entries(quotas)) {
const match = /^weekly (.+) \(7d\)$/.exec(key);
if (match?.[1]) {
processQuotaEntry(connectionId, `7d:${match[1]}`, entry, nowMs);
}
}
}
Comment on lines +178 to +197

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Update updateAccountBuckets to use the nested Map structure. Track which keys are updated in the current payload, and clear any other keys for this connection that were not present in the update. This prevents stale model saturation from persisting indefinitely.

export function updateAccountBuckets(
  connectionId: string,
  usageResult: ClaudeUsageResult | null | undefined,
  nowMs: number = Date.now()
): void {
  if (!connectionId || !usageResult?.quotas) return;
  const { quotas } = usageResult;

  const activeKeys = new Set<string>();

  // Fixed windows.
  if (quotas["session (5h)"]) {
    processQuotaEntry(connectionId, "5h", quotas["session (5h)"], nowMs);
    activeKeys.add("5h");
  }
  if (quotas["weekly (7d)"]) {
    processQuotaEntry(connectionId, "7d", quotas["weekly (7d)"], nowMs);
    activeKeys.add("7d");
  }

  // Per-model weekly windows: "weekly <model> (7d)" → "7d:<model>".
  for (const [key, entry] of Object.entries(quotas)) {
    const match = /^weekly (.+) \(7d\)$/.exec(key);
    if (match?.[1]) {
      const windowKey = `7d:${match[1]}`;
      processQuotaEntry(connectionId, windowKey, entry, nowMs);
      activeKeys.add(windowKey);
    }
  }

  // Clear any other keys for this connection that were NOT in the current update.
  const connBuckets = _buckets.get(connectionId);
  if (connBuckets) {
    for (const key of connBuckets.keys()) {
      if (!activeKeys.has(key)) {
        connBuckets.delete(key);
      }
    }
    if (connBuckets.size === 0) {
      _buckets.delete(connectionId);
    }
  }
}


// ---------------------------------------------------------------------------
// Internal helpers
// ---------------------------------------------------------------------------

function processQuotaEntry(
connectionId: string,
windowKey: string,
entry: UsageQuotaSlim | undefined | null,
nowMs: number
): void {
if (!entry || typeof entry.used !== "number") return;
recordUsage(connectionId, windowKey, entry.used, entry.resetAt ?? null, nowMs);
}

// ---------------------------------------------------------------------------
// Test helpers (never call in production code)
// ---------------------------------------------------------------------------

/** Clear all bucket entries. Tests only — keeps state isolation between cases. */
export function _clearBucketsForTest(): void {
_buckets.clear();
}

/** Return the current bucket count. Tests only — black-box size assertion. */
export function _bucketCountForTest(): number {
return _buckets.size;
}
Comment on lines +222 to +225

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Update _bucketCountForTest to sum the sizes of the inner Maps, keeping the test assertions fully compatible with the nested Map structure.

Suggested change
/** Return the current bucket count. Tests only — black-box size assertion. */
export function _bucketCountForTest(): number {
return _buckets.size;
}
/** Return the current bucket count. Tests only — black-box size assertion. */
export function _bucketCountForTest(): number {
let count = 0;
for (const connBuckets of _buckets.values()) {
count += connBuckets.size;
}
return count;
}

9 changes: 9 additions & 0 deletions src/lib/quota/saturationSignals.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
*/

import { createLogger } from "@/shared/utils/logger";
import { updateAccountBuckets, type ClaudeUsageResult } from "./accountBuckets";
import type { QuotaUnit, QuotaWindow } from "./dimensions";

const log = createLogger("quota:saturation");
Expand Down Expand Up @@ -434,6 +435,14 @@ async function fetchAnthropicSaturation(
(conn.authType === undefined || conn.authType === "oauth");
if (hasOauthToken) {
const usage = await deps.fetchUsage(conn as Record<string, unknown>);
// Update the per-window saturating buckets (Phase 3 #3) off the request
// hot path — this runs behind the 30s saturation cache. Fail-open: any
// bucket error must never affect the primary 0..1 saturation signal.
try {
updateAccountBuckets(connectionId, usage as ClaudeUsageResult, Date.now());
} catch {
// intentionally swallowed — buckets are additive, never gate-breaking
}
const util = planUtilizationFromUsage(usage, dim.window);
if (util !== null) return util;
}
Expand Down
Loading
Loading