Skip to content

fix(providers): classify cloudflare-ai daily neuron exhaustion as quota_exhausted - #6981

Closed
maxmad64bis wants to merge 1 commit into
diegosouzapw:release/v3.8.49from
maxmad64bis:fix/cloudflare-ai-quota-classification
Closed

maxmad64bis wants to merge 1 commit into
diegosouzapw:release/v3.8.49from
maxmad64bis:fix/cloudflare-ai-quota-classification

Conversation

@maxmad64bis

Copy link
Copy Markdown
Contributor

Fixes #6980

Summary

  • Registers a cloudflare-ai provider rule in providerErrorRules.ts matching the daily neuron-allocation exhaustion body, scoped "connection" (account-wide budget — same reasoning as the existing opencode rules).
  • Adds /daily free allocation/i to QUOTA_PATTERNS in classify429.ts as defense-in-depth for the classify429FromError path.
  • Adds a unit test (S2c) to tests/unit/provider-error-rules.test.ts following the existing S1/S2b pattern: matches the exhaustion body → quota_exhausted/scope:"connection"; a generic 429 without the exhaustion wording → no match.

Why

Cloudflare Workers AI's free-tier exhaustion body doesn't contain "quota"/"limit" (after "daily")/"exceed"/"credit"/"plan limit", so it currently falls through to the default rate_limit classification and gets retried on the short ~60s cooldown against a budget that only resets at UTC midnight. reason==="quota_exhausted" without an explicit cooldownMs already resolves to "next UTC midnight" via recordModelLockoutFailure (accountFallback.ts:553-556), so this rule just needs to reach that path — no new cooldown logic required.

Test plan

  • node --import tsx/esm --test tests/unit/provider-error-rules.test.ts — 6/6 pass (including new S2c)
  • eslint — clean
  • prettier --check — clean
  • Pre-commit hooks (docs-sync, t11 any-budget, tracked-artifacts) — pass

Prior art

Same bug class as #1767→#1772→#2100 (origin of classify429.ts). providerErrorRules.ts already establishes this exact pattern for opencode/minimax.

…ta_exhausted

Cloudflare Workers AI's free-tier exhaustion body ("you have used up your
daily free allocation of 10,000 neurons...") doesn't match any
QUOTA_PATTERNS or ERROR_RULES keyword, so it falls through to the default
rate_limit classification and gets retried on the short ~60s cooldown
against a budget that only resets at UTC midnight.

Registers a cloudflare-ai provider rule (scope: "connection", account-wide
budget — same reasoning as the existing opencode rules) in
providerErrorRules.ts, plus a defense-in-depth pattern in
classify429.ts's QUOTA_PATTERNS for the classify429FromError path.

Fixes #6980

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces error-handling rules and unit tests for Cloudflare Workers AI daily neuron budget exhaustion, mapping it to a connection-scoped quota exhaustion state. The reviewer identified critical issues where the rule would fail to match during connection cooldowns due to missing parameters in the calling function, and noted that the rule lacks a required cooldownMs property. The reviewer provided a code suggestion to calculate the cooldown until the next UTC midnight directly within the rule.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +119 to +134
function buildCloudflareAiRules(): ProviderErrorRule[] {
return [
{
id: "cloudflare-ai-daily-neuron-allocation",
match: ({ status, body }) => {
if (status !== 429) return null;
const text = JSON.stringify(body ?? "").toLowerCase();
if (!text.includes("daily free allocation")) return null;
// No cooldownMs: recordModelLockoutFailure already sets
// quota_exhausted without one to "next UTC midnight" — exactly this
// budget's real reset semantics.
return { reason: "quota_exhausted", scope: "connection" };
},
},
];
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There are two critical issues with the current implementation that will prevent the daily neuron exhaustion from being correctly cooled down in production:

1. applyErrorState does not pass structuredError or headers to checkFallbackError

In open-sse/services/accountFallback.ts, applyErrorState is responsible for applying the error state and cooldown to the connection/account. It calls checkFallbackError with only 5 arguments:

const fallbackDecision = checkFallbackError(status, errorText, backoffLevel, null, provider);

Because headers and structuredError are omitted, they default to null/undefined. Consequently, getProviderErrorRuleMatch is called with body = null and headers = null. Thus, this cloudflare-ai-daily-neuron-allocation rule (which matches on body) will never match during the connection cooldown phase, and the connection will fall back to a standard 5-second rate-limit cooldown.

2. Missing cooldownMs in the rule

Even if the rule did match, it does not return a cooldownMs. While recordModelLockoutFailure has fallback logic to set quota_exhausted to "next UTC midnight", applyErrorState (which handles connection-level lockouts) does not. It relies entirely on the cooldownMs returned by checkFallbackError. If no cooldownMs is specified by the provider rule, it falls back to the default scaled backoff cooldown (e.g., 5 seconds).

Recommended Fixes:

  1. Return cooldownMs directly from the rule: Update the rule to calculate and return the cooldownMs (time until next UTC midnight) directly (as suggested below).
  2. Update isDailyQuotaExhausted in accountFallback.ts: Since accountFallback.ts is not in the diff, please also update isDailyQuotaExhausted in that file to include "daily free allocation". This ensures that checkFallbackError natively recognizes the daily quota exhaustion and applies the correct midnight cooldown even when body is not passed:
export function isDailyQuotaExhausted(errorText: string): boolean {
  if (!errorText) return false;
  const lower = errorText.toLowerCase();
  return (
    lower.includes("today's quota") ||
    lower.includes("daily quota") ||
    lower.includes("try again tomorrow") ||
    lower.includes("daily free allocation")
  );
}
Suggested change
function buildCloudflareAiRules(): ProviderErrorRule[] {
return [
{
id: "cloudflare-ai-daily-neuron-allocation",
match: ({ status, body }) => {
if (status !== 429) return null;
const text = JSON.stringify(body ?? "").toLowerCase();
if (!text.includes("daily free allocation")) return null;
// No cooldownMs: recordModelLockoutFailure already sets
// quota_exhausted without one to "next UTC midnight" — exactly this
// budget's real reset semantics.
return { reason: "quota_exhausted", scope: "connection" };
},
},
];
}
function buildCloudflareAiRules(): ProviderErrorRule[] {
return [
{
id: "cloudflare-ai-daily-neuron-allocation",
match: ({ status, body }) => {
if (status !== 429) return null;
const text = JSON.stringify(body ?? "").toLowerCase();
if (!text.includes("daily free allocation")) return null;
// Calculate ms until next UTC midnight to precisely match Cloudflare's reset window
const now = Date.now();
const nextMidnight = new Date(now);
nextMidnight.setUTCHours(24, 0, 0, 0);
const cooldownMs = nextMidnight.getTime() - now;
return { reason: "quota_exhausted", scope: "connection", cooldownMs };
},
},
];
}

@diegosouzapw

Copy link
Copy Markdown
Owner

Thanks for this fix and for following the repo's existing S1/S2/S2b test convention in provider-error-rules.test.ts — that's the right instinct architecturally. Two other contributors (rafaumeu #6983, xier2012 #7165) submitted the same underlying fix; we're merging #6983 since its dedicated test file covers more edge cases (case-insensitivity, explicit cooldownMs semantics, JSON vs string body variants, both the provider-rule and classify429 layers separately). Closing this one as superseded, with credit for the correct diagnosis and fix of #6980.

@diegosouzapw

Copy link
Copy Markdown
Owner

Thanks @maxmad64bis for digging into the Cloudflare AI neuron-exhaustion classification! This exact fix landed today via #6983 (same cluster, same providerErrorRules/classify429 path), so closing this one as superseded. Your report and analysis helped confirm the direction — if you spot a gap the merged fix doesn't cover, please reopen with a pointer at the specific case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(providers): cloudflare-ai daily neuron exhaustion misclassified as transient rate_limit

2 participants