fix(providers): classify cloudflare-ai daily neuron exhaustion as quota_exhausted - #6981
maxmad64bis wants to merge 1 commit into
Conversation
…ta_exhausted
Cloudflare Workers AI's free-tier exhaustion body ("you have used up your
daily free allocation of 10,000 neurons...") doesn't match any
QUOTA_PATTERNS or ERROR_RULES keyword, so it falls through to the default
rate_limit classification and gets retried on the short ~60s cooldown
against a budget that only resets at UTC midnight.
Registers a cloudflare-ai provider rule (scope: "connection", account-wide
budget — same reasoning as the existing opencode rules) in
providerErrorRules.ts, plus a defense-in-depth pattern in
classify429.ts's QUOTA_PATTERNS for the classify429FromError path.
Fixes #6980
There was a problem hiding this comment.
Code Review
This pull request introduces error-handling rules and unit tests for Cloudflare Workers AI daily neuron budget exhaustion, mapping it to a connection-scoped quota exhaustion state. The reviewer identified critical issues where the rule would fail to match during connection cooldowns due to missing parameters in the calling function, and noted that the rule lacks a required cooldownMs property. The reviewer provided a code suggestion to calculate the cooldown until the next UTC midnight directly within the rule.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| function buildCloudflareAiRules(): ProviderErrorRule[] { | ||
| return [ | ||
| { | ||
| id: "cloudflare-ai-daily-neuron-allocation", | ||
| match: ({ status, body }) => { | ||
| if (status !== 429) return null; | ||
| const text = JSON.stringify(body ?? "").toLowerCase(); | ||
| if (!text.includes("daily free allocation")) return null; | ||
| // No cooldownMs: recordModelLockoutFailure already sets | ||
| // quota_exhausted without one to "next UTC midnight" — exactly this | ||
| // budget's real reset semantics. | ||
| return { reason: "quota_exhausted", scope: "connection" }; | ||
| }, | ||
| }, | ||
| ]; | ||
| } |
There was a problem hiding this comment.
There are two critical issues with the current implementation that will prevent the daily neuron exhaustion from being correctly cooled down in production:
1. applyErrorState does not pass structuredError or headers to checkFallbackError
In open-sse/services/accountFallback.ts, applyErrorState is responsible for applying the error state and cooldown to the connection/account. It calls checkFallbackError with only 5 arguments:
const fallbackDecision = checkFallbackError(status, errorText, backoffLevel, null, provider);Because headers and structuredError are omitted, they default to null/undefined. Consequently, getProviderErrorRuleMatch is called with body = null and headers = null. Thus, this cloudflare-ai-daily-neuron-allocation rule (which matches on body) will never match during the connection cooldown phase, and the connection will fall back to a standard 5-second rate-limit cooldown.
2. Missing cooldownMs in the rule
Even if the rule did match, it does not return a cooldownMs. While recordModelLockoutFailure has fallback logic to set quota_exhausted to "next UTC midnight", applyErrorState (which handles connection-level lockouts) does not. It relies entirely on the cooldownMs returned by checkFallbackError. If no cooldownMs is specified by the provider rule, it falls back to the default scaled backoff cooldown (e.g., 5 seconds).
Recommended Fixes:
- Return
cooldownMsdirectly from the rule: Update the rule to calculate and return thecooldownMs(time until next UTC midnight) directly (as suggested below). - Update
isDailyQuotaExhaustedinaccountFallback.ts: SinceaccountFallback.tsis not in the diff, please also updateisDailyQuotaExhaustedin that file to include"daily free allocation". This ensures thatcheckFallbackErrornatively recognizes the daily quota exhaustion and applies the correct midnight cooldown even whenbodyis not passed:
export function isDailyQuotaExhausted(errorText: string): boolean {
if (!errorText) return false;
const lower = errorText.toLowerCase();
return (
lower.includes("today's quota") ||
lower.includes("daily quota") ||
lower.includes("try again tomorrow") ||
lower.includes("daily free allocation")
);
}| function buildCloudflareAiRules(): ProviderErrorRule[] { | |
| return [ | |
| { | |
| id: "cloudflare-ai-daily-neuron-allocation", | |
| match: ({ status, body }) => { | |
| if (status !== 429) return null; | |
| const text = JSON.stringify(body ?? "").toLowerCase(); | |
| if (!text.includes("daily free allocation")) return null; | |
| // No cooldownMs: recordModelLockoutFailure already sets | |
| // quota_exhausted without one to "next UTC midnight" — exactly this | |
| // budget's real reset semantics. | |
| return { reason: "quota_exhausted", scope: "connection" }; | |
| }, | |
| }, | |
| ]; | |
| } | |
| function buildCloudflareAiRules(): ProviderErrorRule[] { | |
| return [ | |
| { | |
| id: "cloudflare-ai-daily-neuron-allocation", | |
| match: ({ status, body }) => { | |
| if (status !== 429) return null; | |
| const text = JSON.stringify(body ?? "").toLowerCase(); | |
| if (!text.includes("daily free allocation")) return null; | |
| // Calculate ms until next UTC midnight to precisely match Cloudflare's reset window | |
| const now = Date.now(); | |
| const nextMidnight = new Date(now); | |
| nextMidnight.setUTCHours(24, 0, 0, 0); | |
| const cooldownMs = nextMidnight.getTime() - now; | |
| return { reason: "quota_exhausted", scope: "connection", cooldownMs }; | |
| }, | |
| }, | |
| ]; | |
| } |
|
Thanks for this fix and for following the repo's existing S1/S2/S2b test convention in provider-error-rules.test.ts — that's the right instinct architecturally. Two other contributors (rafaumeu #6983, xier2012 #7165) submitted the same underlying fix; we're merging #6983 since its dedicated test file covers more edge cases (case-insensitivity, explicit cooldownMs semantics, JSON vs string body variants, both the provider-rule and classify429 layers separately). Closing this one as superseded, with credit for the correct diagnosis and fix of #6980. |
|
Thanks @maxmad64bis for digging into the Cloudflare AI neuron-exhaustion classification! This exact fix landed today via #6983 (same cluster, same providerErrorRules/classify429 path), so closing this one as superseded. Your report and analysis helped confirm the direction — if you spot a gap the merged fix doesn't cover, please reopen with a pointer at the specific case. |
Fixes #6980
Summary
cloudflare-aiprovider rule inproviderErrorRules.tsmatching the daily neuron-allocation exhaustion body, scoped"connection"(account-wide budget — same reasoning as the existingopencoderules)./daily free allocation/itoQUOTA_PATTERNSinclassify429.tsas defense-in-depth for theclassify429FromErrorpath.S2c) totests/unit/provider-error-rules.test.tsfollowing the existingS1/S2bpattern: matches the exhaustion body →quota_exhausted/scope:"connection"; a generic 429 without the exhaustion wording → no match.Why
Cloudflare Workers AI's free-tier exhaustion body doesn't contain "quota"/"limit" (after "daily")/"exceed"/"credit"/"plan limit", so it currently falls through to the default
rate_limitclassification and gets retried on the short ~60s cooldown against a budget that only resets at UTC midnight.reason==="quota_exhausted"without an explicitcooldownMsalready resolves to "next UTC midnight" viarecordModelLockoutFailure(accountFallback.ts:553-556), so this rule just needs to reach that path — no new cooldown logic required.Test plan
node --import tsx/esm --test tests/unit/provider-error-rules.test.ts— 6/6 pass (including newS2c)eslint— cleanprettier --check— cleanPrior art
Same bug class as #1767→#1772→#2100 (origin of
classify429.ts).providerErrorRules.tsalready establishes this exact pattern foropencode/minimax.