fix: stop reporting a gateway cooldown as an upstream refusal (issue #1374)

[sakibsadmanshajib]

The finding is the opposite of the report

I was dispatched to fix a spent daily allowance on the hive-free pool. There was no spent allowance. The pool was healthy the entire time, and the thing that said otherwise was our own diagnostic.

Three Live integration (SDK tests + smoke) runs failed this morning carrying the banner:

UPSTREAM REFUSAL, not a performance regression: the provider rate limited us (429).

That banner is wrong, and it is wrong in a way that cost a full investigation cycle. It is the same defect the step was built to prevent (issue #1088, a red check that misreports its own cause), wearing the opposite sign.

Evidence that the pool was fine

The job's own Probe each free pool member and name any that is gone step, in both failing runs (33243396287 and 33244166031):

free pool members: 3 healthy, 1 unhealthy
  DEAD MEMBER: openai/gemini-flash-latest -- litellm.RateLimitError: ... Error code: 429 -
  'You exceeded your current quota, please check your plan and billing details.'

One member is rate limited, the Google one. Direct probes of the other two providers, one request each:

Probe	Time (UTC)	Result
OpenRouter dots-studio/dots-3-note-preview:free	09:15Z	HTTP 200, real completion, cost: 0
Groq openai/gpt-oss-20b	09:18Z	HTTP 200, x-ratelimit-remaining-requests: 955 of 1000

And the deployed box, one turn on hive-free, run by qamatrix on a key created for the probe and revoked after:

HTTP 200 in 2.495377s
{"model":"hive-free","choices":[{"finish_reason":"length",...}],
 "usage":{"prompt_tokens":78,"completion_tokens":20,"total_tokens":98,
 "completion_tokens_details":{"reasoning_tokens":18,...}}}

The live free tier serves. Forty-five of a thousand daily Groq requests spent is not an exhausted allowance.

What the job actually failed on

sdk-tests-js exit 1 on Error: 500 Failed to store file in tests/files/files.test.ts and tests/batches/batches.test.ts. sdk-tests-py exit 1 on [XPASS(strict)] issue #1274. sdk-tests-java exit 0. The PR branch under test in the second run is named fix/ci-object-storage-blindness-1324. The failure is object storage, which is #1324 and #1390, not a provider.

The three false positives, all from that run's own artifact

The classifier prints its own matching lines, which is how they were found.

1. LiteLLM's router cooldown. The line that set the class:

alias="deepseek-v4-flash" status=429 raw_message="... No deployments available for
selected model, Try again in 5 seconds. Passed model=route-deepseek-v4-flash.
pre-call-checks=False, cooldown_list=['7916fd0b...','e644bb2c...']"

Wrong evidence twice over. The alias is deepseek-v4-flash, which is HIVE_TOOLS_MODEL and not the free pool, so a paid alias's problem was reported as the free tier's. And No deployments available ... cooldown_list=[...] is the gateway declining to dispatch to deployments it has already benched. No upstream call was made; nobody refused anything. It matched only because the regex carried the bare token status=429.

2. LiteLLM's cooldown table. Found only by running the fixed script against the real compose-logs artifact rather than against my own fixtures:

Cooldown Deployments=[('7916fd0b...', {'exception_received': 'litellm.RateLimitError:
RateLimitError: OpenAIExce***', 'status_code': '429', 'timestamp': ..., 'cooldown_time': 5})]

It quotes the exception that caused the cooldown, verbatim, and LiteLLM reprints the whole table on every subsequent dispatch for as long as the entry lives. One past event re-reports itself indefinitely.

3. No fallback model group found. LiteLLM emits this on any failed dispatch to a group with no configured fallback, whatever the underlying error. In the same artifact it appears on a deliberate over-length prompt (400), on deepseek not supporting image input (404), and on a Groq TTS suite asserting an invalid response_format is refused (400). All three are expected behaviour under test. Had the 429 branch not fired first, the dead-member branch would have.

The dead-member branch now requires the NotFoundError shape and model_group=route-free-pool, which is the case its message describes and the exact shape already recorded in apps/edge-api/internal/inference/retry_test.go from run 32830060362.

Corroboration that this was noise, not an allowance

Issue #1374 records seven consecutive failures on the morning of 2026-08-29 (06:47, 07:24, 07:40, 07:51, 08:13, 08:42, 09:11Z). Only two carry the 429 classification; four carry no cause at all. A spent daily allowance classifies every time. An incidental grep hit inside a rolling --tail=2000 window does not.

What changed

scripts/classify-upstream-refusal.py (new). The classification moves out of the YAML heredoc because YAML can carry no regression test, and this logic had shipped false positives nothing could have caught. Follows the existing scripts/report-free-pool-health.py shape: pure stdlib, --selfcheck, same credential redactor. Branch order is unchanged. Fixtures are copied from run 33243396287's own output, including both false positives and the genuine dead-member case, so the branch cannot be tightened into uselessness without a test going red.

.github/workflows/ci.yml. The step body shrinks to the redaction pipeline plus one script call. This also removes a second defect: the old reporting grep ... | head -5 exited 2 on SIGPIPE under set -o pipefail, the exact trap the neighbouring comment warns about for the classification greps, visible in that run as grep: write error: Broken pipe followed by ##[error]Process completed with exit code 2. Reading the file into memory removes it by construction. I did not reproduce the SIGPIPE locally; a five-line fixture is too small to make grep write after head exits. The evidence is the run's output.

scripts/report-free-pool-health.py. A member answering 429 is rate limited, not retired, and the two remedies are opposites: the shipped advice was "replace the member row in supabase/migrations/", which for the Gemini member would have deleted one that is coming back. It now also surfaces the quota window and retry hint, extracted from the full error text rather than truncated away at 300 characters:

  RATE LIMITED MEMBER: openai/gemini-flash-latest -- litellm.RateLimitError: ...
    quota window: GenerateRequestsPerDayPerProjectPerModel-FreeTier, 34s
::warning::free pool member 'openai/gemini-flash-latest' is RATE LIMITED, not gone.
Do NOT replace its row: ...

That line is the one fact this whole investigation needed and could not get. The real log truncated before Google's quotaId, so I still do not know whether the Gemini window is per-day or per-minute. Closing that gap is the point of the change, not a claim to have closed the question.

Makefile. Both self-checks join test-scripts, already a required check. report-free-pool-health.py has shipped a --selfcheck since issue #1064 that no workflow ever invoked, which is precisely the coverage-on-paper shape the Makefile comment already blames for issue #717.

What this does not do

• It does not repoint any CI model knob at a paid alias. HIVE_TOOLS_MODEL resolving to deepseek-v4-pro burned 150 calls and 30.2M credits in one day on 2026-08-25. No model knob is touched.
• It does not fix the Gemini member. That is operational: either the Google project's free-tier quota is spent and resets on its own schedule, or the project has no free tier for that model and needs billing. Whoever holds the Google Cloud console settles it in one look. This change only makes the log say which.
• It does not correct .wolf/decisions.md D-048, and that is deliberate. D-048 records the pool as four Hive-owned keys behind load-balanced failover: two free Groq text keys, an OpenRouter free key, a Gemini key. Migration 20260824_02_free_pool_router.sql ships exactly that, member for member. The suspicion that 20260823_21_groq_text_routes_to_openrouter_free.sql had collapsed the pool's diversity does not hold: that migration is dated the day before the pool existed and moved the legacy tier aliases, not the pool. D-048 matches shipped reality, and a correction that corrects nothing is worse than silence.

The real redundancy behind hive-free, for the record

Three distinct providers, four keys, and between three and four distinct allowances. route-free-pool-groq and route-free-pool-groq-2 are the same model on the same provider differing only by key, and Groq enforces rate limits per organization, not per key. If both keys live in one Groq org they are one allowance wearing two hats. Nothing in the repo settles it. Filed below.

One genuinely stale artefact found on the way: deploy/litellm/config.yaml's "CONCENTRATION RISK" comment (lines 33 to 45) describes the pre-pool topology from 2026-08-23 and says five aliases resolve to one model at one provider. 20260824_02 superseded that the next day, and the checked-in config.yaml contains no route-free-pool entry at all: the pool reaches LiteLLM only through the job's "Sync LiteLLM model list from the DB catalog" step. That comment is what generated this investigation's false premise. Not touched here, because that file is hot and shared; filed instead.

Filed, not built

1. deploy/litellm/config.yaml's concentration-risk comment is a day out of date and actively misleading.
2. Whether GROQ_API_KEY and GROQ_API_KEY_2 are two Groq organizations or two keys on one. Settled by one probe from whoever holds both: compare x-ratelimit-remaining-requests and see whether spending on one moves the other.
3. CI shares the customer-facing free tier by design (D-047). Giving CI its own lane is an owner-level decision, not a builder's call.
4. Minting a live session from a worktree is undocumented and non-obvious. The blank Supabase values in .env are only the first obstacle; the box's public listener deliberately refuses the admin API, so generate_link 404s and minting must run against the internal caddy-supabase listener through an SSH tunnel with a Host header rewrite. Coordinate with ci: give CI a real object store instead of a dead endpoint (#1324, #1380) #1390, which is already correcting .claude/skills/worktree-compose-stack.md.

Testing

$ python3 scripts/classify-upstream-refusal.py --selfcheck
ok: classify-upstream-refusal verdicts

$ python3 scripts/report-free-pool-health.py --selfcheck
ok: report-free-pool-health verdicts and fallbacks

$ make test-scripts
(exit 0)

Verified RED before GREEN. The first run of the new self-check failed with the production string verbatim:

AssertionError: a gateway cooldown must not classify, got: the provider rate limited us
(429). Check whether the window is per minute (transient, so a rerun may pass) or per day
(an allowance that is spent). The refused alias(es): deepseek-v4-flash.

And against the real artifacts from the failing run, before and after:

$ python3 scripts/classify-upstream-refusal.py <run-33243396287-compose-logs>
before: ::error::UPSTREAM REFUSAL ... the provider rate limited us (429).
after:  no upstream refusal signature in the litellm or edge-api logs, so this failure
        is something else.

Shared-file note

This touches .github/workflows/ci.yml, which #1390 also edits. Confirmed with the coordinator that #1390 does not touch the classifier: its ci.yml diff has zero matches for upstream refus, UPSTREAM_FAILURE_CLASS and rate.?limit. Its hunks at @@ -1820,12 @@ and @@ -1838,6 @@ are adjacent to this region but do not overlap. Branched from current origin/main. If #1390 merges first I will rebase and re-read those hunks rather than trust a clean auto-merge.

Buglog entry

{"id":"bug-2026-08-29-classifier-misattributes-gateway-cooldown","date":"2026-08-29","title":"CI upstream-refusal classifier reported a LiteLLM router cooldown on a paid alias as a whole-pool provider rate limit","error_message":"UPSTREAM REFUSAL, not a performance regression: the provider rate limited us (429). Check whether the window is per minute (transient, so a rerun may pass) or per day (an allowance that is spent).","root_cause":"The 'Name the real cause when the upstream refused' step in .github/workflows/ci.yml grepped a 2000-line window of the litellm and edge-api container logs for the bare token 'status=429' among others. Three log shapes matched that are not upstream refusals: LiteLLM's own 'No deployments available for selected model ... cooldown_list=[...]' router cooldown (no upstream call is made), its 'Cooldown Deployments=[...]' table (which quotes the causing exception verbatim and reprints it on every dispatch while the entry lives), and a bare 'No fallback model group found' (which LiteLLM emits on any failed dispatch to a group with no configured fallback, including three refusals the job deliberately tests for). The matched line also carried alias deepseek-v4-flash, so a paid alias's gateway cooldown was reported as a refusal of the hive-free pool. The free pool was healthy throughout: the job's own probe step reported 3 healthy 1 unhealthy in both runs, only the Gemini member was rate limited, and the actual job failures were '500 Failed to store file' and a strict XPASS on issue #1274. Only two of the seven consecutive failures on issue #1374 that morning classified at all, which is the signature of an incidental grep hit rather than a spent allowance.","fix":"Moved the classification from a bash heredoc into scripts/classify-upstream-refusal.py, which make test-scripts runs as a required check. Gateway cooldown lines in both shapes are dropped before any branch reads them. The dead-member branch now requires the NotFoundError shape together with model_group=route-free-pool. A classified refusal names the alias it saw. Fixtures are copied from run 33243396287's own output. This also removes a SIGPIPE exit 2 from the old 'grep ... | head -5' reporting pipeline under set -o pipefail. Separately, report-free-pool-health.py now distinguishes a rate-limited member from a retired one, whose remedies are opposites, and surfaces the provider's quota window and retry hint from the full error text instead of truncating them away at 300 characters.","tags":["ci","litellm","free-pool","signal-quality","false-positive","issue-1088","issue-1064","issue-1374"]}

🤖 Generated with Claude Code

https://claude.ai/code/session_01WyEwUxZCArdn1ZUDkTvuQ1

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

Next included review available in 26 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4f44e4f4-8244-4139-8cf4-70d372f960a3

📥 Commits

Reviewing files that changed from the base of the PR and between 13f277c and c7719c5.

📒 Files selected for processing (4)

• .github/workflows/ci.yml
• Makefile
• scripts/classify-upstream-refusal.py
• scripts/report-free-pool-health.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sakibsadmanshajib]

Adversarial review, two streams

Stream 1: CodeRabbit CLI — RAN, no findings

$ coderabbit review --committed --base main
Review complete
No findings ✔
4 files reviewed:
  - .github/workflows/ci.yml
  - Makefile
  - scripts/classify-upstream-refusal.py
  - scripts/report-free-pool-health.py

Not skipped and not rate limited on this run, so this is a real clean pass rather than an absence.

Stream 2: Antigravity (gemini-3.1-pro-high, effort high) — RAN, four findings

Two accepted and fixed in c7719c5, two answered.

---

Finding 1 and 3 (accepted, fixed). Dropping LiteLLM's cooldown bookkeeping before every branch risks a false negative on permanent refusals. In the reviewer's words:

A reprinted daily budget (TPD) or out-of-credit (402) error represents a permanent refusal. Dropping the cooldown table universally means you risk blinding the script to permanent refusals if the original refusal rotated out of the log.

Correct, and it matters: I would have traded the false positive this PR fixes for a false negative on the only two conditions no rerun can clear. The cooldown table quotes the exception that caused each entry, so when the original line has scrolled out of the --tail=2000 window the table is the last surviving evidence.

Fixed on the principle the reviewer identified, which is whether the quoted condition is still true when reprinted:

• A 429 in a cooldown entry is stale by construction, because LiteLLM reprints the table on every dispatch for as long as the entry lives. That staleness is the defect. The transient branch keeps reading the filtered lines.
• A daily token budget or a 402 stays true. The spend branches now read the unfiltered lines.

Two new self-check cases pin it: a TPD quoted inside a cooldown entry classifies as the daily-budget refusal, a 402 quoted there classifies as out of credit, and a plain 429 there still does not classify.

Verified this does not reopen the original false positive. Run 33243396287's compose-logs artifact still classifies as no refusal, because its cooldown entries quote a RateLimitError and a NotFoundError and never a daily budget or a 402.

---

Finding 4 (answered, not changed). DEAD_POOL_MEMBER requiring model_group=route-free-pool is too narrow, and the reviewer proposes instead requiring NotFoundError alongside No fallback model group found, with a filter for known test-failure strings.

I am not taking the proposed alternative, for a concrete reason: it does not separate the cases. The deepseek-v4-flash line in that same artifact is already litellm.NotFoundError and No fallback model group found:

alias="deepseek-v4-flash" status=404 raw_message="litellm.NotFoundError: NotFoundError:
OpenrouterException - {"message":"No endpoints found that support image input","code":404}
No fallback model group found for original model_group=route-deepseek-v4-flash..."

So the alternative classifies it and needs the denylist to rescue it. That denylist is a list of expected test failures maintained by hand, and the next test that asserts a 404 refusal re-opens exactly this bug. Trading a narrow matcher for a list that rots is a worse trade than the narrowness.

The underlying concern is real, though: a genuinely dead member in another model group should not vanish. It does not. The same commit makes an unclassified run print the closest matching lines it saw, so a NotFoundError on route-openrouter-embedding or any other group reaches the reader as evidence rather than being dropped without trace. What it no longer does is print the free pool's verdict text over it, and that text is specifically about the free pool: it ends "The 'Probe each free pool member' step earlier in this job names them." Widening the matcher without rewriting the message would just relocate the misattribution this PR exists to remove.

---

Findings 2, 5, 6, 7: no material issues. The reviewer confirmed GATEWAY_COOLDOWN cannot span lines (the script splits first, so re.S is inert), that WINDOW_HINT has no catastrophic backtracking path, that the exit-code refactor is semantically identical, and that the GITHUB_ENV write correctly replaces the old echo >> $GITHUB_ENV.

I hardened two of those anyway, because "correct today" and "stays correct" differ:

• The exit code is computed from the payload again rather than by scanning rendered lines for an ::error:: prefix. Identical behaviour, but an exit code that depends on message wording breaks the next time someone rewords a message, and this one decides whether a free-tier outage stops the run.
• _window_hint bounds its input before scanning. No nested quantifier means no catastrophic backtracking, but the error string is provider-controlled and unbounded, and the scan is linear per start position.

And I added the test the reviewer's point 7 implies but that did not exist: the GITHUB_ENV contract is now pinned, including that the value stays on one line so it cannot corrupt later entries, and that nothing is written when there is no refusal so a later step cannot read a stale class.

---

Self-review findings, same commit

Three things neither stream raised.

1. An unclassified run went silent. Over a log visibly full of 429s, "no upstream refusal signature" with nothing else is the failure mode that would have replaced the one being fixed. It now reports how many cooldown lines it discounted and why, and shows the closest matching lines.
2. classify() had no coverage for the empty and whitespace-only log. Added.
3. The GITHUB_ENV write had no coverage at all. Added, as above.

Verification after every change

$ python3 scripts/classify-upstream-refusal.py --selfcheck
ok: classify-upstream-refusal verdicts

$ python3 scripts/report-free-pool-health.py --selfcheck
ok: report-free-pool-health verdicts and fallbacks

$ make test-scripts
(exit 0)

$ python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))"
yaml OK

Against run 33243396287's real artifact, which is the case that started this:

no upstream refusal signature in the litellm or edge-api logs, so this failure is
something else. The compose-logs artifact below is the next place to look.
(23 LiteLLM cooldown line(s) were discounted on purpose: ... A daily budget or a 402
quoted in the same place WOULD still have been reported, since those stay true. ...)
closest matching lines (redacted, first 5):
  ...

[sakibsadmanshajib]

Correction to my review comment above, because the distinction matters and I blurred it.

There are two CodeRabbit surfaces on this PR and they did not have the same outcome:

• The CodeRabbit CLI, which I ran locally against main, did run and reported no findings. That is the clean pass I quoted.
• The CodeRabbit GitHub bot check on this PR reports Review rate limited. That one is SKIPPED, not a pass, exactly as the dispatch brief predicted for the repo-wide quota.

So the honest tally for stage 6 is: CodeRabbit CLI RAN clean, CodeRabbit bot SKIPPED (rate limited), Antigravity RAN with four findings, two of which I accepted and fixed in c7719c5. Two independent streams did run, so the review requirement is met, but nobody should read the green CodeRabbit check on this page as a review having happened. It is the absence of one.

[sakibsadmanshajib]

Filed the deferred items from this investigation, so they do not live only in the PR body:

• OpenRouter account is at $1.59 of $10 purchased credit, and nothing warns before it hits zero #1411 — the OpenRouter account is at $1.59 of $10 purchased credit and nothing warns before zero. Measured twice, four minutes apart, and actively draining. This outranks everything in this PR: every paid alias is OpenRouter and there are deliberately no chat fallbacks, so zero means hard refusals. The free tier is unaffected, because OpenRouter gates free throughput on lifetime purchases rather than balance. Needs someone to confirm whether the key in the shared .env is the same one CI and the box use.
• deploy/litellm/config.yaml's CONCENTRATION RISK comment describes a topology superseded the next day #1412 — deploy/litellm/config.yaml's CONCENTRATION RISK comment describes the topology from the day before the free pool existed. This is what produced the wrong root-cause hypothesis that started this work.
• Unknown whether GROQ_API_KEY and GROQ_API_KEY_2 are two Groq orgs or two keys on one, which decides if the free pool has 4 allowances or 3 #1413 — unknown whether GROQ_API_KEY and GROQ_API_KEY_2 are two Groq organizations or two keys on one, which decides whether the pool has four allowances or three. Two calls settle it.
• Minting a live session from a worktree is undocumented: live-auth.mjs cannot reach the admin API through the public origin #1414 — minting a live session from a worktree is undocumented; the public origin refuses the admin API and the procedure needs an SSH tunnel with a Host rewrite. Procedure credited to qamatrix, who derived and verified it.

None of these is fixed here. This PR stays scoped to the classifier.