fix: upstream provider funding refusal must not read as the customer owing money (#1411)

[sakibsadmanshajib]

Summary

Issue #1411 measured the OpenRouter account funding every paid alias at
about $1.59 of a $10 purchase, with no fallback provider configured. That
means an upstream 402 Payment Required (OpenRouter's own account out of
funds) is an imminent, previously untested failure mode. This PR
establishes what actually happens on that path and fixes the one real
defect found.

Investigation (with evidence, not just reasoning from the code)

1. Real upstream shape, confirmed live against OpenRouter's own docs
   (https://openrouter.ai/docs/api_reference/errors-and-debugging.md,
   fetched 2026-08-29): { error: { code: number, message: string, metadata?: {...} } }, HTTP status equal to error.code. 402 maps to
   error_type: payment_required.
2. Hold release was already correct. Every dispatch site
   (inference/orchestrator.go, inference/stream.go,
   inference/stream_responses.go, chat/dispatch.go,
   images/handler.go, audio/handler.go) already releases the
   reservation on any non-2xx upstream response before writing the
   customer error, and accounting.Service.releaseLocked is idempotent
   and reachable on a background context. No charge is ever attempted on
   this path. Now pinned by new integration tests (see below).
3. Message sanitization was already correct. The provider-blind
   allowlist (PR fix: stop upstream billing and quota text reaching the chat surface #1303) already default-denies money vocabulary
   ("insufficient", "credits", "balance", ...) for any status outside
   400/404/413/422, so a 402 always collapsed to a generic fallback
   message. Existing fixture in provider_blind_allowlist_test.go already
   covered this.
4. The real defect: the HTTP status code itself. Nothing remapped
   httpStatus. WriteProviderBlindUpstreamError forwarded a 402 that is
   about the UPSTREAM's own account balance straight through to the Hive
   customer, whose own reservation had already succeeded (they have
   credit; the upstream vendor relationship does not). A literal 402
   tells the customer the opposite: that they must pay. This is the fix
   in this PR.
5. RAG has no reservation at all, a pre-existing, already-tracked gap
   (issue RAG chat and agent tasks serve inference without reserving credits, so insufficient credits cannot fire #669), unrelated to and unchanged by this PR. For RAG, "is the
   hold released" is moot because there is no hold.
6. No pre-zero balance warning exists, confirmed by OpenRouter account is at $1.59 of $10 purchased credit, and nothing warns before it hits zero #1411's own body.
   Scoped as a separate, future change (monitoring/ops, no shared code
   path with this fix).

Fix

apps/edge-api/internal/errors/provider_blind.go,
WriteProviderBlindUpstreamError: map upstream 402 Payment Required to
the same 503 Service Unavailable / upstream_unavailable verdict a
503/504 already gets, reusing the existing "temporarily unavailable"
message path instead of inventing a new one. The original upstream status
(402) stays in the operator log for diagnosis; only the customer-facing
status changes.

One call site, so every caller already routing through it inherits the
fix: chat session dispatch (Open WebUI), /v1/chat/completions and
/v1/responses (sync and streaming), /v1/messages (delegates
in-process to the chat-completions chain), images, audio, and RAG's error
path.

Not touched, deliberately: images/handler.go and audio/handler.go's
own direct 402 writes for HIVE's own reservation refusal (a different,
pre-existing, already-tracked inconsistency, issue #633). That is a
different condition, the caller's own Hive balance, and must keep
meaning "you must pay."

Money-path bound

No token class billed changes, and no charge amount changes in either
direction. This only changes an HTTP status code and message on a path
that already released the hold and never finalized a charge, both before
and after this change. math/big, append-only ledger, and the one-credit
floor are all untouched.

Tests

• TestWriteProviderBlindUpstreamErrorRemaps402ToUpstreamUnavailable
  (unit, errors package): pins the status remap, the sanitized message,
  and that the operator log keeps the real 402 and raw body.
• TestExecuteSync_UpstreamPaymentRequired_ReleasesHoldAndDoesNotBill and
  TestExecuteStreaming_UpstreamPaymentRequired_ReleasesHoldAndDoesNotBill
  (integration, inference package): drive the real
  Orchestrator.executeSync / executeStreaming lifecycle against an
  in-process httptest.Server serving OpenRouter's documented 402 shape,
  asserting the customer-visible status/body, that release was called
  with reason upstream_error, and that finalize was never called (never
  billed).

All three are simulated: an in-process httptest.Server stub, never
a call to the real OpenRouter API. Draining the real wallet to observe
this would cause the exact outage this PR exists to prevent. The shape of
the simulated body is real (confirmed against OpenRouter's own docs
above); only the transport is faked.

Mutation check: temporarily disabled the 402 case in
WriteProviderBlindUpstreamError (renamed the switch case to an unused
status so it fell through to the default branch). All three new tests
went red with the exact 402-not-503 failure. Restored the fix; all three
went green again. Full edge-api suite (go test ./apps/edge-api/...)
passes, 0 regressions across all 20+ packages including rag, chat,
images, audio.

Buglog entry

{"date":"2026-08-29","error_message":"OpenRouter 402 Payment Required (own account out of funds) forwarded verbatim as HTTP 402 to the Hive customer","root_cause":"WriteProviderBlindUpstreamError only sanitized the MESSAGE body for an upstream refusal; it never remapped the HTTP STATUS itself, so a provider funding refusal (about Hive's own OpenRouter balance) was indistinguishable, at the status-code level, from a caller quota refusal (about the customer's own Hive balance)","fix":"remap upstream 402 to 503/upstream_unavailable in WriteProviderBlindUpstreamError, reusing the existing 503/504 temporarily-unavailable message path; original status kept in the operator log","tags":["billing","provider-blind","accounting","reservations","issue-1411"]}

Related

Closes/addresses #1411 (the specific customer-visible refusal shape).
Touches the same file family as #744 (provider hostname leak, already
fixed, this PR does not regress it) and #1327 (label-list boundary,
unrelated code path, not touched). Does not touch #600 (orphan-hold
reaper, not needed here since release already fires synchronously on
this path) or #633 (Hive's own quota-refusal status divergence, a
separate condition).

Plan and full investigation write-up: Obsidian vault,
hive/plan-2026-08-29-upstream-funds-exhaustion.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WyEwUxZCArdn1ZUDkTvuQ1

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

Next included review available in 49 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 29f8c083-88c3-4908-afff-d988c35effb1

📥 Commits

Reviewing files that changed from the base of the PR and between 908e48a and 48d34cf.

📒 Files selected for processing (3)

• apps/edge-api/internal/errors/provider_blind.go
• apps/edge-api/internal/errors/provider_blind_payment_required_test.go
• apps/edge-api/internal/inference/upstream_payment_required_test.go

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sakibsadmanshajib]

Adversarial review pipeline (per .claude/rules/orchestrator.md)

CodeRabbit CLI: SKIPPED. Rate limited (Review limit reached... You've used all 3 included reviews, 27 minute reset window at the time this PR was opened). Not counted as a clean pass, flagged here explicitly per the standing fallback rule.

Plain adversarial pass: performed directly on the diff (no defects found). Checked specifically for:

• Whether the responseStatus remap leaks into any OTHER status branch (429/503/504/400/422): confirmed no, every other case leaves responseStatus == httpStatus, verified both by re-reading the switch and by the full existing test suite staying green (no case previously pinned on those statuses changed behavior).
• Whether the operator log loses the real upstream status after the remap: confirmed no, logProviderBlindUpstreamError is called with the original httpStatus (402), not responseStatus (503); pinned by TestWriteProviderBlindUpstreamErrorRemaps402ToUpstreamUnavailable's log assertions.
• Whether /v1/messages (Anthropic-compatible surface) actually inherits the fix or has its own status handling: traced anthropic/translate_writer.go → reshapeToAnthropicError → writeAnthropicError(w, status, ...), confirmed status passes through unchanged from the recorded WriteProviderBlindUpstreamError call, so it does inherit.
• Whether any frontend/SDK code could be relying on the old (402, api_error, upstream_error) triple from this specific call site: grepped for insufficient_quota and confirmed the pre-existing convention for HIVE's own quota refusals (a different condition, images/audio's direct 402 writes, reservation_guard.go's 429) is untouched and uses different code values than this path did before or after, so no collision.

Security review (mandatory, money + input-adjacent path): performed directly. No SQL, no command exec, no file paths, no new regex, no new shared mutable state, no secrets. The change touches only which HTTP status/message the gateway writes back for an already-classified upstream failure; it introduces no new trust boundary and does not change what is billed, reserved, or released, only pinned tighter by the new integration tests (release still fires with reason upstream_error, finalize is still never called).

Static analysis: go vet ./apps/edge-api/internal/errors/... ./apps/edge-api/internal/inference/... clean. staticcheck not present in the toolchain image (NOSTATICCHECK), noted rather than silently skipped.

Full suite: go test ./apps/edge-api/... green across all packages (errors, inference, chat, rag, images, audio, anthropic, auth, authz, ...), 0 regressions.

Mutation test: temporarily disabled the 402 case (renamed the switch case to an unreachable status), all three new tests went red with the exact "402, want 503" failure, restored the fix, all three green again. Full command output kept in the PR description.

No inline threads opened: no defect was found to attach one to. If a later stream (CodeRabbit, once the rate limit resets, or a human reviewer) finds something, address it as a normal follow-up commit on this branch.