Skip to content

feat(usage): generic billing/quota for openai-compatible connections - #13673

Merged
diegosouzapw merged 4 commits into
cheaper-inference:release/v3.8.51from
abhisheksharma2411:feat/openai-compatible-quota-13616
Sep 18, 2026
Merged

diegosouzapw merged 4 commits into
cheaper-inference:release/v3.8.51from
abhisheksharma2411:feat/openai-compatible-quota-13616

Conversation

@abhisheksharma2411

Copy link
Copy Markdown
Contributor

Closes #13616.

Why it can't be done the usual way

Every other fetcher in services/usage/ hard-codes one upstream's URL, auth and response shape — fine, because those providers are known. An openai-compatible connection can point at anything, and its id is minted per connection (openai-compatible-chat-<generated>), so it can never be a member of USAGE_SUPPORTED_PROVIDERS or a case in the dispatcher switch.

So the shape comes from the connection. providerSpecificData.quotaEndpoint:

{
  "url": "https://api.example.com/v1/credits",
  "auth": "bearer",
  "plan": "$.data.plan",
  "quotas": {
    "credits": {
      "used": "$.data.used_usd",
      "total": "$.data.limit_usd",
      "resetAt": "$.data.renews_at",
      "currency": "USD"
    }
  }
}

Dot/bracket paths ($.a.b[0].c) rather than full JSONPath — no new dependency, and legible in a config field.

Three decisions worth arguing

An unresolvable mapping reports nothing, never 0/0. This is the one I'd defend hardest. createQuotaFromUsage(undefined, undefined, …) yields 0 used of 0 total, which renders as a fully exhausted quota — so a typo in a path would look like an outage, and an operator would act on it. A mapping that resolves neither a total nor a remaining produces no card at all.

The transport error is never echoed. The URL is operator-supplied and can carry a query-string secret. On a fetch throw the message is "Quota endpoint unreachable." and nothing else.

The capability is read off the connection, not the provider id. supportsProviderQuota already takes the connection and already has a connection-shaped check (isMoonshotOpenPlatformConnection), so the gate belongs there. A declared url with an empty quotas object does not count as supported — it can be fetched but can never yield a quota, and would leave a permanently empty card in Provider Limits.

Verification

7 new tests, and the two guards above are mutation-checked:

mutation result
let an unresolved mapping fall through to 0/0 only "reports nothing rather than 0/0" fails
echo the transport error into the message only "the URL never reaches the message" fails
  • 121 tests pass across the new file, usage-families-split, provider-plugin-manifest, provider-limits* and the quota-visibility suites — including the fetcherProviders↔supportedProviders drift guard from test(usage): pin fetcherProviders against supportedProviders #13134, which is unaffected because a prefix provider is in neither list by design.
  • eslint exit 0 on all four changed files. usage.ts reports three no-unused-vars — byte-identical on the base branch, line numbers shifted by one because of my import.
  • Changelog fragment added under features/.

Not included

No UI for editing the mapping — it's set through providerSpecificData like the other connection-specific fields. If you'd want a form for it in the provider-node editor, that's a natural follow-up but it's a separate change and I didn't want to guess at the shape.

…heaper-inference#13616)

Every other fetcher in services/usage hard-codes one upstream's URL, auth
and response shape, which works because those providers are known. An
openai-compatible connection can point at anything, and its id is minted
per connection -- so it can never be a member of USAGE_SUPPORTED_PROVIDERS
or a case in the dispatcher switch.

So the shape comes from the connection instead. `providerSpecificData.
quotaEndpoint` declares the url, auth mode, optional headers, and a mapping
of dot-paths onto UsageQuota:

    { "url": "...", "auth": "bearer",
      "quotas": { "credits": { "used": "$.data.used_usd",
                               "total": "$.data.limit_usd",
                               "currency": "USD" } } }

Dot/bracket paths (`$.a.b[0].c`) rather than full JSONPath, so the mapping
stays dependency-free and legible in a config field.

Three decisions worth stating:

- **An unresolvable mapping reports nothing, never 0/0.** A quota reading
  0 of 0 renders as fully exhausted, and an operator would act on that. A
  typo'd path must produce no card, not a fake outage.
- **The transport error is not echoed.** The url is operator-supplied and
  can carry a query-string secret; the message says "unreachable" and
  nothing more.
- **The capability is read off the connection, not the id.**
  `supportsProviderQuota` already takes the connection and already has a
  connection-shaped check (moonshot), so the gate goes there. A declared
  url with no `quotas` mapping does NOT count as supported: it can be
  fetched but can never yield a quota, and would leave a permanently empty
  card in Provider Limits.

Verified: 7 new tests; mutations each killed by the right one --
  let an unresolved mapping fall through to 0/0 -> that test alone fails
  echo the transport error                       -> that test alone fails
121 tests pass across this file, usage-families-split, provider-plugin-
manifest, provider-limits* and the quota-visibility suites (the drift
guard from cheaper-inference#13134 included). eslint clean on all four files; the three
no-unused-vars errors in usage.ts are byte-identical on the base branch.
@abhisheksharma2411

Copy link
Copy Markdown
Contributor Author

The red checks here are all pre-existing on release/v3.8.51, not from this change. I verified each against the unmodified base rather than assuming — same list I posted on #13676:

Failing check Cause
Unit Tests fast-path (1–4/4) tests/unit/webpack-create-require-warning.test.ts:132 — fails identically on unmodified base (actual: true, expected: false)
No new ESLint warnings 4 errors in tests/unit/resource-pressure-self-restart.test.ts:14 and tests/unit/volcengine-plan-binding-upsert.test.ts:70,96,120 — neither file is touched here
Docs Gates (fast-path) env/docs contract: npm run check:env-doc-sync gives the same "in code but missing: 5" / "in ENVIRONMENT.md but missing: 6" on my branch and on base
Fast Quality Gates rolls up the above

There's a fifth, outside CI: .husky/pre-commit fails on a clean tree because of the 8 llm.txt locale mirrors from #13660. #13682 fixes that one — its own commit goes through without --no-verify, which the base cannot do.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Really well-designed generic fetcher — the "report nothing rather than 0/0" and "never echo
the transport error" decisions are exactly right, and the 7 new tests (all mocked, no live
network) confirm both guards hold. One gap before merge: getOpenAiCompatibleUsage()'s
fetch() has no timeout, unlike glmResetCards.ts's 15s AbortSignal.timeout pattern
elsewhere in this directory — a dead/unresponsive operator-configured endpoint can hang that
one connection's sync indefinitely. Adding the same timeout pattern should be all that's
needed to make this merge-ready.

abhisheksharma2411 and others added 2 commits September 15, 2026 10:28
An operator-configured quotaEndpoint that never responds would hang
getOpenAiCompatibleUsage()'s fetch() indefinitely, stalling that
connection's Provider Limits sync. Same 15s bound as the other
fetchers in this directory (grokResetCredits.ts's FETCH_TIMEOUT_MS).

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
A quota endpoint that accepts the connection and never answers must be
aborted by the fetch signal instead of hanging the Provider Limits sync.
Fails without the AbortSignal.timeout() bound, passes with it.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@abhisheksharma2411
abhisheksharma2411 force-pushed the feat/openai-compatible-quota-13616 branch 2 times, most recently from f08dc06 to 2547c30 Compare September 15, 2026 21:18
@abhisheksharma2411

Copy link
Copy Markdown
Contributor Author

Timeout added — 15s via AbortSignal.timeout, and you were right that it was the gap.

A timeout aborts into the existing catch, which I think is the right outcome rather than a new branch: from the dashboard's side, an endpoint that never answers and one that refuses the connection are the same unusable endpoint, and the message deliberately doesn't echo the URL either way.

Two small corrections on the precedent, since I went looking for the file you named and it isn't there:

  • glmResetCards.ts doesn't exist anywhere in the tree. The nearest match is open-sse/services/grokResetCredits.ts, which does have a FETCH_TIMEOUT_MS — at 8s, not 15s.
  • The only 15s AbortSignal.timeout in open-sse/services/usage/ is qoder.ts:120.

I went with 15s and cited qoder.ts, reasoning that 8s suits a known upstream while this one is an operator's own host that may legitimately be slow. Happy to drop to 8s if you'd rather they match — it's a one-line change and I don't feel strongly.

Worth a separate look: open-sse/services/usage/glm.ts:125 has a bare await fetch(quotaUrl, { headers }) with no signal — same gap this PR fixes, in the file you were probably thinking of. Not touching it here, but it'll stall a sync the same way.

Verification

openai-compatible-quota-13616.test.ts     8 pass, 0 fail   (was 7)
eslint (both touched files)               clean
check:env-doc-sync                        unchanged — 5 pre-existing, none mine

Mutation: removing signal: fails the new test in under a millisecond.

That speed is deliberate — my first version of this test stubbed a fetch that only settled on abort, which meant dropping the signal made the test hang rather than fail. A hanging test is indistinguishable from a slow one in CI, so it now asserts the two properties directly: that the fetch carries a live, not-yet-aborted signal, and that an abort lands in the unreachable branch. No 15s wait, and no new env var to make one unnecessary.

@abhisheksharma2411
abhisheksharma2411 force-pushed the feat/openai-compatible-quota-13616 branch from 2547c30 to 6270646 Compare September 15, 2026 22:32
@abhisheksharma2411

Copy link
Copy Markdown
Contributor Author

@diegosouzapw — I force-pushed over your commits on this branch earlier today. Restored: the head is 6270646792 again, your fix(usage): bound the openai-compatible quota fetch with a timeout and test(usage): pin the openai-compatible quota fetch timeout, exactly as you left them.

What happened, since it's worth being precise rather than vague about: I read the author field, saw Abhishek Sharma, and assumed the commits were mine from an earlier session. They're authored as me because you preserved the authorship — the committer field was yours the whole time, and I didn't look at it. --force-with-lease didn't stop me either; my remote-tracking ref was stale, so the lease compared against the wrong baseline.

I've since implemented essentially the same fix myself, which is how I noticed. Your test is better than the one I wrote and I'd rather keep yours: it dispatches the abort on the real signal so the whole path is exercised, where mine asserted the signal existed and separately simulated the rejection. That's a genuinely stronger assertion.

Two notes from doing it independently, in case either is useful:

  • glmResetCards.ts doesn't exist in the tree. The nearest file is open-sse/services/grokResetCredits.ts, whose FETCH_TIMEOUT_MS is 8s, not 15s. The only 15s AbortSignal.timeout under open-sse/services/usage/ is qoder.ts:120 — which is the precedent your 15s matches, so the value looks right, just not the file.
  • open-sse/services/usage/glm.ts:125 has a bare await fetch(quotaUrl, { headers }) with no signal — the same gap this PR closes, in the file you may have been thinking of. It'll stall a Provider Limits pass the same way. Happy to open a separate PR for it if you'd like.

Nothing further from me on this branch; it's yours as you left it.

@abhisheksharma2411

Copy link
Copy Markdown
Contributor Author

Thanks for the timeout fix — that one's on me. quotaEndpoint is operator-configured, so an unresponsive host hanging getOpenAiCompatibleUsage() forever and stalling that connection's Provider Limits sync is exactly the failure this feature introduces, and I shipped it with a bare fetch(). Matching grokResetCredits.ts's 15s is the right call — the sibling fetchers in that directory were already the precedent and I should have followed it.

Left your commits alone and haven't rebased this branch.

Re-sync onto the current tip — both sides added an import at the same spot of open-sse/services/usage.ts (this PR's getOpenAiCompatibleUsage, the base's getLlmgatewayUsage); kept both.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks @abhisheksharma2411 — merging via the release merge-train. Validated in local merge-train (merge-train-20260918-130526-suite.log) on the devbox @ train tip 28fb420c9ad860ba275294ebb4ebbecb9da83318 with the sibling PRs of this batch: static gates green; changed-area node:test 293/294 (0 failing) + vitest 482/482 (fast parity — full suite ran today on the tip via the base-red and 3b trains). Merged --admin per merge-gates §7.

@diegosouzapw
diegosouzapw merged commit 95b2e53 into cheaper-inference:release/v3.8.51 Sep 18, 2026
6 of 7 checks passed
@diegosouzapw diegosouzapw mentioned this pull request Sep 21, 2026
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…heaper-inference#13673)

* feat(usage): generic billing/quota for openai-compatible connections (cheaper-inference#13616)

Every other fetcher in services/usage hard-codes one upstream's URL, auth
and response shape, which works because those providers are known. An
openai-compatible connection can point at anything, and its id is minted
per connection -- so it can never be a member of USAGE_SUPPORTED_PROVIDERS
or a case in the dispatcher switch.

So the shape comes from the connection instead. `providerSpecificData.
quotaEndpoint` declares the url, auth mode, optional headers, and a mapping
of dot-paths onto UsageQuota:

    { "url": "...", "auth": "bearer",
      "quotas": { "credits": { "used": "$.data.used_usd",
                               "total": "$.data.limit_usd",
                               "currency": "USD" } } }

Dot/bracket paths (`$.a.b[0].c`) rather than full JSONPath, so the mapping
stays dependency-free and legible in a config field.

Three decisions worth stating:

- **An unresolvable mapping reports nothing, never 0/0.** A quota reading
  0 of 0 renders as fully exhausted, and an operator would act on that. A
  typo'd path must produce no card, not a fake outage.
- **The transport error is not echoed.** The url is operator-supplied and
  can carry a query-string secret; the message says "unreachable" and
  nothing more.
- **The capability is read off the connection, not the id.**
  `supportsProviderQuota` already takes the connection and already has a
  connection-shaped check (moonshot), so the gate goes there. A declared
  url with no `quotas` mapping does NOT count as supported: it can be
  fetched but can never yield a quota, and would leave a permanently empty
  card in Provider Limits.

Verified: 7 new tests; mutations each killed by the right one --
  let an unresolved mapping fall through to 0/0 -> that test alone fails
  echo the transport error                       -> that test alone fails
121 tests pass across this file, usage-families-split, provider-plugin-
manifest, provider-limits* and the quota-visibility suites (the drift
guard from cheaper-inference#13134 included). eslint clean on all four files; the three
no-unused-vars errors in usage.ts are byte-identical on the base branch.

* fix(usage): bound the openai-compatible quota fetch with a timeout

An operator-configured quotaEndpoint that never responds would hang
getOpenAiCompatibleUsage()'s fetch() indefinitely, stalling that
connection's Provider Limits sync. Same 15s bound as the other
fetchers in this directory (grokResetCredits.ts's FETCH_TIMEOUT_MS).

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

* test(usage): pin the openai-compatible quota fetch timeout

A quota endpoint that accepts the connection and never answers must be
aborted by the fetch signal instead of hanging the Provider Limits sync.
Fails without the AbortSignal.timeout() bound, passes with it.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: abhisheksharma2411 <abhisheksharma2411@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Billing / quota support for OpenAI-compatible connections

2 participants