Repository navigation
feat(usage): generic billing/quota for openai-compatible connections - #13673
Conversation
…heaper-inference#13616) Every other fetcher in services/usage hard-codes one upstream's URL, auth and response shape, which works because those providers are known. An openai-compatible connection can point at anything, and its id is minted per connection -- so it can never be a member of USAGE_SUPPORTED_PROVIDERS or a case in the dispatcher switch. So the shape comes from the connection instead. `providerSpecificData. quotaEndpoint` declares the url, auth mode, optional headers, and a mapping of dot-paths onto UsageQuota: { "url": "...", "auth": "bearer", "quotas": { "credits": { "used": "$.data.used_usd", "total": "$.data.limit_usd", "currency": "USD" } } } Dot/bracket paths (`$.a.b[0].c`) rather than full JSONPath, so the mapping stays dependency-free and legible in a config field. Three decisions worth stating: - **An unresolvable mapping reports nothing, never 0/0.** A quota reading 0 of 0 renders as fully exhausted, and an operator would act on that. A typo'd path must produce no card, not a fake outage. - **The transport error is not echoed.** The url is operator-supplied and can carry a query-string secret; the message says "unreachable" and nothing more. - **The capability is read off the connection, not the id.** `supportsProviderQuota` already takes the connection and already has a connection-shaped check (moonshot), so the gate goes there. A declared url with no `quotas` mapping does NOT count as supported: it can be fetched but can never yield a quota, and would leave a permanently empty card in Provider Limits. Verified: 7 new tests; mutations each killed by the right one -- let an unresolved mapping fall through to 0/0 -> that test alone fails echo the transport error -> that test alone fails 121 tests pass across this file, usage-families-split, provider-plugin- manifest, provider-limits* and the quota-visibility suites (the drift guard from cheaper-inference#13134 included). eslint clean on all four files; the three no-unused-vars errors in usage.ts are byte-identical on the base branch.
|
The red checks here are all pre-existing on
There's a fifth, outside CI: |
|
Really well-designed generic fetcher — the "report nothing rather than 0/0" and "never echo |
An operator-configured quotaEndpoint that never responds would hang getOpenAiCompatibleUsage()'s fetch() indefinitely, stalling that connection's Provider Limits sync. Same 15s bound as the other fetchers in this directory (grokResetCredits.ts's FETCH_TIMEOUT_MS). Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
A quota endpoint that accepts the connection and never answers must be aborted by the fetch signal instead of hanging the Provider Limits sync. Fails without the AbortSignal.timeout() bound, passes with it. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
f08dc06 to
2547c30
Compare
|
Timeout added — 15s via A timeout aborts into the existing Two small corrections on the precedent, since I went looking for the file you named and it isn't there:
I went with 15s and cited Worth a separate look: VerificationMutation: removing That speed is deliberate — my first version of this test stubbed a fetch that only settled on abort, which meant dropping the signal made the test hang rather than fail. A hanging test is indistinguishable from a slow one in CI, so it now asserts the two properties directly: that the fetch carries a live, not-yet-aborted signal, and that an abort lands in the unreachable branch. No 15s wait, and no new env var to make one unnecessary. |
2547c30 to
6270646
Compare
|
@diegosouzapw — I force-pushed over your commits on this branch earlier today. Restored: the head is What happened, since it's worth being precise rather than vague about: I read the I've since implemented essentially the same fix myself, which is how I noticed. Your test is better than the one I wrote and I'd rather keep yours: it dispatches the abort on the real signal so the whole path is exercised, where mine asserted the signal existed and separately simulated the rejection. That's a genuinely stronger assertion. Two notes from doing it independently, in case either is useful:
Nothing further from me on this branch; it's yours as you left it. |
|
Thanks for the timeout fix — that one's on me. Left your commits alone and haven't rebased this branch. |
Re-sync onto the current tip — both sides added an import at the same spot of open-sse/services/usage.ts (this PR's getOpenAiCompatibleUsage, the base's getLlmgatewayUsage); kept both. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
|
Thanks @abhisheksharma2411 — merging via the release merge-train. Validated in local merge-train (merge-train-20260918-130526-suite.log) on the devbox @ train tip 28fb420c9ad860ba275294ebb4ebbecb9da83318 with the sibling PRs of this batch: static gates green; changed-area node:test 293/294 (0 failing) + vitest 482/482 (fast parity — full suite ran today on the tip via the base-red and 3b trains). Merged --admin per merge-gates §7. |
95b2e53
into
cheaper-inference:release/v3.8.51
…heaper-inference#13673) * feat(usage): generic billing/quota for openai-compatible connections (cheaper-inference#13616) Every other fetcher in services/usage hard-codes one upstream's URL, auth and response shape, which works because those providers are known. An openai-compatible connection can point at anything, and its id is minted per connection -- so it can never be a member of USAGE_SUPPORTED_PROVIDERS or a case in the dispatcher switch. So the shape comes from the connection instead. `providerSpecificData. quotaEndpoint` declares the url, auth mode, optional headers, and a mapping of dot-paths onto UsageQuota: { "url": "...", "auth": "bearer", "quotas": { "credits": { "used": "$.data.used_usd", "total": "$.data.limit_usd", "currency": "USD" } } } Dot/bracket paths (`$.a.b[0].c`) rather than full JSONPath, so the mapping stays dependency-free and legible in a config field. Three decisions worth stating: - **An unresolvable mapping reports nothing, never 0/0.** A quota reading 0 of 0 renders as fully exhausted, and an operator would act on that. A typo'd path must produce no card, not a fake outage. - **The transport error is not echoed.** The url is operator-supplied and can carry a query-string secret; the message says "unreachable" and nothing more. - **The capability is read off the connection, not the id.** `supportsProviderQuota` already takes the connection and already has a connection-shaped check (moonshot), so the gate goes there. A declared url with no `quotas` mapping does NOT count as supported: it can be fetched but can never yield a quota, and would leave a permanently empty card in Provider Limits. Verified: 7 new tests; mutations each killed by the right one -- let an unresolved mapping fall through to 0/0 -> that test alone fails echo the transport error -> that test alone fails 121 tests pass across this file, usage-families-split, provider-plugin- manifest, provider-limits* and the quota-visibility suites (the drift guard from cheaper-inference#13134 included). eslint clean on all four files; the three no-unused-vars errors in usage.ts are byte-identical on the base branch. * fix(usage): bound the openai-compatible quota fetch with a timeout An operator-configured quotaEndpoint that never responds would hang getOpenAiCompatibleUsage()'s fetch() indefinitely, stalling that connection's Provider Limits sync. Same 15s bound as the other fetchers in this directory (grokResetCredits.ts's FETCH_TIMEOUT_MS). Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * test(usage): pin the openai-compatible quota fetch timeout A quota endpoint that accepts the connection and never answers must be aborted by the fetch signal instead of hanging the Provider Limits sync. Fails without the AbortSignal.timeout() bound, passes with it. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: abhisheksharma2411 <abhisheksharma2411@users.noreply.github.com>
Closes #13616.
Why it can't be done the usual way
Every other fetcher in
services/usage/hard-codes one upstream's URL, auth and response shape — fine, because those providers are known. An openai-compatible connection can point at anything, and its id is minted per connection (openai-compatible-chat-<generated>), so it can never be a member ofUSAGE_SUPPORTED_PROVIDERSor acasein the dispatcher switch.So the shape comes from the connection.
providerSpecificData.quotaEndpoint:{ "url": "https://api.example.com/v1/credits", "auth": "bearer", "plan": "$.data.plan", "quotas": { "credits": { "used": "$.data.used_usd", "total": "$.data.limit_usd", "resetAt": "$.data.renews_at", "currency": "USD" } } }Dot/bracket paths (
$.a.b[0].c) rather than full JSONPath — no new dependency, and legible in a config field.Three decisions worth arguing
An unresolvable mapping reports nothing, never
0/0. This is the one I'd defend hardest.createQuotaFromUsage(undefined, undefined, …)yields 0 used of 0 total, which renders as a fully exhausted quota — so a typo in a path would look like an outage, and an operator would act on it. A mapping that resolves neither a total nor a remaining produces no card at all.The transport error is never echoed. The URL is operator-supplied and can carry a query-string secret. On a fetch throw the message is
"Quota endpoint unreachable."and nothing else.The capability is read off the connection, not the provider id.
supportsProviderQuotaalready takes the connection and already has a connection-shaped check (isMoonshotOpenPlatformConnection), so the gate belongs there. A declaredurlwith an emptyquotasobject does not count as supported — it can be fetched but can never yield a quota, and would leave a permanently empty card in Provider Limits.Verification
7 new tests, and the two guards above are mutation-checked:
0/0usage-families-split,provider-plugin-manifest,provider-limits*and the quota-visibility suites — including thefetcherProviders↔supportedProvidersdrift guard from test(usage): pin fetcherProviders against supportedProviders #13134, which is unaffected because a prefix provider is in neither list by design.eslintexit 0 on all four changed files.usage.tsreports threeno-unused-vars— byte-identical on the base branch, line numbers shifted by one because of my import.features/.Not included
No UI for editing the mapping — it's set through
providerSpecificDatalike the other connection-specific fields. If you'd want a form for it in the provider-node editor, that's a natural follow-up but it's a separate change and I didn't want to guess at the shape.