Skip to content

feat(ai): add LLM Gateway and LLM Gateway DevPass providers - #7610

Open
RATCHAW wants to merge 12 commits into
earendil-works:mainfrom
RATCHAW:add-llgateway-provider
Open

RATCHAW wants to merge 12 commits into
earendil-works:mainfrom
RATCHAW:add-llgateway-provider

Conversation

@RATCHAW

@RATCHAW RATCHAW commented Aug 4, 2026

Copy link
Copy Markdown

Adds LLM Gateway (an OpenRouter-style router) as built-in openai-completions providers, contributed on behalf of the LLM Gateway team.

Replaces #7480, which was auto-closed seconds after opening and stopped tracking the branch, so its diff never showed anything past the first commit. This one carries the complete work.

Two providers ship here. They share an endpoint, a request shape and an attribution header; only the account behind the key differs, which is what decides how usage is billed:

Provider Billing
llmgateway Pay-as-you-go from LLM Gateway credits
llmgateway-devpass The DevPass flat-rate coding subscription

What's included

  • Catalogue: fetchLlmGatewayModels() sources the public https://api.llmgateway.io/v1/models (~152 tool-capable models). Models are kept only when at least one provider mapping declares tools: true — the gateway's routing rejects tool calls to mappings without it, and its supported_parameters field is unreliable in both directions. Thinking levels are derived from the mappings' declared reasoning_efforts when they agree (e.g. gpt-5.4-pro exposes only medium/high/xhigh; claude models gain xhigh/max). One fetch builds both catalogues.
  • Compat (mirrored in both detectors): the gateway only honors max_tokens (max_completion_tokens is silently dropped), strips store and function.strict (so supportsStrictMode: false), and honors anthropic cache_control on claude-* routed models.
  • Auth: LLMGATEWAY_API_KEY / LLMGATEWAY_DEVPASS_API_KEY, plus a browser sign-in flow modeled on the OpenRouter one — LLM Gateway has no OAuth token endpoint; its dashboard mints an API key delivered to a state-checked loopback callback (source=pi-agent is whitelisted on their side). Minted keys live 90 days and can't be refreshed; on expiry the API 401s and the user re-runs /login.
  • Overflow: pattern for the gateway's pre-flight "exceeds the configured context size" error.
  • Session affinity and usage: the gateway pins sticky-session routing on x-session-id (the OpenRouter format rather than the OpenAI one), and normalizes reasoning_tokens to the top level of usage while omitting completion_tokens_details, so reasoning falls back to that field.
  • Attribution: every generated model carries a static x-source: pi-agent header (same mechanism as the Copilot/Kimi static headers), so LLM Gateway attributes the traffic to Pi like it does for other coding agents.
  • Tests: OAuth-flow suites for both providers (16 + 6 cases, incl. forged-callback handling and the DevPass catalogue gate), adapter/env-key/overflow cases, session-affinity and usage-normalization cases, and LLMGATEWAY_API_KEY-gated e2e blocks in stream and context-overflow.

DevPass specifics

  • The sign-in flow is parameterised by organisation: org=devpass mints the key in the user's personal DevPass org so usage bills the subscription, org=default keeps minting in their dashboard org. Everything else about the request is identical.
  • Credentials are scoped per provider (separate auth.json entry, separate env var) so a pay-as-you-go key can't silently bill through the provider labelled as the subscription, or the reverse.
  • DevPass keys get a 403 for models a coding plan doesn't cover, so the DevPass catalogue is narrowed with the gateway's own predicate — paid and stable, with a mapping that has tools, streaming and cached-input pricing (prompt caching is a hard requirement of the flat-rate plan). That's 142 of the 152 models; the excluded ones are free or unstable.
  • Prices are kept on the DevPass models rather than zeroed, matching how Pi already treats the GitHub Copilot and Codex subscriptions. They read as relative model cost, not as subscriber spend.

Notes for review

  • Uptime: committing the two *.models.ts shards makes hydrate:model-data --strict depend on api.llmgateway.io being reachable, like the other live-fetched catalogues. The endpoint is public/unauthenticated.
  • Duplicated predicate: /v1/models has no coding-plan filter, so the DevPass gate mirrors isCodingModel from the gateway's own source (theopenco/llmgateway, packages/shared/src/coding-models.ts). If those rules change, the catalogue drifts until Pi regenerates. Adding a filter parameter upstream would remove the duplication; happy to follow up with that.
  • The gateway's own API is used rather than models.dev (which also lists llmgateway) for the richer metadata: per-mapping tool/reasoning capability, deprecation, and cache pricing.
  • Per-model output limits come from the catalogue's max_output (added in feat: expose max_output in /v1/models theopenco/llmgateway#3379, now deployed), falling back to 4096 — the gateway 400s on an over-limit max_tokens, so the fallback stays conservative.

🤖 Generated with Claude Code

RATCHAW and others added 4 commits August 2, 2026 12:35
Add llmgateway.io as a built-in OpenAI-completions provider, mirroring
the OpenRouter integration:

- catalogue fetcher sources https://api.llmgateway.io/v1/models,
  keeping models with at least one tool-capable provider mapping
  (routing rejects tool calls to mappings without tools support) and
  deriving thinking levels from the mappings' declared reasoning
  efforts when they agree
- compat: the gateway only honors max_tokens (max_completion_tokens is
  silently dropped), strips store and function.strict, and accepts
  anthropic cache_control on claude-routed models
- auth: LLMGATEWAY_API_KEY plus a browser sign-in flow that mints an
  API key via the gateway's loopback callback (state-checked; minted
  keys live 90 days and cannot be refreshed)
- overflow: detect the gateway's pre-flight context-size error

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LLM Gateway attributes traffic per coding agent via the x-source
request header (pi-agent is registered on their side); stamp it on
every generated model like the Copilot and Kimi static headers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DevPass is LLM Gateway's flat-rate coding subscription. It shares the
endpoint, request shape and x-source attribution of the pay-as-you-go
llmgateway provider; only the account behind the key differs, so it
ships as a sibling provider rather than a mode of the existing one:

- auth: the browser sign-in flow is now parameterised by organisation.
  org=devpass mints the key in the user's personal DevPass org (usage
  bills the subscription), org=default keeps minting in their dashboard
  org as before. Credentials and LLMGATEWAY_DEVPASS_API_KEY are scoped
  per provider so one key cannot silently bill the wrong plan.
- catalogue: DevPass keys are answered with 403 for models a coding
  plan does not cover, so the fetcher narrows the DevPass catalogue
  with the gateway's own predicate (paid and stable, with a mapping
  that has tools, streaming and cached-input pricing). Prompt caching
  is a hard requirement of the flat-rate plan, hence the price check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both behaviors ship in this branch without direct coverage:

- the gateway pins sticky-session routing on x-session-id, so it takes
  the openrouter session-affinity format rather than the openai one
- it normalizes reasoning_tokens to the top level of usage and omits
  completion_tokens_details, so reasoning falls back to that field

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

This PR was auto-closed. Only contributors approved with lgtm can open PRs. Open an issue first and ask a maintainer for approval.

Maintainers review auto-closed issues daily. Issues that do not meet the quality bar in CONTRIBUTING.md will not be reopened or receive a reply.

If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open. The command must be at the start of the reply (optionally after one or more @username mentions) or at the end.

See CONTRIBUTING.md.

@github-actions github-actions Bot closed this Aug 4, 2026
RATCHAW and others added 3 commits August 4, 2026 18:03
Adds LLM Gateway to the cross-provider e2e batteries every built-in
provider participates in (abort, empty messages, token stats, tool calls
without results, unicode surrogates, tool result images, total tokens,
cross-provider handoff), plus an offline catalog test covering the
x-source attribution header, the anthropic cache-control routing for
claude-backed models, and the DevPass catalog narrowing.

Live blocks are gated on LLMGATEWAY_API_KEY and skip without it.
# Conflicts:
#	packages/ai/README.md
#	packages/ai/scripts/generate-models.ts
#	packages/ai/test/cross-provider-handoff.test.ts
@badlogic badlogic reopened this Aug 4, 2026
RATCHAW and others added 5 commits August 6, 2026 12:25
# Conflicts:
#	packages/ai/src/api/openai-completions.ts
#	packages/ai/test/oauth-auth.test.ts
# Conflicts:
#	packages/ai/scripts/generate-models.ts
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolve the session affinity conflict in openai-completions.ts by
enabling the headers for OpenRouter (now the upstream default) and
LLM Gateway. Adapt tests to upstream changes: stub the LLM Gateway
catalog URL in the Fireworks model generation harness, and pass a
normalized transcript context in the usage parsing test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- generate-models: keep LLM Gateway fetcher alongside main's Radius and
  classifier catalogs; exclude LLM Gateway from strict mode next to Cerebras
- openai-completions: take main's runtime supportsStrictMode default (false),
  which already covers LLM Gateway
- llmgateway oauth: import page helpers from utils/oauth-page.ts (moved upstream)
- regenerate models.generated.ts and llmgateway catalogs for the new
  chat/image/classifier catalog split
- providers.md: re-add LLM Gateway env vars and login notes to main's rewrite
- tests: stub LLM Gateway fetch in fireworks generation test; generated compat
  no longer carries supportsStrictMode: false since it matches the runtime default

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants