Skip to content

feat: add Laya gateway and OSS classifier providers - #43626

Merged
tin-berri merged 2 commits into
mainfrom
litellm_laya_gateway_classifier
Oct 2, 2026
Merged

tin-berri merged 2 commits into
mainfrom
litellm_laya_gateway_classifier

Conversation

@tin-berri

@tin-berri tin-berri commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Laya cannot serve native decisions or classify Auto Router requests
  • A Jev-specific configuration cannot describe multiple OSS classifier providers

How it solves it:

  • Add Laya gateway support and shared OSS classifier configuration
  • Keep legacy Jev inputs and TypeSafe accounting identities compatible

User Flow

Before: creating a router with OSS configuration returns 400

  1. Configure the gateway's Laya endpoint and existing solver models
  2. POST /model/new with classifier_type: oss_classifier and opensource_classifier_config.provider: laya
  3. The gateway rejects the classifier type

After: the router saves and classifies through Laya

  1. Configure the gateway's Laya endpoint and existing solver models
  2. POST /model/new with classifier_type: oss_classifier and opensource_classifier_config.provider: laya
  3. The gateway returns 200; Test Routing selects a solver using Laya

Use provider: jev for Jev or compatible self-hosted classifiers

Relevant issues

  • Add native Laya decisions and shared Jev/Laya classifier settings
  • Backend for UI #43768; merge this first
  • Combined classifier guide: docs #1907

Changes

  • Add /laya/v1/systemone, checkpoint authorization and usage accounting
  • Write oss_classifier and opensource_classifier_config, accepting legacy Jev type/block and TypeSafe provider aliases
  • Normalize configuration before credential merging, permissions, dependency checks and customization counting
  • Reject dual configuration blocks; retain hidden credentials for the same provider and clear incompatible connections when switching
  • Keep Jev model permissions, pricing and routing identities under typesafe/; Laya uses laya/<checkpoint>
  • Pin metadata.model_group to the server-resolved model on every built-in pass-through route, so a client cannot choose its own per-model budget key; user-defined routes are unchanged

Screenshots / Proof of Fix

Real Laya English checkpoint and PostgreSQL-backed gateways, with LAYA_API_BASE pointing at the local model server

classifier_type: oss_classifier
opensource_classifier_config:
  provider: laya
  model: english
  timeout_ms: 15000
tiers:
  SIMPLE: small-solver

Before (ac8c5aa)

Saved classifier

  1. POST /model/new with the configuration above
  2. Receive 400: unsupported classifier type

Native decisions

  1. POST /laya/v1/systemone with model: english, state Hello, and a SIMPLE greeting choice
  2. Receive 404

After (c4d5670)

Saved classifier

  1. POST the same model configuration: 200; stored fields use the canonical names
  2. POST /auto_router/test_routing for the saved router: 200; laya/english selects small-solver
  3. Edit a real legacy Jev row with canonical fields: 200; its saved endpoint/key survive and the row uses canonical fields
  4. Supply both configuration blocks, including a null legacy block: 400 before persistence

Native decisions

  1. POST the same native decision: 200 with a SIMPLE answer and measured usage
  2. Repeat with a key lacking Laya checkpoint access: 403

Validation: 2,407 focused backend tests and make check pass; 12 production-Prisma customization counts agree with the runtime predicate. Alias and create-persistence mutations are caught by the tests

Caveats

Medium

  • Documentation CI requires docs #1907 to merge first

Low

  • Hosted Jev was not called without a configured TypeSafe key
  • Jev compatibility used the real local Laya server; pricing has runtime regression coverage
  • Untouched legacy rows retain their stored format until edited
  • Pass-through spend logs now carry the request model as model_group where they previously carried an empty string

Note

Medium Risk
Touches auth, pass-through budgeting/accounting, and classifier credential merging across auto-router admin and member APIs; misconfiguration could affect model access enforcement or spend attribution on built-in pass-through routes.

Overview
Adds Laya as a first-class proxy provider: native POST /laya/v1/systemone pass-through to a self-hosted server (LAYA_API_BASE / optional LAYA_API_KEY), with strict checkpoint validation (english, multilingual, typed-decisions), no streaming/custom_body, and auth/accounting keyed as laya/<checkpoint> (including routing metadata from responses).

Refactors complexity-router Jev-only settings into classifier_type: oss_classifier and opensource_classifier_config with provider: jev | laya, while accepting legacy jev / jev_classifier_config / typesafe aliases. Shared resolve_complexity_router_config_write merges stored connection fields safely on admin writes and blocks team members from changing classifier api_base/api_key; model create/patch responses can redact classifier secrets for member flows.

Laya can drive the same System One HTTP classifier client as TypeSafe/Jev, with separate pricing rows and logging. Built-in pass-through routes now set metadata.model_group from the server-resolved LiteLLM model (and strip untrusted client pricing overrides where applicable); user-defined pass-through behavior for per-model budgets stays off.

Reviewed by Cursor Bugbot for commit 9018785. Bugbot is set up for automated code reviews on this repo. Configure here.

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.76033% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...roxy/management_helpers/auto_router_permissions.py 95.00% 1 Missing ⚠️
...er_strategy/complexity_router/complexity_router.py 90.00% 1 Missing ⚠️
...itellm/router_strategy/complexity_router/config.py 98.71% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High risk] Adds new provider integration and renames classifier configuration.

The PR appears safe to merge based on the findings assessed here.

Summary

The PR adds native Laya decisions and shared Jev/Laya classifier configuration while retaining legacy Jev compatibility.

  • Both Laya provider-catalog links now use the shared classifier guide.
  • No references to the removed Laya cookbook remain.

Reviews (14) · Last reviewed commit: "feat: add Laya gateway and classifier ba..."

Comment thread litellm/router_strategy/complexity_router/config.py
Comment thread ui/litellm-dashboard/src/components/add_model/jev_classifier_config.ts Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@codspeed

codspeed Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_laya_gateway_classifier (9018785) with main (19da815)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (6c2ede0) during the generation of this report, so 19da815 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please review b8074dd again; both findings are addressed, with live reset verification and regression coverage for explicit connection clearing

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

Comment thread ui/litellm-dashboard/src/components/add_model/JevClassifierConfig.tsx Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please review f4b1df1; the remaining URL-edit finding is fixed and verified through the live dashboard, PostgreSQL and a regression mutation

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@tin-berri
tin-berri marked this pull request as ready for review September 29, 2026 20:02
@tin-berri
tin-berri requested a review from a team September 29, 2026 20:02
@tin-berri
tin-berri force-pushed the litellm_laya_gateway_classifier branch from f4b1df1 to bacd71d Compare September 29, 2026 21:20
@tin-berri tin-berri changed the title feat: add Laya gateway and decision classifier support feat: add Laya gateway and classifier backend Sep 29, 2026
@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please review bacd71d, the backend-only split; all dashboard behavior now lives in companion UI PR #43768

Comment thread cookbook/laya/README.md Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@tin-berri
tin-berri force-pushed the litellm_laya_gateway_classifier branch from e6c8f8a to e3ff388 Compare October 1, 2026 20:49
@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai Please review e3ff388 after removing the duplicate cookbook and consolidating its native API example into the shared docs guide

@tin-berri

Copy link
Copy Markdown
Contributor Author

@cursor review Please refresh your result for e3ff388; this follow-up removes duplicated documentation, and runtime source is unchanged

Comment thread provider_endpoints_support.json Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@tin-berri
tin-berri force-pushed the litellm_laya_gateway_classifier branch from e3ff388 to c4d5670 Compare October 1, 2026 20:56
@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai Please verify c4d5670: both provider catalog links now use the shared classifier guide, with no remaining cookbook references

@tin-berri

Copy link
Copy Markdown
Contributor Author

@cursor review Please refresh c4d5670 after two documentation URL corrections; runtime code is identical to your clean e3ff388 review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

MockRequest in tests/pass_through_unit_tests gains an httpx.URL and an ASGI
scope, which get_request_route now reads inside
_init_kwargs_for_pass_through_endpoint, and the POST-only /laya/v1/systemone
route joins the protocol-constrained exemptions. A built-in pass-through pins
metadata.model_group to the resolved model so a client cannot choose its own
per-model budget key; test_pass_through_endpoints now proves that on a
non-Laya route and drops a duplicated assertion.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 9018785. Configure here.

@tin-berri
tin-berri merged commit 0dc2340 into main Oct 2, 2026
103 of 107 checks passed
@tin-berri
tin-berri deleted the litellm_laya_gateway_classifier branch October 2, 2026 18:29
yuneng-berri added a commit that referenced this pull request Oct 2, 2026
… ASGI scope

#43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake
yuneng-berri added a commit that referenced this pull request Oct 2, 2026
* test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG

The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context

* test(secret-detection): give the hand-built redaction request an ASGI path

Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path

* test(integration): isolate litellm callback lists per sdk test

usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards

* test(integration): keep the owner-lookup fault proxy off the shared read replica

The owned proxy points DATABASE_URL at a scratch database but inherited
DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared
database and rejected the freshly created key with token_not_found_in_db.
Drop the replica variable like the other scratch-database owned proxies

* test(integration): request every seeded key in the team owner breakdown

The aggregated team activity endpoint now caps breakdown.api_keys at the top
100 keys by default (#43398), so the 300 seeded keys came back as 100 rows.
The test guarantees each key is reported with its own owner, so ask for an
api_key_limit that covers all seeded keys

* test(integration): give every owned Redis its own port in the redis-cache container

On CircleCI every owned Redis ran on the fixed port 16379 inside the shared
redis-cache container. When an earlier server still held that port, the new
one failed to bind, readiness pinged the old server, the pidfile read failed
and cleanup then reported "Owned Redis still serves after shutdown"

Reserve an ephemeral port for the docker-exec path the same way the local
binary path already does, and refuse to start when something already serves
the chosen port so the failure names the real cause

* test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach

The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail

* test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves

vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to

* test(e2e): check only stored message content for a leaked card number

The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there

* test(e2e): assert the proxy decodes token-array embeddings for titan

The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer

* test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking

us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5

* test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests

Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed

* test(e2e): cite the tokenizer and date behind the titan token-array fixture

* test(e2e): let migration seed replicas finish their request-log indexes before cloning

Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema

* test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope

#43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake

* test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list

test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly
yuneng-berri added a commit that referenced this pull request Oct 2, 2026
…) (#44254)

* test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG

The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context

* test(secret-detection): give the hand-built redaction request an ASGI path

Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path

* test(integration): isolate litellm callback lists per sdk test

usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards

* test(integration): keep the owner-lookup fault proxy off the shared read replica

The owned proxy points DATABASE_URL at a scratch database but inherited
DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared
database and rejected the freshly created key with token_not_found_in_db.
Drop the replica variable like the other scratch-database owned proxies

* test(integration): request every seeded key in the team owner breakdown

The aggregated team activity endpoint now caps breakdown.api_keys at the top
100 keys by default (#43398), so the 300 seeded keys came back as 100 rows.
The test guarantees each key is reported with its own owner, so ask for an
api_key_limit that covers all seeded keys

* test(integration): give every owned Redis its own port in the redis-cache container

On CircleCI every owned Redis ran on the fixed port 16379 inside the shared
redis-cache container. When an earlier server still held that port, the new
one failed to bind, readiness pinged the old server, the pidfile read failed
and cleanup then reported "Owned Redis still serves after shutdown"

Reserve an ephemeral port for the docker-exec path the same way the local
binary path already does, and refuse to start when something already serves
the chosen port so the failure names the real cause

* test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach

The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail

* test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves

vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to

* test(e2e): check only stored message content for a leaked card number

The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there

* test(e2e): assert the proxy decodes token-array embeddings for titan

The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer

* test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking

us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5

* test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests

Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed

* test(e2e): cite the tokenizer and date behind the titan token-array fixture

* test(e2e): let migration seed replicas finish their request-log indexes before cloning

Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema

* test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope

#43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake

* test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list

test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly

(cherry picked from commit 9b5562f)

This branch is waiting to be deployed

1 waiting deployment
e2e-changed — 90187852 Waiting Oct 2, 2026 by tin-berri via oauth #2617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants