fix(together_ai): default endpoints to api.together.ai instead of api.together.xyz by mateo-berri · Pull Request #38233 · BerriAI/litellm

mateo-berri · 2026-08-25T17:20:19Z

TLDR

Problem this solves:

Together AI's canonical host is now api.together.ai
LiteLLM defaults still point at legacy api.together.xyz
Rerank hardcodes the legacy host, ignoring api_base and TOGETHER_AI_API_BASE
Passing either Together host as api_base fails provider resolution

How it solves it:

Default provider api_base and rerank URL to https://api.together.ai/v1
Rerank now honors api_base and TOGETHER_AI_API_BASE like chat
Both hosts map to together_ai when passed as api_base
Deletes a dead models/info fetch against the legacy host

User Flow

Before: a developer moving their LiteLLM gateway to Together's new api.together.ai host finds rerank pinned to the legacy host and bare-model api_base requests rejected

They set TOGETHER_AI_API_BASE=https://api.together.ai/v1 and restart their proxy with a together_ai/ chat model and a together_ai/ rerank model configured
They send POST https://litellm-domain/v1/chat/completions with {"model": "together-gemma", ...} and get a 200; Together's request log shows the call arriving on api.together.ai
They send POST https://litellm-domain/v1/rerank with {"model": "together-rerank", "query": ..., "documents": [...]} and the request goes out to https://api.together.xyz/v1/rerank anyway; an egress rule that only allows api.together.ai blocks every rerank call
In an app using the SDK directly they call litellm.completion(model="google/gemma-3n-E4B-it", api_base="https://api.together.xyz/v1") and get back a 400 "LLM Provider NOT provided"
With no env var set at all, every Together request defaults to the legacy api.together.xyz host

After: the same setup sends every Together call to the host the developer chose, and both Together hosts resolve as api_base

They set TOGETHER_AI_API_BASE=https://api.together.ai/v1 and restart their proxy with the same config
They send POST https://litellm-domain/v1/chat/completions with {"model": "together-gemma", ...} and get a 200 arriving on api.together.ai
They send POST https://litellm-domain/v1/rerank with the same body and the request now goes out to https://api.together.ai/v1/rerank; pointing the env var back at https://api.together.xyz/v1 moves chat and rerank there together
The same litellm.completion(model="google/gemma-3n-E4B-it", api_base="https://api.together.xyz/v1") call returns a completion routed to Together, and so does api_base="https://api.together.ai/v1"
With no env var set, every Together request defaults to the current api.together.ai host

Relevant issues

Linear ticket

Resolves LIT-5962

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxies, real TOGETHER_API_KEY, no mocks, both legs booted with --num_workers 2 (2 uvicorn workers, no database). Before leg from a worktree at the merge base 1d695a7 on port 41227; after leg from the PR tip 5e6b6c6 on port 57971. Config together_qa_config.yaml:

model_list:
  - model_name: together-gemma
    litellm_params:
      model: together_ai/google/gemma-3n-E4B-it
      api_key: os.environ/TOGETHER_API_KEY
  - model_name: together-gemma-xyz-pinned
    litellm_params:
      model: together_ai/google/gemma-3n-E4B-it
      api_base: https://api.together.xyz/v1
      api_key: os.environ/TOGETHER_API_KEY
  - model_name: together-rerank
    litellm_params:
      model: together_ai/mixedbread-ai/mxbai-rerank-large-v2
      api_key: os.environ/TOGETHER_API_KEY

general_settings:
  master_key: sk-litellm-together-qa

Together currently has zero serverless rerank models (GET https://api.together.ai/v1/models lists Salesforce/Llama-Rank-V1 and mixedbread-ai/mxbai-rerank-large-v2 as dedicated-only), so every live rerank call returns Together's genuine 400 model_not_available. That real provider response plus the proxy debug log is the host evidence: the request authenticated against the real Together API on the host under test. Chat calls are full live completions.

Before (`1d695a7`, merge base)

chat defaults to the legacy host

curl -s http://localhost:41227/v1/chat/completions -H "Authorization: Bearer sk-litellm-together-qa" -H "Content-Type: application/json" -d '{"model":"together-gemma","messages":[{"role":"user","content":"say hi in 3 words"}],"max_tokens":30}'
200 with content "Hello, how can I help?"; debug log shows the outbound POST going to https://api.together.xyz/v1/

rerank defaults to the legacy host

curl -s http://localhost:41227/v1/rerank -H "Authorization: Bearer sk-litellm-together-qa" -H "Content-Type: application/json" -d '{"model":"together-rerank","query":"what is reranking","documents":["reranking orders documents by relevance","bananas are yellow"]}'
Together's real 400 model_not_available comes back; debug log shows the outbound POST going to https://api.together.xyz/v1/rerank

per-deployment api_base pins the legacy host (unchanged by this PR)

Same chat curl with "model":"together-gemma-xyz-pinned" (deployment carries api_base: https://api.together.xyz/v1)
200 with a real completion; debug log shows https://api.together.xyz/v1/

rerank ignores TOGETHER_AI_API_BASE

Restart the proxy with TOGETHER_AI_API_BASE=https://api.together.ai/v1
Re-run the chat curl: debug log now shows https://api.together.ai/v1/ (chat honors the env var)
Re-run the rerank curl: debug log STILL shows https://api.together.xyz/v1/rerank (rerank hardcodes the host)

bare model + Together api_base fails provider resolution

litellm.completion(model="google/gemma-3n-E4B-it", api_base="https://api.together.xyz/v1", messages=[...]) from the SDK
Raises litellm.exceptions.LiteLLMUnknownProvider: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=google/gemma-3n-E4B-it, custom_llm_provider=None (the legacy host sits in the endpoint list with no provider mapping); with api_base=https://api.together.ai/v1 it raises litellm.BadRequestError: LLM Provider NOT provided. ... You passed model=google/gemma-3n-E4B-it (the current host is not recognized at all)

After (`5e6b6c6`)

chat defaults to the current host

Same chat curl against the after-leg proxy on port 57971 (no env var set)
200 with content "Hello there!"; debug log shows the outbound POST going to https://api.together.ai/v1/

rerank defaults to the current host

Same rerank curl
Together's real 400 model_not_available comes back (its message now even links https://api.together.ai/models/...); debug log shows the outbound POST going to https://api.together.ai/v1/rerank

rerank honors TOGETHER_AI_API_BASE

Restart the proxy with TOGETHER_AI_API_BASE=https://api.together.xyz/v1 (inverted on purpose: the env var now points at the legacy host)
Chat curl: debug log shows https://api.together.xyz/v1/
Rerank curl: debug log shows https://api.together.xyz/v1/rerank; both surfaces follow the env var, and the legacy host still works end to end

bare model + Together api_base resolves to together_ai

litellm.completion(model="google/gemma-3n-E4B-it", api_base="https://api.together.xyz/v1", messages=[{"role":"user","content":"say hi in 3 words"}], max_tokens=30) from the SDK with only TOGETHER_API_KEY in the env
Returns a real completion "Hello there!" with model=together_ai/google/gemma-3n-E4B-it; same call with api_base="https://api.together.ai/v1" also returns a real completion

per-deployment api_base still pins the legacy host

curl -s http://localhost:57971/v1/chat/completions ... -d '{"model":"together-gemma-xyz-pinned", ...}' (deployment carries api_base: https://api.together.xyz/v1)
200 with a real completion; debug log shows the outbound POST going to https://api.together.xyz/v1/

QA run notes:

Together's serverless catalog currently lists zero rerank models
Router retries rerank's 400 three times; pre-existing, unchanged here
Bare-model config entries with api_base still fail; tracked separately

Type

🐛 Bug Fix

Caveats (if any)

api.together.xyz still resolves and keeps working when pinned
Proxy config entries with a bare model plus api_base fail deployment validation for every endpoint-matched provider (groq and deepseek fail identically); pre-existing router behavior, tracked separately

Rerank credential model

Together rerank keeps rerank_api/main.py's uniform credential precedence: api_base resolves dynamic/config/global/default and the key falls back to the Together env key, exactly as the cohere, infinity, and bedrock branches and Together chat already do. A caller cannot select the base on the proxy: api_base sits in _BANNED_REQUEST_BODY_PARAMS and is_request_body_safe (litellm/proxy/auth/auth_utils.py) rejects it with a 400 on every authed route via pre_db_read_auth_checks, unless the admin opts in through allow_client_side_credentials or a deployment's configurable_clientside_auth_params (the huntr 4001e1a2 hardening). SDK callers own the process env that holds the key. So the env-key fallback only ever pairs with an admin-selected base, which is the mainline this PR enables: a config api_base plus TOGETHER_API_KEY in the env

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
5e6b6c6 passes /live-pr-risk

Note

Medium Risk
Changes default outbound URLs and provider inference for Together (chat and rerank), which can affect egress allowlists and deployments that relied on implicit .xyz defaults; legacy host pinning and env overrides remain supported and tests were added.

Overview
Aligns Together AI routing with the api.together.ai canonical host while keeping api.together.xyz working when explicitly configured.

Provider resolution: Adds api.together.ai/v1 to openai-compatible endpoints and maps both Together hosts to together_ai in endpoint-based get_llm_provider, including API key resolution from the existing Together env vars. The default together_ai api_base is now https://api.together.ai/v1 instead of the legacy .xyz URL.

Rerank: Stops hardcoding https://api.together.xyz/v1/rerank; rerank builds the URL from api_base (sync and async). rerank_api/main.py passes through dynamic_api_base / TOGETHER_AI_API_BASE with the same default as chat.

Cleanup: Removes unused Together-specific get_model_info / legacy prompt-template code from prompt_templates/factory.py that called the old models/info endpoint.

Regression tests cover provider resolution and rerank default/custom/env api_base behavior.

^{Reviewed by Cursor Bugbot for commit 5e6b6c6. Bugbot is set up for automated code reviews on this repo. Configure here.}

….together.xyz Together AI moved its canonical API host from api.together.xyz to api.together.ai. Default the provider api_base and the rerank handler to the new host, make rerank honor api_base and TOGETHER_AI_API_BASE like chat already does, map both hosts to together_ai when passed as api_base, and delete the dead models/info fetch in factory.py.

greptile-apps · 2026-08-25T17:24:33Z

Greptile Summary

The PR migrates Together AI defaults to the canonical .ai host while preserving legacy-host routing.

Adds provider resolution for both Together hosts and ensures an explicit key takes precedence over environment credentials.
Makes synchronous and asynchronous rerank requests honor the resolved API base.
Removes the unused legacy model-information fetch and adds host-resolution and rerank regression tests.

Confidence Score: 3/5

The PR does not yet appear safe to merge because the previously reported credential and endpoint isolation failure remains reachable.

The rerank path still allows a selected API base to determine the authenticated POST destination while the credential can independently fall back to the server’s Together environment key.

Files Needing Attention: litellm/rerank_api/main.py and litellm/llms/together_ai/rerank/handler.py

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/get_llm_provider_logic.py	Maps both Together hosts to the provider, defaults to the canonical host, and preserves explicit API-key precedence.
litellm/llms/together_ai/rerank/handler.py	Routes rerank calls through the selected API base, but the previously reported credential and endpoint isolation issue remains.
litellm/rerank_api/main.py	Resolves a configurable Together rerank base while retaining the previously reported unsafe pairing with server-resolved credentials.
litellm/constants.py	Registers the canonical Together endpoint alongside the supported legacy endpoint.
litellm/litellm_core_utils/prompt_templates/factory.py	Removes an unused Together model-information request against the legacy host.
tests/test_litellm/litellm_core_utils/test_get_llm_provider_endpoint_match.py	Covers both Together host mappings, explicit-key precedence, and the canonical default.
tests/test_litellm/rerank_api/test_main.py	Covers default, explicit, and environment-selected rerank API bases with mocked HTTP requests.

_{Reviews (4): Last reviewed commit: "fix(together_ai): let an explicit api_ke..." | Re-trigger Greptile}

codecov · 2026-08-25T17:31:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…n api_base match

mateo-berri · 2026-08-25T17:40:54Z

@greptileai On the summary security concern: every rerank provider already pairs caller api_base with the server-key fallback, matching chat's trust model.

greptile-apps · 2026-08-25T17:41:42Z

That’s fair. Re-checking the flow, Together rerank resolves api_base from the provider-resolved caller base, the per-call base, the global base, or the Together default, while api_key uses the provider-resolved/caller key before the configured Together key and environment fallback. This matches the existing chat trust model. The new endpoint mapping also uses exact host/path matching, so an arbitrary caller URL cannot trigger Together’s server-key fallback and receive that key. I withdraw the summary security concern; no blocking security finding here.

mateo-berri · 2026-08-25T17:47:54Z

@greptileai

mateo-berri · 2026-08-25T17:52:42Z

@greptileai

mateo-berri · 2026-08-25T18:03:38Z

@greptileai

mateo-berri · 2026-08-25T18:08:27Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5e6b6c6. Configure here.}

codspeed-hq · 2026-08-25T18:23:30Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_together_api_host (5e6b6c6) with litellm_internal_staging (bb27bfd)¹}

No successful run was found on litellm_internal_staging (92fe358) during the generation of this report, so bb27bfd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…ges surfaces Adds streaming, async, /v1/responses, and /v1/messages coverage for the Together AI overhaul (BerriAI#38233, BerriAI#38248, BerriAI#38230, BerriAI#38265, BerriAI#38275), plus the legacy api.together.xyz host and TOGETHER_AI_API_BASE through litellm.completion. Each new test fails under a one-line mutation of the merged code.

greptile-apps Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/get_llm_provider_logic.py

Comment thread litellm/litellm_core_utils/get_llm_provider_logic.py

Comment thread litellm/llms/together_ai/rerank/handler.py Outdated

fix(together_ai): let an explicit api_key beat the Together env key o…

5e6b6c6

…n api_base match

cursor Bot reviewed Aug 25, 2026

View reviewed changes

mateo-berri enabled auto-merge August 25, 2026 18:16

yucheng-berri approved these changes Aug 25, 2026

View reviewed changes

mateo-berri merged commit 41aa1ca into litellm_internal_staging Aug 25, 2026
84 checks passed

mateo-berri deleted the litellm_together_api_host branch August 25, 2026 18:17

mateo-berri mentioned this pull request Aug 26, 2026

test(together_ai): regression suite across chat, responses, and messages surfaces #38283

Merged

8 tasks

Navigation Menu

Uh oh!

Uh oh!

fix(together_ai): default endpoints to api.together.ai instead of api.together.xyz - #38233

fix(together_ai): default endpoints to api.together.ai instead of api.together.xyz#38233
mateo-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_together_api_host

mateo-berri commented Aug 25, 2026 •

edited by cursor Bot

Loading

Uh oh!

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

greptile-apps Bot commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mateo-berri commented Aug 25, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (1d695a7, merge base)

chat defaults to the legacy host

rerank defaults to the legacy host

per-deployment api_base pins the legacy host (unchanged by this PR)

rerank ignores TOGETHER_AI_API_BASE

bare model + Together api_base fails provider resolution

After (5e6b6c6)

chat defaults to the current host

rerank defaults to the current host

rerank honors TOGETHER_AI_API_BASE

bare model + Together api_base resolves to together_ai

per-deployment api_base still pins the legacy host

Type

Caveats (if any)

Rerank credential model

Final Attestation

Uh oh!

greptile-apps Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

greptile-apps Bot commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

mateo-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 25, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateo-berri commented Aug 25, 2026 •

edited by cursor Bot

Loading

Before (`1d695a7`, merge base)

After (`5e6b6c6`)

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

codecov Bot commented Aug 25, 2026 •

edited

Loading