feat(providers): add Umans AI as an OpenAI-compatible provider by jcraftsman · Pull Request #44400 · BerriAI/litellm

jcraftsman · 2026-10-03T17:46:26Z

TLDR

Problem this solves:

Umans AI, an OpenAI and Anthropic compatible inference API, is not a LiteLLM provider
Users must hand-wire openai/ plus api_base for every Umans model

How it solves it:

Registers umans-ai as a JSON-configured OpenAI-compatible provider
Wires chat completions, Responses, and Anthropic Messages to https://api.code.umans.ai/v1
Adds Umans AI to the Admin UI Add Model form and the endpoint support matrix
Prices the seven public Umans AI models so spend tracking works

User Flow

Before: a proxy admin who adds a Umans AI model gets a provider error and no working deployment

They add model: umans-ai/umans-deepseek-v4-flash-0731 with their Umans AI key to the proxy config and start the proxy
The proxy logs LLM Provider NOT provided for that model at boot
POST http://localhost:4000/v1/chat/completions (or /v1/responses, /v1/messages) returns 400 "no healthy deployments"

After: the same config serves Umans AI on all three endpoints

They add model: umans-ai/umans-deepseek-v4-flash-0731 with UMANS_AI_API_KEY to the proxy config and start the proxy
The model loads with no provider error, and Umans AI shows up in the Add Model provider list
POST http://localhost:4000/v1/chat/completions, /v1/responses, and /v1/messages are forwarded to https://api.code.umans.ai/v1

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy (litellm.proxy.proxy_cli, DB-free, master_key=sk-1234) with one model umans-flash -> umans-ai/umans-deepseek-v4-flash-0731, UMANS_AI_API_KEY=sk-not-a-real-key. The proof uses a placeholder key, so success here means the request reaches api.code.umans.ai and comes back with Umans AI's own 401 authentication_error instead of litellm failing to resolve the provider

Proxy config:

model_list:
  - model_name: umans-flash
    litellm_params:
      model: umans-ai/umans-deepseek-v4-flash-0731
      api_key: os.environ/UMANS_AI_API_KEY
general_settings:
  master_key: sk-1234
  dangerously_permit_weak_or_unset_master_key: true

Run with DATABASE_URL/DIRECT_URL unset, UMANS_AI_API_KEY=sk-not-a-real-key exported:

.venv/bin/python -m litellm.proxy.proxy_cli --config umans_proxy_config.yaml --port 4000 --num_workers 1

The proxy appends a generic routing note to each error message below, trimmed here for readability

Before (`b024950`)

On main, umans-ai/umans-deepseek-v4-flash-0731 resolves no provider. The router logs LLM Provider NOT provided. Pass in the LLM provider you are trying to call at boot and the deployment is never created, so every route returns "no healthy deployments"

Chat completions

Request

curl -s localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"umans-flash","messages":[{"role":"user","content":"Say hello"}]}'

Response: HTTP 400

{"error":{"message":"litellm.BadRequestError: You passed in model=umans-flash. There are no healthy deployments for this model","type":"invalid_request_error","param":null,"code":"400"}}

Responses

Request

curl -s localhost:4000/v1/responses \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"umans-flash","input":"Say hello"}'

Response: HTTP 400

{"error":{"message":"litellm.BadRequestError: You passed in model=umans-flash. There are no healthy deployments for this model","type":"invalid_request_error","param":null,"code":"400"}}

Messages

Request

curl -s localhost:4000/v1/messages \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"umans-flash","max_tokens":32,"messages":[{"role":"user","content":"Say hello"}]}'

Response: HTTP 400

{"type":"error","error":{"type":"invalid_request_error","message":"litellm.BadRequestError: You passed in model=umans-flash. There are no healthy deployments for this model"}}

Add Model form lists Umans AI

Request

curl -s localhost:4000/public/providers/fields \
  -H 'Authorization: Bearer sk-1234' \
  | jq '.[] | select(.litellm_provider == "umans-ai")'

Response: empty, Umans AI is not in the provider list

After (`a8c58cc`)

Each endpoint reaches https://api.code.umans.ai/v1/<path> and returns Umans AI's own 401 authentication_error, which is the response a customer would see with a bad key. (Each capture used a freshly restarted proxy, since the first 401 puts the deployment in router cooldown.)

Chat completions

Request

curl -s localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"umans-flash","messages":[{"role":"user","content":"Say hello"}]}'

Response: HTTP 401

{"error":{"message":"litellm.AuthenticationError: AuthenticationError: Umans-aiException - Invalid API key","type":"authentication_error","param":null,"code":"401"}}

Responses

Request

curl -s localhost:4000/v1/responses \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"umans-flash","input":"Say hello"}'

Response: HTTP 401, with Umans AI's upstream error body passed through

{"error":{"message":"litellm.AuthenticationError: AuthenticationError: Umans-aiException - {\"error\":{\"type\":\"authentication_error\",\"message\":\"Invalid API key\"}}","type":"authentication_error","param":null,"code":"401"}}

Messages

Request

curl -s localhost:4000/v1/messages \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"umans-flash","max_tokens":32,"messages":[{"role":"user","content":"Say hello"}]}'

Response: HTTP 401

{"type":"error","error":{"type":"authentication_error","message":"litellm.AuthenticationError: AuthenticationError: Umans-aiException - {\"error\":{\"type\":\"authentication_error\",\"message\":\"Invalid API key\"}}"}}

Add Model form lists Umans AI

Request

curl -s localhost:4000/public/providers/fields \
  -H 'Authorization: Bearer sk-1234' \
  | jq '.[] | select(.litellm_provider == "umans-ai")'

Response

{
  "provider": "UMANS_AI",
  "provider_display_name": "Umans AI",
  "litellm_provider": "umans-ai",
  "credential_fields": [
    {"key": "api_base", "label": "API Base", "placeholder": "https://api.code.umans.ai/v1", "required": false, "field_type": "text"},
    {"key": "api_key", "label": "API Key", "placeholder": null, "required": true, "field_type": "password"}
  ],
  "default_model_placeholder": "umans-ai/umans-deepseek-v4-flash-0731"
}

Unit tests: tests/unit/llms/openai_like/test_umans_ai_provider.py covers env and explicit credential resolution, the Add Model form entry, the endpoint matrix, respx-captured chat, Responses, and Messages requests in both non-streaming and streaming mode, and cost map invariants: main matches backup, every entry is a priced chat model, the Add Model default model is priced, and billing with cache reads uses the map rates (13 tests). Also passing locally: all of tests/unit/llms/openai_like/ (232 tests), test_aaamodel_prices_and_context_window_json_is_valid, plus tests/unit/litellm_core_utils/test_get_llm_provider_logic.py and test_exception_mapping_utils.py (1,276 tests). Mutation checks: changing base_url to /v2 turns 7 tests red, and changing one price in the backup map turns both cost map tests red

Type

🆕 New Feature

Caveats (if any)

Medium

Proof uses a placeholder key, so it stops at Umans AI's 401

Low

Embeddings, image, audio, and OCR not wired
- Umans AI serves chat models only
Docs page docs/providers/umans_ai not included here

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

CLAassistant · 2026-10-03T17:46:33Z

All committers have signed the CLA.

greptile-apps · 2026-10-03T17:50:24Z

Confidence Score: 4/5

[Medium risk] Adds a new OpenAI-compatible LLM provider integration.

The provider integration appears functional, but the explicit test-guide requirements should be satisfied before merging

Findings

Vendor prices pinned in tests ▶
Disallowed explanatory comment ▶
Unsupported audio reaches Umans ▶

Summary

The PR registers Umans AI as an OpenAI-compatible provider for chat completions, Responses, and Messages, and adds provider metadata, seven model-price entries, and request-path tests

The earlier pricing and streaming-coverage findings have been addressed or resolved
The new pricing tests need to follow the repository’s guidance on outside facts and source comments

_{Reviews (2) · Last reviewed commit: "feat(providers): price Umans AI GLM 5.3"}

greptile-apps · 2026-10-03T17:50:27Z

    "scx-ai",
    "prism",
    "sail",
+    "umans-ai",


Unsupported audio reaches Umans

Adding umans-ai to openai_compatible_providers also adds it to the audio-transcription provider set. An audio-transcription call for an Umans model is then sent to Umans /v1/audio/transcriptions, even though this PR marks transcription unsupported. Keeping transcription dispatch separate would avoid sending audio to an unsupported endpoint.

Knowledge Base Used: Provider adapters and capabilities

Every JSON provider in openai_compatible_providers shares this, so a Umans-only exclusion would be inconsistent. Umans answers transcription requests with a 404

That’s fair context. The important distinction is that provider_endpoints_support.json is metadata and does not gate SDK dispatch: because OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS is derived directly from openai_compatible_providers, an Umans transcription call is still classified as supported and makes a network request to /v1/audio/transcriptions; the upstream 404 is not a LiteLLM-side fast-fail.

However, this is shared behavior for all JSON-configured OpenAI-compatible providers, and adding a one-off exclusion here would be inconsistent and require a broader refactor of the derived provider sets. I’m withdrawing this as a blocking issue; the current implementation is reasonable given the existing architecture.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

That’s fair. Since openai_compatible_providers is also used for transcription dispatch for every JSON-configured OpenAI-compatible provider, excluding only umans-ai would be inconsistent; the upstream 404 is the established behavior for unsupported operations. I’m withdrawing this finding—no Umans-specific change is needed here.

codspeed · 2026-10-03T17:54:51Z

Merging this PR will not alter performance

⚠️

1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

test_completion_streaming

✅ 31 untouched benchmarks

_{Comparing jcraftsman:litellm_umans_ai_provider (b7a1177) with main (ca0ff24)¹}

No successful run was found on main (5f0e51f) during the generation of this report, so ca0ff24 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-10-03T17:58:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

jcraftsman · 2026-10-03T20:14:15Z

@greptileai

greptile-apps · 2026-10-03T20:16:58Z

+        return json.load(f)
+
+
+@pytest.mark.parametrize(("model", "input_cost", "output_cost", "cache_read_cost"), UMANS_PRICING)


Vendor prices pinned in tests These assertions pin seven vendor prices. The test guide requires an invariant or a dated source beside outside facts; satisfy that requirement before merging.

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Replaced the pinned prices with invariants: main and backup agree, every entry is priced, the default model is priced, and billing uses map rates

greptile-apps · 2026-10-03T20:16:58Z

+
+
+UMANS_PRICING: Final = (
+    # model, then USD per token for input, output and cache read (app.umans.ai/pricing)


Disallowed explanatory comment This comment only labels tuple fields and the price source. Repository guidance prohibits such source comments; remove it before merging.

Suggested change

# model, then USD per token for input, output and cache read (app.umans.ai/pricing)

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Removed. The pricing tuple and its comment are gone, and the new invariant tests carry no explanatory comments

Adds cost map entries (main and bundled backup) for the six public Umans AI models at the rates on app.umans.ai/pricing, so proxy spend and budgets count Umans usage. Adds streaming tests for chat completions, Responses and Anthropic Messages.

Adds the umans-ai/umans-glm-5.3 cost map entry (main and bundled backup) at the rates on app.umans.ai/pricing and covers it in the pricing test.

…d prices Replaces literal vendor prices with checks LiteLLM owns: main and backup agree, every entry is a priced chat model, the Add Model default model is priced, and billing (cache reads included) uses the cost map rates. Marks test locals Final and types the SSE helpers as read-only mappings.

jcraftsman requested a review from a team October 3, 2026 17:46

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

jcraftsman requested review from litellm-stampbot, mateo-berri and ryan-crabbe-berri as code owners October 3, 2026 18:29

jcraftsman force-pushed the litellm_umans_ai_provider branch from 4a95581 to 4b04d3c Compare October 3, 2026 19:47

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

jcraftsman force-pushed the litellm_umans_ai_provider branch 5 times, most recently from e5bd038 to 0e2ed9a Compare October 8, 2026 08:47

jcraftsman requested a review from kerry-berri as a code owner October 8, 2026 08:47

jcraftsman force-pushed the litellm_umans_ai_provider branch from 0e2ed9a to d3ab5d0 Compare October 8, 2026 15:19

devin-ai-integration Bot mentioned this pull request Oct 8, 2026

fix(model_prices): add Bedrock flex prices, Anthropic web search flags, Gemini shutdown dates, OpenRouter alias drift, Mistral Large 4 context #44910

Merged

jcraftsman force-pushed the litellm_umans_ai_provider branch 2 times, most recently from 4b0151b to c543fad Compare October 10, 2026 15:16

jcraftsman added 4 commits October 11, 2026 09:12

feat(providers): add Umans AI as an OpenAI-compatible provider

e8f0c5c

feat(providers): price Umans AI GLM 5.3

41c1d78

Adds the umans-ai/umans-glm-5.3 cost map entry (main and bundled backup) at the rates on app.umans.ai/pricing and covers it in the pricing test.

jcraftsman force-pushed the litellm_umans_ai_provider branch from c543fad to b7a1177 Compare October 11, 2026 07:13

		return json.load(f)


		@pytest.mark.parametrize(("model", "input_cost", "output_cost", "cache_read_cost"), UMANS_PRICING)



		UMANS_PRICING: Final = (
		# model, then USD per token for input, output and cache read (app.umans.ai/pricing)

Conversation

jcraftsman commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (b024950)

Chat completions

Responses

Messages

Add Model form lists Umans AI

After (a8c58cc)

Chat completions

Responses

Messages

Add Model form lists Umans AI

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

CLAassistant commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

jcraftsman Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Oct 3, 2026

Codecov Report

Uh oh!

jcraftsman commented Oct 3, 2026

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

jcraftsman Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

jcraftsman Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jcraftsman commented Oct 3, 2026 •

edited

Loading

Before (`b024950`)

After (`a8c58cc`)

CLAassistant commented Oct 3, 2026 •

edited

Loading

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

jcraftsman Oct 3, 2026 •

edited

Loading

codspeed Bot commented Oct 3, 2026 •

edited

Loading