Skip to content

fix(deepseek): correct max_output_tokens for deepseek-v4 models - #31018

Open
New1Direction wants to merge 21 commits into
BerriAI:litellm_oss_stagingfrom
New1Direction:fix/deepseek-v4-max-output-tokens
Open

New1Direction wants to merge 21 commits into
BerriAI:litellm_oss_stagingfrom
New1Direction:fix/deepseek-v4-max-output-tokens

Conversation

@New1Direction

Copy link
Copy Markdown

The deepseek-v4-flash and deepseek-v4-pro entries had max_output_tokens and max_tokens set to 8192, which is carried over from the older deepseek-chat models. DeepSeek's pricing page lists 384K max output for both v4 models (the 1M context window was already correct).

This corrects all four affected entries — deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro, and the non-prefixed deepseek-v4-flash / deepseek-v4-pro aliases — in both the root model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json.

Source: https://api-docs.deepseek.com/quick_start/pricing

Srivatsa03 and others added 19 commits June 22, 2026 17:30
BerriAI#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution
… threshold (BerriAI#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length
…nt (BerriAI#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards
… so 5xx is retryable (BerriAI#24608) (BerriAI#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (BerriAI#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (BerriAI#24608)

* style(bedrock): black-format stream-error helper (BerriAI#24608)
…rriAI#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
…erriAI#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes BerriAI#30835

Co-authored-by: Cursor <cursoragent@cursor.com>
…1000 of it (BerriAI#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff
…nai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
…s_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
…SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.
…ting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.
…e spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.
run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.
@CLAassistant

CLAassistant commented Jun 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codspeed

codspeed Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Congrats! CodSpeed is installed 🎉

🆕 16 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

@New1Direction
New1Direction force-pushed the fix/deepseek-v4-max-output-tokens branch from 4adff6b to 491be59 Compare June 22, 2026 22:14
@codecov

codecov Bot commented Jun 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Corrects stale max_output_tokens and max_tokens values for deepseek-v4-flash and deepseek-v4-pro (both prefixed and un-prefixed aliases) in the root and backup model pricing JSON files — the previous value of 8192 was inherited from older deepseek-chat entries, while DeepSeek's official pricing page specifies 384K for both V4 models.

  • Updates four entries in model_prices_and_context_window.json and the matching four in litellm/model_prices_and_context_window_backup.json from 8192384000.
  • Adds targeted assertions to the two existing test_deepseek_v4_models_in_*_cost_map test functions, verifying both max_output_tokens and max_tokens equal 384,000 for all four model keys; no network calls are made.

Confidence Score: 5/5

Safe to merge — changes are purely data corrections to two JSON pricing files, verified against DeepSeek's official docs, with new test assertions to lock in the values.

All changes are confined to JSON metadata fields and additive test assertions. The corrected 384K value is confirmed by DeepSeek's pricing page (the same URL already cited in the JSON source field). Both JSON files are updated consistently, and the new tests exercise all four affected model keys in both cost maps without making any network calls.

No files require special attention.

Important Files Changed

Filename Overview
model_prices_and_context_window.json Updates max_output_tokens and max_tokens from 8192 → 384000 for all four deepseek-v4 model entries; values confirmed against official DeepSeek pricing docs
litellm/model_prices_and_context_window_backup.json Mirror of the root JSON change — same four entries corrected identically to keep the backup in sync
tests/test_litellm/test_utils.py Additive assertions appended to two existing test functions to verify the corrected 384K token limit for all four model keys in both the root and backup cost maps; no network calls made

Reviews (3): Last reviewed commit: "test(deepseek): assert max_output_tokens..." | Re-trigger Greptile

deepseek-v4-flash and deepseek-v4-pro (and their non-prefixed aliases) had
max_output_tokens/max_tokens of 8192, carried over from the older deepseek-chat
models. DeepSeek's pricing page lists 384K max output for both v4 models; the
1M context window was already correct. Updates the root cost map and the
bundled backup.

Source: https://api-docs.deepseek.com/quick_start/pricing
@New1Direction
New1Direction force-pushed the fix/deepseek-v4-max-output-tokens branch from 491be59 to 11d6b34 Compare June 22, 2026 22:50
@New1Direction
New1Direction changed the base branch from main to litellm_oss_staging June 22, 2026 22:50
@Sameerlite

Copy link
Copy Markdown
Contributor

Thanks for the contribution! A couple of things to help get this ready:

  • Could you add proof that the change works as expected? Screenshots, a sample request/response, test output, or a before/after comparison really help speed up review.

Also kicking off a fresh Greptile review on this.

@greptileai

The existing deepseek-v4 cost-map tests checked prices and max_input_tokens
but not max_output_tokens, which is the field this PR corrects. Extend both
tests to assert max_output_tokens (and max_tokens) == 384000 for all four
entries so the value stays guarded.
@New1Direction

Copy link
Copy Markdown
Author

Thanks! Here is a before/after using litellm.get_model_info (LITELLM_LOCAL_MODEL_COST_MAP=True):

# before (current data)
deepseek/deepseek-v4-flash → max_output_tokens = 8192
deepseek/deepseek-v4-pro   → max_output_tokens = 8192

# after (this PR)
deepseek/deepseek-v4-flash → max_output_tokens = 384000
deepseek/deepseek-v4-pro   → max_output_tokens = 384000

(max_input_tokens stays 1000000 in both cases.)

I also pushed test coverage. The existing test_deepseek_v4_models_in_cost_map / test_deepseek_v4_models_in_backup_cost_map checked the prices and max_input_tokens, but not max_output_tokens — the field this PR corrects. I extended both to assert max_output_tokens == 384000 for all four entries; the new assertions fail on the old 8192 data and pass on this change, so the value stays guarded going forward.

Source: DeepSeek lists 384K max output for both v4 models — https://api-docs.deepseek.com/quick_start/pricing

@Sameerlite

Copy link
Copy Markdown
Contributor

Thanks for your patience on this one! Your changes look solid — the Greptile review is just stale because of commits since the last review. Kicking off a fresh pass now.

@greptileai

@Sameerlite
Sameerlite force-pushed the litellm_oss_staging branch from dc6611b to e0c8a6b Compare June 24, 2026 10:33
@Sameerlite

Copy link
Copy Markdown
Contributor

Great work on this fix, @New1Direction — the data correction is well-documented and the new test assertions are a nice touch to lock in the values going forward! Just one thing blocking merge:\n\n- Merge conflicts — the branch has conflicts with the base branch. Could you rebase or merge the latest from litellm_internal_staging to resolve them?\n\nOnce that's done, this should be ready to go!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.