fix(logging): redact tool call arguments to valid JSON and preserve null content by yucheng-berri · Pull Request #38182 · BerriAI/litellm

yucheng-berri · 2026-08-25T07:02:49Z

TLDR

Problem this solves:

turn_off_message_logging stores tool call arguments as invalid JSON
it also invents redacted-by-litellm content for tool-only turns
replaying such a session via previous_response_id fails with 500

How it solves it:

keep storing the explicit redacted-by-litellm marker in arguments, so a redacted call stays distinguishable in the logs
swap that marker for {} in memory while rebuilding the provider request during session replay, so converters can parse it
keep null assistant content null instead of inventing text, on chat choices and on Responses API output items alike
guard Langfuse observation output against unparseable arguments

User Flow

Before: a developer whose proxy has turn_off_message_logging: true cannot continue any Responses API session that contains a tool call

They send POST https://litellm-domain/v1/responses with tools and get back a function_call with real arguments
They follow up with POST https://litellm-domain/v1/responses carrying previous_response_id and a new question
The follow-up fails with HTTP 500, Unable to convert openai tool calls ... Expecting value: line 1 column 1, because the stored turn now holds "arguments": "redacted-by-litellm"
On https://litellm-domain/ui/?page=logs the assistant turn of that request shows invented redacted-by-litellm content where the model actually sent none, and the stored tool call arguments are unparseable for anything reading the log

After: the same follow-up works and the stored log stays redacted

They send the same POST https://litellm-domain/v1/responses with tools and get back the same function_call with real arguments
The follow-up with previous_response_id returns HTTP 200 with the model's answer: the marker is swapped for {} only in memory while building the provider request
Sessions stored before the upgrade replay the same way, since old and new rows hold the same marker
On https://litellm-domain/ui/?page=logs the assistant turn shows the tool call with no invented content, matching what the model really sent, and the stored arguments keep the explicit redaction marker

Relevant issues

Linear ticket

Resolves LIT-6102

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup: proxy with turn_off_message_logging: true, store_prompts_in_spend_logs: true, a langfuse_otel callback, Postgres attached, real Gemini and Anthropic upstreams. TOOLS is one get_weather function tool in the endpoint's native shape

Before (`a9c7b84`)

case 1: /v1/chat/completions tool call, live response untouched

curl -s $PROXY/v1/chat/completions -d '{"model":"gemini-flash","messages":[{"role":"user","content":"Weather in Berlin? You must use the get_weather tool."}],"tools":TOOLS}'
Client sees real arguments: "content": null, "tool_calls": [{"function": {"arguments": "{\"city\": \"Berlin\"}", "name": "get_weather"}}]

case 2: /v1/chat/completions stream=true tool call, live stream untouched

Same call with "stream": true
Streamed chunks carry "function":{"arguments":"{\"city\": \"Tokyo\"}","name":"get_weather"}

case 3: /v1/responses tool call, live response untouched

curl -s $PROXY/v1/responses -d '{"model":"gemini-flash","input":"Weather in Berlin? You must use the get_weather tool.","tools":TOOLS}'
Client sees "function_call": {"name": "get_weather", "arguments": "{\"city\": \"Berlin\"}"}

case 4: what the spend log stored

SELECT response FROM "LiteLLM_SpendLogs" ORDER BY "startTime" DESC LIMIT 3; then count the redacted fields
Output: 3 "arguments": "redacted-by-litellm" and 3 "content": "redacted-by-litellm"; the arguments are invalid JSON and the content is invented (the model sent none)
The logs page renders the fabricated assistant content and the raw sentinel:

case 5: replay the session to the same model

curl -s $PROXY/v1/responses -d '{"model":"gemini-flash","previous_response_id":"<id from case 3>","input":"and tomorrow?"}'
HTTP 500: litellm.APIConnectionError: Unable to convert openai tool calls={'content': 'redacted-by-litellm', ... 'arguments': 'redacted-by-litellm' ...}

case 6: replay the session cross-provider

Same call with "model": "claude-haiku"
HTTP 400: AnthropicException - Failed to parse tool call arguments for tool 'get_weather' ... Expecting value: line 1 column 1 (char 0). Arguments: redacted-by-litellm

case 7: replay a session stored by the old release

On this side every stored session is an old-format row, so this is case 5: any replay of stored history fails the same way

After (92ee7e3b57)

case 1: /v1/chat/completions tool call, live response untouched

Same call as before
Client sees the same real arguments: "content": null, "tool_calls": [{"function": {"arguments": "{\"city\": \"Berlin\"}", "name": "get_weather"}}]

case 2: /v1/chat/completions stream=true tool call, live stream untouched

Same call as before
Streamed chunks carry "function":{"arguments":"{\"city\": \"Tokyo\"}","name":"get_weather"}

case 3: /v1/responses tool call, live response untouched

Same call as before
Client sees "function_call": {"name": "get_weather", "arguments": "{\"city\": \"Berlin\"}"}

case 4: what the spend log stored

Same query
Output: 3 "arguments": "redacted-by-litellm" and 3 "content": null; the stored row keeps the explicit redaction marker and no content is invented
The logs page shows the tool call with no invented content and the marker visible in the arguments:

case 5: replay the session to the same model

Same call as before
HTTP 200, the model answers the follow-up: the stored marker is swapped for {} in memory while building the provider request, so the converter parses it

case 6: replay the session cross-provider

Same call as before
The request now clears argument parsing and reaches Anthropic; this rig's Anthropic key is routed through a sandbox gateway so the upstream answers with an auth error, proving the converter no longer crashes

case 7: replay a session stored by the old release

Same call with the previous_response_id of a session written by the unfixed build
HTTP 200: old rows and new rows hold the same marker, so the same in-memory normalization covers both and the session continues

Langfuse destination A/B (real cloud.langfuse.com, base `a9c7b84` vs head `2e986da`)

Same two proxies with a langfuse_otel callback exporting to https://cloud.langfuse.com; every value below was read back through the Langfuse public API, no local collector

case 8: /v1/chat/completions tool call reaches Langfuse on both sides

Same gemini-flash get_weather call on each proxy
Both traces show the redacted input and a parsed observation output, since this branch already guarded the marker before this PR:

"input": [{"role": "user", "content": "redacted-by-litellm"}]
"output": [{"name": "get_weather", "type": "function_call", "arguments": {}}]

case 9: native /v1/responses tool call, the branch this PR guards

curl -s $PROXY/v1/responses -d '{"model":"gpt-4o-mini","input":"Weather in Berlin? You must use the get_weather tool.","tools":TOOLS,"tool_choice":"required"}' on each proxy
Before (trace 89bd360f...): the observation has "output": null and the proxy log shows the crash that ate it:

File ".../langfuse_otel.py", line 200, in _set_observation_output
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

After (trace 2b0f5d14...): the observation output arrives intact, and its id matches the function_call id the client received, proving it is the same call:

"output": [{"id": "fc_01e481945093009c006a8e259c3ecc87d08b19d495c968d940",
            "name": "get_weather", "call_id": "call_iDyZKYqmqvcKrej2v7rN4MGc",
            "type": "function_call", "arguments": {}}]

The spend rows stayed identical on both sides ("arguments": "redacted-by-litellm", no invented text), confirming the stored format is untouched and this run also exercised the Responses-shape redaction writer live

Type

🐛 Bug Fix

Caveats (if any)

stored arguments keep the redacted-by-litellm marker, which is not valid JSON; readers that json.loads stored arguments must keep handling that, as they did before this PR
custom tool calls carry no function field, so the replay normalizer skips them via a getattr guard
Gemini may still reject a replayed redacted turn that carries a thought signature (provider 400 instead of the old proxy 500), since the real arguments are unrecoverable by design

Behavior changes

stored spend-log values are unchanged: redacted tool call arguments keep the redacted-by-litellm marker, so log readers can still tell a redacted call from a genuinely empty one
redacted assistant turns with no content now keep content: null instead of the sentinel string, and Responses API output items with text: null keep it too; Langfuse and OTEL payloads omit those fields for such turns
session replay swaps the redacted-by-litellm arguments for {} in memory while building the provider request; the stored row is untouched
earlier revisions of this PR stored {} instead of the marker; that never shipped, and external log readers keep seeing the marker exactly as before

…ull content Resolves LIT-6102

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

greptile-apps · 2026-08-25T07:06:44Z

Greptile Summary

This PR preserves null assistant content during redaction and makes redacted tool calls replayable without changing the stored sentinel

Centralizes the in-memory replay placeholder in constants.py
Normalizes redacted function arguments before provider conversion
Safely handles unparseable function arguments in Langfuse output
Adds regression coverage for redaction, replay, and telemetry behavior

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/constants.py	Defines the shared in-memory placeholder alongside the existing redaction sentinel, completing the requested centralization
litellm/litellm_core_utils/redact_messages.py	Preserves null content while consistently writing the centralized redaction sentinel to tool-call arguments
litellm/responses/litellm_completion_transformation/session_handler.py	Replaces stored redaction sentinels with valid empty JSON only in replayed in-memory messages
litellm/responses/litellm_completion_transformation/transformation.py	Normalizes redacted Responses API function-call arguments before conversion to chat-completion messages
litellm/integrations/langfuse/langfuse_otel.py	Uses safe JSON parsing so redacted arguments do not prevent Langfuse observation output
tests/test_litellm/responses/litellm_completion_transformation/test_session_handler.py	Adds replay regression coverage using the previously accepted test seam
tests/test_litellm/litellm_core_utils/test_redact_messages.py	Covers multi-tool redaction and preservation of null content across object and dictionary response forms

_{Reviews (6): Last reviewed commit: "fix(logging): keep the redaction sentine..." | Re-trigger Greptile}

codecov · 2026-08-25T07:08:57Z

Codecov Report

❌ Patch coverage is 92.50000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/redact_messages.py	91.66%	2 Missing ⚠️
...tellm_completion_transformation/session_handler.py	90.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…nd satisfy test-quality gate

yucheng-berri · 2026-08-25T07:16:48Z

@greptileai

…d by basedpyright

yucheng-berri · 2026-08-25T07:32:46Z

@greptileai

yucheng-berri · 2026-08-25T07:47:20Z

@greptileai

codspeed-hq · 2026-08-25T08:33:00Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_redaction_sentinel_fix (2e986da) with litellm_internal_staging (bb27bfd)¹}

No successful run was found on litellm_internal_staging (a9f42f0) during the generation of this report, so bb27bfd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

yucheng-berri · 2026-08-25T17:01:36Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit bba6e8d. Configure here.}

…ts and preserve null output text

yucheng-berri · 2026-08-25T19:26:37Z

@greptileai

yucheng-berri · 2026-08-25T19:53:44Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2e986da. Configure here.}

yucheng-berri · 2026-08-25T20:28:49Z

@greptileai

yucheng-berri · 2026-08-25T20:28:49Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2e986da. Configure here.}

…ions test: fix staging CI regressions from #38182, #38144, #38265, #37962, and #37969

…erriAI#38265, BerriAI#37962, and BerriAI#37969 - test_custom_callback_input: audio redaction assertion expects None content (redaction leaves None untouched, gpt-audio-1.5 returns content=None) - local_testing conftest: drain GLOBAL_LOGGING_WORKER in isolate_litellm_state teardown so mocked-router tests stop leaking pending logging tasks into test_gcs_pub_sub - test_together_ai: tools is always a supported param now; only response_format is gated by function-calling support - test_keys: /team/new omits models instead of sending null (422), so the key's team really exists and auth no longer raises TeamNotFoundError - test_team_delete_member_add_race: per-test unique team and user ids so xdist workers sharing one Postgres stop deleting each other's team mid-race

fix(logging): redact tool call arguments to valid JSON and preserve n…

b5d1cd6

…ull content Resolves LIT-6102

devin-ai-integration Bot reviewed Aug 25, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

refactor(logging): centralize redacted tool-call arguments constant a…

e5f0c36

…nd satisfy test-quality gate

This comment was marked as resolved.

Sign in to view

fix(responses): drop Final annotations on loop-assigned locals flagge…

aace164

…d by basedpyright

fix(responses): skip custom tool calls in redacted-arguments normalizer

bba6e8d

cursor Bot reviewed Aug 25, 2026

View reviewed changes

fix(logging): keep the redaction sentinel in stored tool-call argumen…

2e986da

…ts and preserve null output text

cursor Bot reviewed Aug 25, 2026

View reviewed changes

yassin-berriai approved these changes Aug 25, 2026

View reviewed changes

yucheng-berri merged commit ba8d8b6 into litellm_internal_staging Aug 25, 2026
84 of 85 checks passed

yucheng-berri deleted the litellm_redaction_sentinel_fix branch August 25, 2026 23:38

mateo-berri mentioned this pull request Aug 26, 2026

test: fix staging CI regressions from #38182, #38144, #38265, #37962, and #37969 #38304

Merged

7 tasks

yuneng-berri added a commit that referenced this pull request Aug 28, 2026

Merge pull request #38304 from BerriAI/litellm_fix_staging_ci_regress…

2b10dc5

…ions test: fix staging CI regressions from #38182, #38144, #38265, #37962, and #37969

Uh oh!

Uh oh!

Conversation

yucheng-berri commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (a9c7b84)

case 1: /v1/chat/completions tool call, live response untouched

case 2: /v1/chat/completions stream=true tool call, live stream untouched

case 3: /v1/responses tool call, live response untouched

case 4: what the spend log stored

case 5: replay the session to the same model

case 6: replay the session cross-provider

case 7: replay a session stored by the old release

After (92ee7e3b57)

case 1: /v1/chat/completions tool call, live response untouched

case 2: /v1/chat/completions stream=true tool call, live stream untouched

case 3: /v1/responses tool call, live response untouched

case 4: what the spend log stored

case 5: replay the session to the same model

case 6: replay the session cross-provider

case 7: replay a session stored by the old release

Langfuse destination A/B (real cloud.langfuse.com, base a9c7b84 vs head 2e986da)

case 8: /v1/chat/completions tool call reaches Langfuse on both sides

case 9: native /v1/responses tool call, the branch this PR guards

Type

Caveats (if any)

Behavior changes

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

greptile-apps Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

This comment was marked as resolved.

Uh oh!

codecov Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

codspeed-hq Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment

yucheng-berri commented Aug 25, 2026 •

edited

Loading

Before (`a9c7b84`)

Langfuse destination A/B (real cloud.langfuse.com, base `a9c7b84` vs head `2e986da`)

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

codecov Bot commented Aug 25, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 25, 2026 •

edited

Loading