Skip to content

feat(proxy): redact or drop individual batch records instead of rejecting the file - #37561

Merged
yucheng-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_lit5276_batch_record_redaction
Aug 20, 2026
Merged

feat(proxy): redact or drop individual batch records instead of rejecting the file#37561
yucheng-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_lit5276_batch_record_redaction

Conversation

@yucheng-berri

@yucheng-berri yucheng-berri commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • One bad record rejected an entire batch upload
  • Unusable for a file holding thousands of rows
  • No way to see which records a guardrail touched

How it solves it:

  • Rewritten records are submitted in their rewritten form
  • Blocked records are left out, the rest go
  • The response reports every changed record

User Flow

Before: a platform team with a PII guardrail cannot use batch at all, because one row with a customer email rejects the whole job

  1. They upload a 4 row file with POST https://litellm-domain/v1/files, purpose=batch, where row 2 contains an email address and row 3 asks for something the guardrail blocks
  2. The upload returns 400 naming row 3
  3. They have no way to submit the other three rows without hand-editing the file first
  4. Scaled to a real job of thousands of rows, they would have to find and fix every offending row before any of it runs

After: the same upload is accepted, the offending rows are handled per record, and the response says what happened to each

  1. They upload the same file with POST https://litellm-domain/v1/files, purpose=batch
  2. The upload returns 200 with a normal file id, plus litellm_batch_guardrail reporting submitted_records: 3 and naming row 2 as redacted and row 3 as dropped
  3. They fetch https://litellm-domain/v1/files/{id}/content and see three rows, with row 2's email replaced by the guardrail's mask and rows 1 and 4 exactly as written
  4. They create the batch against that file id as normal
  5. A proxy admin sees the same per-record outcome in the proxy logs, so the change is not visible only to the caller

Relevant issues

Linear ticket

Resolves LIT-5276

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Stacked on #37519, which makes guardrails run on batch uploads at all. Base for this QA is that PR's head, 8b2f6d6dd0

Shared setup. config.yaml configures two guardrails, one that masks and one that blocks, using the example guardrail shipped in litellm/proxy/example_config_yaml/custom_guardrail.py:

guardrails:
  - guardrail_name: "mask-guard"
    litellm_params:
      guardrail: custom_guardrail.myCustomGuardrail   # masks the literal "litellm"
      mode: "pre_call"
      default_on: true
  - guardrail_name: "block-guard"
    litellm_params:
      guardrail: custom_guardrail.myBlockingGuardrail # raises 400 on "blockme"
      mode: "pre_call"
      default_on: true

files_settings:
  - custom_llm_provider: openai
    api_key: os.environ/OPENAI_API_KEY

Both files hold four records: a clean row, a row containing litellm, a row containing blockme, and another clean row. pr2_tagged.jsonl is the same file with every record carrying litellm_metadata.tags, which is how a caller attributes rows to a cost center. Uploads go to the real OpenAI files API, and the content is read back from OpenAI rather than from the proxy's own copy. The proxy ran on port 4276 because 4000 was already in use

Before (8b2f6d6)

1. mixed file

  1. curl -sS http://127.0.0.1:4276/v1/files -H "Authorization: Bearer sk-..." -F purpose=batch -F file=@pr2_mixed.jsonl
  2. HTTP 400 and {"error":{"message":"{'error': 'Violated guardrail policy', 'guardrail': 'block-guard', 'guardrail_name': 'block-guard', 'guardrail_mode': 'pre_call'}","code":"400"}}
  3. No file id is returned, so none of the four records can be submitted

2. mixed file whose records carry tags

  1. curl -sS http://127.0.0.1:4276/v1/files -H "Authorization: Bearer sk-..." -F purpose=batch -F file=@pr2_tagged.jsonl
  2. HTTP 400 with the same body, so the tagged rows never reach a provider either

3. guardrail whose backend is unreachable under a fail-closed policy

  1. risk2_input.jsonl holds three records; the middle one trips a guardrail that raises GuardrailRaisedException("Singulr API unreachable (block_on_error=True): timed out"), the shape several integrations use when their backend is down
  2. curl -sS http://127.0.0.1:4280/v1/files -H "Authorization: Bearer sk-..." -F purpose=batch -F file=@risk2_input.jsonl
  3. HTTP 400, since at this base any guardrail raising rejects the whole file

After (eef8331)

Full run, recorded live against real OpenAI:

batch guardrail redaction, recorded live against real OpenAI

1. mixed file

  1. curl -sS http://127.0.0.1:4276/v1/files -H "Authorization: Bearer sk-..." -F purpose=batch -F file=@pr2_mixed.jsonl
  2. HTTP 200 and
{
  "id": "file-KKVMnf3YSUCotz3Ks4Eutj", "object": "file", "purpose": "batch", "bytes": 594, "status": "processed",
  "litellm_batch_guardrail": {
    "submitted_records": 3,
    "modified_records": [
      {"line": 2, "custom_id": "row-2", "action": "redacted", "guardrail": null},
      {"line": 3, "custom_id": "row-3", "action": "dropped", "guardrail": "block-guard"}
    ]
  }
}
  1. curl -sS -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/files/file-KKVMnf3YSUCotz3Ks4Eutj/content
  2. Three records reached OpenAI, with row 2 masked and rows 1 and 4 unchanged:
row-1 -> a perfectly clean row
row-2 -> this row mentions ******** and should be masked
row-4 -> another clean row
  1. The proxy log carries the same outcome, so it is not visible only to the caller:
batch guardrails changed 2 of 4 records in pr2_mixed.jsonl: line 2 (custom_id row-2) redacted, line 3 (custom_id row-3) dropped

2. mixed file whose records carry tags

  1. curl -sS http://127.0.0.1:4276/v1/files -H "Authorization: Bearer sk-..." -F purpose=batch -F file=@pr2_tagged.jsonl
  2. HTTP 200 with the same litellm_batch_guardrail report, submitted_records: 3, row 2 redacted and row 3 dropped
  3. curl -sS -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/files/file-Fp9HxXfXzorfM2BqxhzKGU/content
  4. Every surviving record kept its tags, including the one the guardrail rewrote, so a masked row is still attributed the same way its untouched neighbours are:
row-1 -> a perfectly clean row                          litellm_metadata={'tags': ['cost-center-42']}
row-2 -> this row mentions ******** and should be masked litellm_metadata={'tags': ['cost-center-42']}
row-4 -> another clean row                              litellm_metadata={'tags': ['cost-center-42']}

3. guardrail whose backend is unreachable under a fail-closed policy

  1. Same file and same guardrail as case 3 above
  2. HTTP 400 and {"error":{"message":"Guardrail raised an exception, Guardrail: down-guard, Message: Singulr API unreachable (block_on_error=True): timed out"}}
  3. The upload is refused rather than quietly shrunk. Run against 686ea1e8a1, an earlier head of this PR, the same file returned HTTP 200 reporting {"line": 2, "custom_id": "scanner-down", "action": "dropped", "guardrail": "down-guard"} and only clean-1 and clean-2 reached OpenAI, so a scanner outage read as a policy block and the caller could not tell the difference

4. real in-tree guardrail whose backend returns 503 for one record

  1. Config runs the shipped xecguard guardrail against a stand-in backend that answers 503 only for the record containing tripwire; block_on_error is left at its default, which is on
  2. curl -sS http://127.0.0.1:4503/v1/files -H "Authorization: Bearer sk-..." -F purpose=batch -F file=@risk2_input.jsonl
  3. HTTP 400 and {"error":{"message":"{'error': \"XecGuard API unreachable (block_on_error=True): Server error '503 Service Unavailable' ...\"}"}}
  4. Against 9180fb97de, the previous head of this PR, the same upload returned HTTP 200 with {"line": 2, "custom_id": "scanner-down", "action": "dropped", "guardrail": "xecguard-pre"} and only clean-1 and clean-2 reached OpenAI. That guardrail reports an unreachable backend as an HTTPException carrying a block status, which the previous head read as a content verdict

Type

🆕 New Feature

Caveats (if any)

  • Only guardrails that run pre_call see batch records
  • Model-level guardrails still run post-routing, so batch misses them
  • The report is on create, not on file retrieve
  • A guardrail that cannot be reached aborts the upload
  • Drop or abort follows litellm's own intervention classification
  • A dropped record names its guardrail, not a reason
  • One report entry per changed record, so the response grows with them
  • A raise that does not set blocked_content aborts rather than drops
  • A guardrail raising from an error is read as a failure to judge
  • A record's own guardrails key is ignored for the scan
  • A record a guardrail reroutes refuses the whole upload

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

High Risk
Changes how guardrail enforcement applies to batch uploads: dropping or rewriting records instead of rejecting the file. A misclassified block vs. infrastructure failure would silently skip scanning some rows.

Overview
Batch purpose=batch uploads no longer fail the whole file when a pre-call guardrail rewrites or blocks one row. Masked records are submitted as rewritten; blocked records are omitted; the rest go through. The create-file response (and proxy logs) include litellm_batch_guardrail naming each redacted or dropped line.

GuardrailRaisedException now carries blocked_content so a real policy verdict can drop a row while a fail-closed outage, timeout, or unparseable backend still aborts the upload. Content blocks from HTTP 400/403/422, PII errors, and in-tree BLOCKED paths set the flag; sensitive-data reroutes still reject the file because a batch cannot follow a per-record model change.

Rewrites are spooled off-heap, untouched lines are copied byte-for-byte, and a record cannot opt out of the team/key guardrail chain via its own guardrails key. If every record is dropped, the upload is rejected as empty.

Reviewed by Cursor Bugbot for commit 175991c. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR now handles batch guardrail outcomes per record while distinguishing explicit content verdicts from guardrail failures

  • Rewritten records are uploaded in their redacted form, while explicitly blocked records are omitted
  • Guardrail reports expose submitted, redacted, and dropped records to callers and proxy logging
  • Fail-closed backend errors abort the upload rather than silently dropping uninspected records

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename Overview
litellm/proxy/openai_files_endpoints/batch_guardrails.py Implements record scanning, verdict classification, spooled rewrites, record dropping, and per-record reporting
litellm/proxy/openai_files_endpoints/files_endpoints.py Integrates batch scanning and rewritten upload streams into file creation with deterministic spool cleanup
litellm/exceptions.py Adds an explicit blocked_content signal so batch handling can distinguish verdicts from technical failures
litellm/proxy/guardrails/guardrail_hooks/straiker/straiker.py Separates Straiker content verdicts from fail-closed backend failures using the new exception signal
tests/test_litellm/proxy/openai_files_endpoint/test_batch_guardrails.py Covers rewritten uploads, dropped records, fail-closed errors, reports, and exception-classification behavior

Reviews (11): Last reviewed commit: "fix(proxy): register the scan spool befo..." | Re-trigger Greptile

Comment thread litellm/proxy/openai_files_endpoints/batch_guardrails.py
Comment on lines +3443 to +3445

class _Redactor(CustomGuardrail):
async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test patches proxy globals

The new endpoint test directly patches proxy globals, callbacks, and routing rather than injecting mocked dependencies, coupling the test to implementation details and forcing unrelated test rewrites when proxy initialization or module ownership changes.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

veria-ai[bot]

This comment was marked as resolved.

@veria-ai

veria-ai Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

PR overview

This pull request changes proxy batch-file handling so individual records can be redacted or dropped rather than causing the entire file to be rejected. The affected endpoint also reports guardrail scan results for batch processing.

One issue has been addressed, but a low-impact log-integrity issue remains. An authenticated caller can place control characters in a filename or batch record identifier to forge lines in server logs when guardrail results are reported; the impact is limited to misleading or disrupting log output.

Open issues (1)

Fixed/addressed: 1 · PR risk: 5/10

devin-ai-integration[bot]

This comment was marked as resolved.

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.97980% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...m/proxy/openai_files_endpoints/batch_guardrails.py 98.61% 2 Missing ⚠️
...lm/proxy/openai_files_endpoints/files_endpoints.py 92.59% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit b61af8d. Configure here.

@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_guardrail_redaction branch from 45b549c to abfb3d7 Compare August 20, 2026 02:22
@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_record_redaction branch from b61af8d to 6173989 Compare August 20, 2026 02:28
devin-ai-integration[bot]

This comment was marked as resolved.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

Drop or abort now follows litellm's own is_guardrail_intervention, so a dropped record and the logged status cannot disagree. @greptileai please review head 6173989

greptile-apps[bot]

This comment was marked as resolved.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 6173989. Configure here.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

Dropped records now name the guardrail that raised. @greptileai please review head 686ea1e

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 686ea1e. Configure here.

len(scan_result.changes),
scan_result.scanned_records,
file.filename,
scan_result.summary(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: Log injection through batch identifiers

scan_result.summary() includes each client-supplied custom_id, and the preceding argument includes the client-supplied filename. An authenticated caller can include CR/LF characters in either value and trigger a guardrail change to inject forged lines into server logs. Escape control characters and cap the rendered detail, or log only the counts here while retaining the structured report in metadata.

devin-ai-integration[bot]

This comment was marked as resolved.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review. Latest commit fixes the metadata loss, memory retention, blocking rewrite, log injection and typing findings

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 2c062b0. Configure here.

@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_record_redaction branch from 2c062b0 to 9180fb9 Compare August 20, 2026 07:59
@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review, new head pushed with the latest round of fixes

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

greptile-apps[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 47c8134. Configure here.

@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_record_redaction branch from 47c8134 to b65623b Compare August 20, 2026 19:17
@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review, new head

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

greptile-apps[bot]

This comment was marked as resolved.

cursor[bot]

This comment was marked as resolved.

@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_guardrail_redaction branch from 142045e to d2d124e Compare August 20, 2026 19:27
@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_record_redaction branch from b65623b to 5be8c5a Compare August 20, 2026 19:30
@yucheng-berri

Copy link
Copy Markdown
Contributor Author

Rebased onto current staging, which is where the missing test-quality gate script came from

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review, rebased head

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 5be8c5a. Configure here.

Base automatically changed from litellm_lit5276_batch_guardrail_redaction to litellm_internal_staging August 20, 2026 19:51
…ting the file

A single record tripping a guardrail rejected the whole upload, which is unusable for a file
holding thousands of rows. A record a guardrail rewrites is now submitted in its rewritten
form, a record it blocks is left out, and the create response reports every changed record by
both custom_id and line so a caller can reconcile against the file it sent. The same outcome
is written to the proxy log and to request metadata, so it is not visible only to the caller.

A rewritten record goes straight to a spool and only its offset is carried, so a masking
guardrail touching most rows of a large upload does not build a second copy of the file on the
heap, and the rewrite runs off the event loop the way the sibling full-file validation does.
Both proxy-injected metadata keys are captured from the record and restored exactly, including
an explicit null, so a masked row keeps the tags that decide how it is attributed.

A record is dropped only when a guardrail judged its content. `GuardrailRaisedException` now
carries `blocked_content` for that, because half its raise sites in the repo signal an
unreachable or unparseable backend under a fail-closed policy, and treating those as blocks
would turn "refuse this request" into "drop this record and submit the rest". The default is
off, so a raise that does not say what it means aborts the upload instead of silently
shrinking the file.
…lly reached

A guardrail that reports a technical failure as an HTTPException carrying a block status was
read as a content block, so an unreachable backend under a fail-closed policy quietly shrank the
file instead of failing the upload. Two in-tree integrations do exactly that, and one of them
defaults to fail-closed, so the broken configuration was the default one. Such an exception is
raised `from` the underlying error, which is a deliberate statement that something else caused
it, and no content verdict in the repo is raised that way, so the chain now settles it. Implicit
context is left alone, since a block raised inside an unrelated `except` would read as a failure.

Two annotation errors in the same family: the one GuardrailRaisedException subclass in tree never
opted into blocked_content, so a real block took the whole upload down with it, and straiker's
block helper is reached both from its verdict and from its fail-closed handler, so it claimed a
verdict for an outage. The helper now takes the flag from its caller.

A record could also opt itself out of the chain. Guardrail selection reads a body-level
`guardrails` key ahead of the proxy-injected list, and online that key can only add to the key
and team selection, never replace it, so a batch record naming an empty list skipped every
guardrail that was not default_on and was still reported as scanned. Every injected key is now
stripped before dispatch and restored afterwards.

A guardrail that reroutes a record to another model is honoured on the online path by rewriting
the model, which the scan read as a rewrite and submitted in the same file, sending content to
the provider the reroute existed to avoid. Every record of a batch file goes to one provider, so
the upload is refused instead, naming the line.

The scan spool is closed on the paths that never read it back.
…and close its spools

The narrowed request metadata was installed under `litellm_metadata` only, but a record is
scanned as the chat request it describes, and the guardrails that pick a policy from a request
header read `metadata` instead. Noma choosing an application and Aim choosing a user both look
there, so the header allowlist added for them did not reach either one and a batch record was
still evaluated under the fallback policy. The scan metadata now goes into both bags, which are
both stripped and restored, so neither survives into the record that ships.

The scan spool was closed on the paths that abort, which are exactly the paths where it is
empty, and left open on the one path where it holds the rewritten records. Nothing closed the
rewrite output either, where before this feature the uploaded handle belonged to Starlette. The
upload now owns both and closes them however it exits.
The scan spool was added to the request's cleanup list only after the rewrite returned, so a
rewrite that raised, which for a spilled file can be as ordinary as the disk filling up, jumped
to the handler with the list still empty and left the scan's own handle open. The rewrite also
left its half-written output behind on that path, since nothing owns that handle until it is
returned. Both now close.
@yucheng-berri
yucheng-berri force-pushed the litellm_lit5276_batch_record_redaction branch from 5be8c5a to 175991c Compare August 20, 2026 19:55
@yucheng-berri

Copy link
Copy Markdown
Contributor Author

Rebased onto staging now that #37519 has merged

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review, rebased head

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

if isinstance(exc, GuardrailRaisedException):
return exc.blocked_content
if exc.__cause__ is not None:
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Upstream failures become content blocks

When Zscaler AI Guard propagates an upstream HTTP 400, 403, or 422 without an explicit cause, this fallback classifies the technical failure as a content verdict, causing the affected record to be dropped while the remaining batch is submitted.

Knowledge Base Used: Guardrails

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 175991c. Configure here.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

Status-code heuristic is pre-existing staging code, and zscaler is untouched here. This PR narrows drops via blocked_content and cause

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review and rescore

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai The other 19 vendor sites conflate outage with verdict in one shape; fixing them is a separate contract change across 12 integrations. please re-review and rescore

@yucheng-berri
yucheng-berri merged commit e07a712 into litellm_internal_staging Aug 20, 2026
73 checks passed
@yucheng-berri
yucheng-berri deleted the litellm_lit5276_batch_record_redaction branch August 20, 2026 20:12
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Full assurance run at head 175991c375 (and #37519 at merged staging commit 3a31331435), live against the real OpenAI files API — all 10 scenarios passed: reject-on-guardrail-change and unreadable-record 400s at #37519's base, and here the redact/drop report (submitted_records: 3, line 2 redacted, line 3 dropped), OpenAI content readback with row-2 masked and rows 1/4 byte-identical, tag survival on rewritten records, fail-closed guardrail abort (no silent drop), all-blocked rejection, and the per-record proxy log line.

live recording of the full run

Key screenshots

mixed upload report
OpenAI readback checks
tags survive rewrite
fail-closed 400
all records blocked 400
proxy log line

@codspeed-hq

codspeed-hq Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_lit5276_batch_record_redaction (175991c) with litellm_internal_staging (d542c82)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (3a31331) during the generation of this report, so d542c82 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants