Skip to content

fix(azure_storage): name Data Lake objects without base64 padding or slashes - #43914

Merged
yucheng-berri merged 1 commit into
mainfrom
litellm_azure_storage_adls_safe_filename
Oct 1, 2026
Merged

yucheng-berri merged 1 commit into
mainfrom
litellm_azure_storage_adls_safe_filename

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • /v1/responses ids are resp_ plus standard base64, so they end in = or == and may contain /
  • The Azure Storage callback used {id}.json as the Data Lake object name
  • With an account key the aio SDK signs a padded name with %3D but sends a bare =, so Azure answers AuthenticationFailed
  • A / in the name is an InvalidUri on both auth paths
  • The callback swallows both errors, so those audit records are silently dropped

How it solves it:

  • The object name drops = and maps / to _; the id field inside the JSON is unchanged
  • Both upload paths (account key SDK, Entra ID REST) go through the same helper
  • The /v1/responses id itself is untouched, so previous_response_id keeps working

Intentional product change: on the Entra ID path a padded id used to land as resp_...==.json and now lands as resp_....json; the account key path never landed those objects at all. Tooling that looks objects up by {response_id}.json has to apply the same rewrite (the e2e reader added in #43082 will pick it up once that PR rebases on this one)

User Flow

Before: a proxy admin with success_callback: [azure_storage] and an account key sees every /v1/responses call in the logs page but most are missing from storage

  1. They send POST https://litellm-domain/v1/responses with {"model": "gpt-5.6", "input": "hi"} and get 200 with an id like resp_bGl0ZWxsbTpj...ZmFi==
  2. The proxy log shows ClientAuthenticationError: (AuthenticationFailed) Server failed to authenticate the request
  3. Listing the day's folder in the container shows the chat completion records but no resp_...== object

After: the same call lands in storage under a name Azure accepts

  1. They send the same POST https://litellm-domain/v1/responses and get the same id back
  2. The proxy log shows no Azure error
  3. The day's folder holds resp_bGl0ZWxsbTpj...ZmFi.json, and its id field is the exact id the client received

Linear ticket

Resolves LIT-9079

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Real Azure Data Lake Gen2 account (isolated resource group), the real AzureBlobStorageLogger, sequential uploads, ids shaped exactly like the gateway's /v1/responses ids (resp_ plus standard base64 of litellm:custom_llm_provider:...;response_id:..., padded to 0, 1 and 2 =), plus one id whose base64 contains //, one whose base64 contains +, and a chatcmpl- control. Each object is then listed back through the Data Lake SDK. Same script, same account, before and after

Deterministic proof in the same PR: tests/integration/observability/test_azure_storage_file_names.py runs a real proxy (Postgres, Redis, scripted upstream) and points the callback at a local TLS Data Lake sink that verifies the SharedKey signature the way the service does. Audit over test_azure_storage_file_names.py plus the four observability suites that share the changed tests/integration/_support/wire.py (41 nodes) with uv run --no-sync python tests/integration/run.py, same --seed and --order-seed on every leg, run ids phase12/72182792/base, head1, head2. The live Azure runs below were captured on the previous tip, whose azure_storage.py naming code is byte-identical to this one (the rebase onto main only dropped the unrelated client TTL hunks from the stacked base)

Before (merge base e61733b, main; live runs on 0b7c9ee, same naming code)

Account key path

  1. python lit9079_logger.py <unfixed worktree> (uploads the six ids, lists YYYY-MM-DD/)
  2. Output
    pad0     id_tail='k3ZWR4' landed=True  name=4OTk3ZWR4.json error=-
    pad1     id_tail='RmYjM=' landed=False name=- error=ClientAuthenticationError: (AuthenticationFailed) Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
    pad2     id_tail='F4eA==' landed=False name=- error=ClientAuthenticationError: (AuthenticationFailed) Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
    slash    id_tail='79//79' landed=False name=- error=HttpResponseError: (InvalidUri) The request URI is invalid.
    plus     id_tail='++++++' landed=True  name=+++++++++.json error=-
    chatcmpl id_tail='62cbd3' landed=True  name=pl-62cbd3.json error=-
    

Entra ID path

  1. Same script with AZURE_STORAGE_ACCOUNT_KEY unset and the tenant, client id and client secret set
  2. Output: the three padded ids are created under their raw = names, then the / id fails and the run stops
    Error creating file resource: Client error '400 Bad Request' for url 'https://<account>.dfs.core.windows.net/litellm-logs/resp_//79//79//79//79//79//79//79//79.json?resource=file'
    

Live proxy, account key path

  1. Two-worker proxy on this sha with success_callback: ["azure_storage"], general_settings: disable_responses_id_security: true (so the gateway's own resp_ id reaches the callback instead of the encrypted per-key rewrite the id security hook applies by default), model gpt-5.6 on the real OpenAI API, real account key
  2. for n in $(seq 0 11); do curl -s http://127.0.0.1:42177/v1/responses -H "Authorization: Bearer $KEY" -d "{\"model\":\"gpt-5.6\",\"input\":\"lit9079pad $n\"}"; done plus one /v1/chat/completions control. All 13 return 200, every /v1/responses id is resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9...=, one = of padding
  3. Listing 2026-09-30/ after the flush: the chatcmpl- control landed, none of the 12 resp_ objects exist. proxy.log carries 12 ClientAuthenticationError entries, Error occurred: (AuthenticationFailed) Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Deterministic integration test

  1. uv run --no-sync pytest tests/integration/observability/test_azure_storage_file_names.py -q against a proxy started from the unfixed tree
  2. assert sink.unauthenticated_targets() == () fails with four /litellm-logs/<day>%2Fresp_...=.json?resource=file targets, 1 failed in 15s
  3. Audit base leg (head test files against the unfixed proxy): 41 collected, 39 passed, 2 failed, 0 skipped. The reds are this cell and throttled_sink (an s3 throttle timing cell, red on every leg on this box)

After (7218279; live runs on 0764cdc, same naming code)

Account key path

  1. python lit9079_logger.py <this branch> (rerun at 7218279 after the rebase: all six rows land again)
  2. Output
    pad0     id_tail='YyZGF4' landed=True name=1NmYyZGF4.json error=-
    pad1     id_tail='BjY2M=' landed=True name=JkNzBjY2M.json error=-
    pad2     id_tail='V4eA==' landed=True name=GI3MDV4eA.json error=-
    slash    id_tail='79//79' landed=True name=9__79__79.json error=-
    plus     id_tail='++++++' landed=True name=+++++++++.json error=-
    chatcmpl id_tail='4b79af' landed=True name=pl-4b79af.json error=-
    

Entra ID path

  1. Same script with the account key unset and the Entra ID variables set
  2. Output
    pad0     id_tail='ViZTZ4' landed=True name=jZDViZTZ4.json error=-
    pad1     id_tail='gwNjY=' landed=True name=E2ZDgwNjY.json error=-
    pad2     id_tail='F4eA==' landed=True name=mNlNmF4eA.json error=-
    slash    id_tail='79//79' landed=True name=9__79__79.json error=-
    plus     id_tail='++++++' landed=True name=+++++++++.json error=-
    chatcmpl id_tail='6f2f31' landed=True name=pl-6f2f31.json error=-
    

Live proxy, account key path

  1. Same proxy config and the same 12 plus 1 commands against this branch (port 58045)
  2. Every /v1/responses id again ends in one =. Listing 2026-09-30/: all 12 land as resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9....json with the = removed, and each stored JSON id is byte-equal to the padded id the client received. proxy.log has zero AuthenticationFailed and zero ClientAuthenticationError

Deterministic integration test

  1. Same pytest command against a proxy started from this branch
  2. 1 passed in 16.74s; six /v1/responses ids with one and two = all land, every stored payload carries the original id, all names match ^[A-Za-z0-9._+-]+\.json$ and are distinct
  3. Audit head legs, run four times: 41 collected, 0 skipped, identical collected sets, test_azure_storage_file_names.py green on every leg. Two or three s3 and passthrough timing cells (slow_sink, throttled_sink, cache_hit_twins, upstream_failure_events, passthrough_sigkill) flip between legs with httpx.ReadTimeout in the request burst; each passes when run alone on the head rig and throttled_sink and passthrough_sigkill also fail on the base rig, so they read as box load rather than this diff, which does not touch their code path

Mutation checks on the unit tests (tests/unit/integrations/azure_storage/test_azure_storage.py, 25 passed): keeping = in the translation table turns the three padded cases plus both upload path tests red; putting the raw f-string back at the SDK call site turns the account key upload test red; both restored

Type

🐛 Bug Fix

Caveats (if any)

Medium

  • Entra ID operators who look objects up by {response_id}.json see padded ids under a new name from this release on (see the intentional product change above)

Low

  • + is left in the name on purpose, a + id lands today on both paths, so rewriting it would rename working objects
  • Distinctness is guaranteed for ids from one base64 alphabet; a standard base64 id with / and a URL-safe id with _ at the same position could map to one name, which needs two upstream ids that differ in exactly that character
  • Characters outside base64 (?, #, space) pass through untouched as before; no gateway id contains them
  • With the default id security hook enabled the proxy replaces the gateway resp_ id with an encrypted URL-safe one before the callback sees it, so the earlier live A/B on that default landed every record on both sides; the A/B above sets disable_responses_id_security: true to expose the padded id, which is the shape the ticket reports

QA runbook

  • Manual check against a real Data Lake account: a record logged through /v1/responses is found under the padding-free object name
    • Start a proxy with success_callback: ["azure_storage"], AZURE_STORAGE_ACCOUNT_NAME, AZURE_STORAGE_FILE_SYSTEM and AZURE_STORAGE_ACCOUNT_KEY
    • POST http://localhost:4000/v1/responses with {"model": "<your model>", "input": "hi"} and note the resp_...= id
    • List the day's folder in the container and expect an object named like the id with the trailing = removed, whose JSON id equals the id you received
    • Repeat with AZURE_STORAGE_ACCOUNT_KEY unset and AZURE_STORAGE_TENANT_ID, AZURE_STORAGE_CLIENT_ID, AZURE_STORAGE_CLIENT_SECRET set, and expect the same object at the container root
    • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ran /live-pr-risk and found no regressions/backward incompatible risks

REVIEWER MUST KNOW BEFORE APPROVING

  • Entra ID path object name for padded ids
    • before: a /v1/responses record with id resp_...== lands as resp_...==.json
    • after: the same record lands as resp_....json, the JSON id field is unchanged
    • approval: pending, yucheng-berri in the Devin session linked below
  • Ids that differ only by / versus _, or by =, share one object name
    • before: an id containing / is rejected by the service (InvalidUri) on both paths and its record is dropped
    • after: it lands under the _ name and overwrites a record whose id already had _ there, if one exists; no gateway resp_, litellm uuid or chatcmpl- id can produce this pair
    • approval: pending, yucheng-berri in the Devin session linked below

Link to Devin session: https://app.devin.ai/sessions/5c920e07636044cebaef8e7a723300fe
Open in Devin Desktop: https://app.devin.ai/desktop/session/5c920e07636044cebaef8e7a723300fe?variant=devin
Requested by: @yucheng-berri


Devin Review


Note

Medium Risk
Changes observability object naming for padded or slash-containing ids (operators looking up {response_id}.json must apply the same rewrite), but fixes failed uploads that previously dropped audit records.

Overview
Fixes silent audit log loss when the Azure Storage callback uploads /v1/responses records whose ids include base64 = padding or / characters.

Adds adls_safe_file_name, which strips = and maps / → _ for the Data Lake object name (both account-key SDK and Entra ID REST upload paths now use it). The JSON body still stores the original id unchanged.

Test coverage adds unit cases for the rewrite and upload paths, plus integration harness (TLS wire peer, SharedKey-verifying Data Lake sink) and an end-to-end proxy test that padded resp_ ids authenticate and land under ADLS-safe names.

Reviewed by Cursor Bugbot for commit 7218279. Bugbot is set up for automated code reviews on this repo. Configure here.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@greptile-apps

greptile-apps Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Changes how file names are generated for cloud storage uploads.

The PR appears safe to merge based on the changes in its six-file diff.

Summary

The PR rewrites Azure Data Lake object names to remove base64 padding and replace slashes while preserving the original ID in the stored JSON. It applies the same naming helper to account-key and Entra ID uploads and adds unit and proxy integration coverage.

Reviews (3) · Last reviewed commit: "fix(azure_storage): name Data Lake objec..."

Comment thread litellm/integrations/azure_storage/azure_storage.py
Comment thread litellm/integrations/azure_storage/azure_storage.py

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment on lines +36 to +40
def adls_safe_file_name(payload_id: str | None) -> str:
"""`=` padding and `/` in a base64 payload id are what the Data Lake service rejects, so the name drops the
padding and maps `/` to `_`. Standard base64 has no `_` and its padding is fixed by the length, so ids from
that alphabet stay distinct; anything else is left as is."""
return f"{(payload_id or str(uuid.uuid4())).translate(_ADLS_SAFE_NAME)}.json"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Distinct payload IDs overwrite one object

When IDs differ only by / versus _, adls_safe_file_name gives both the same name. The later upload overwrites the earlier audit record.

Learn more

The Azure callback handles every StandardLoggingPayload, whose id contract permits any string. Replacing / with _ is injective only for IDs restricted to standard base64, but this shared callback also receives provider IDs and LiteLLM call IDs. Deleting every = creates additional collisions for IDs outside canonical padded base64.

Example: Payload IDs provider/a_b and provider/a/b both become provider_a_b.json. The second create_file() call reuses that path, so storage retains only the second payload.

Recommended fix: Encode the complete ID with an injective filename-safe encoding, such as unpadded URL-safe base64 of its UTF-8 bytes. Apply the same helper in both writers and all readers.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the mapping is not injective over arbitrary strings, and the docstring says so (distinct only within one base64 alphabet). Keeping it as is on purpose:

  • Every id this callback sees is a litellm call id (uuid4), a provider id, or a gateway resp_ id (resp_ plus standard base64). Standard base64 never contains _, and the padding is fixed by the length, so two gateway ids cannot collide. A collision needs a provider id and a gateway id that differ only in / vs _ at the same position, which does not occur in practice
  • An injective encoding of the whole id would rename every object, including chatcmpl-... and uuid names that land fine today, and make the object name unreadable next to the logs page id. That is a larger observable change than this bug fix; the leftover collision case is noted as a Low caveat in the PR body

@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

bugbot run

1 similar comment
@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor

cursor Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

Copy link
Copy Markdown
Contributor

bugbot run

@cursor

cursor Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

…slashes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@CLAassistant

CLAassistant commented Sep 30, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration
devin-ai-integration Bot changed the base branch from litellm_azure_storage_client_ttl to main September 30, 2026 21:28
@devin-ai-integration
devin-ai-integration Bot requested a review from a team September 30, 2026 21:28
@greptile-apps

greptile-apps Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings could not be posted inline.

  • P1 Client expiry check is reversed litellm/integrations/azure_storage/azure_storage.py:342 ▶

    On the account-key path, each upload requests the cached Data Lake client. This > check closes the client while it is still within its TTL, so successive audit records cannot reuse its connection. Once the TTL passes, the check leaves the expired client in place, preventing the intended recovery from stale connections. Please restore the expiry condition and the removed TTL-reuse tests.

@codspeed

codspeed Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_azure_storage_adls_safe_filename (7218279) with main (6fd9334)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (e61733b) during the generation of this report, so 6fd9334 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

The reversed TTL check is pre-existing main code, fixed separately in #43082 with its reuse tests; this PR only changes object names

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Azure Storage Explorer view of litellm-logs/2026-09-30/ after the fix: padded resp_ ids land under padding-stripped names, / mapped to _

Storage Explorer listing

@yucheng-berri

Copy link
Copy Markdown
Contributor

@greptileai review latest head

@yucheng-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 7218279. Configure here.

@yucheng-berri
yucheng-berri merged commit 2c3866e into main Oct 1, 2026
103 of 109 checks passed
@yucheng-berri
yucheng-berri deleted the litellm_azure_storage_adls_safe_filename branch October 1, 2026 01:17
jan-sauer-reef added a commit to jan-sauer-reef/litellm that referenced this pull request Oct 1, 2026
…ject_key_prefix

* upstream/main: (62 commits)
  fix(guardrails): scan Responses API input in Azure Prompt Shield (BerriAI#43786)
  feat(lens): investigate sampled traces and retain batch results (BerriAI#43942)
  fix(proxy): restore pre-config-wins handling of pass-through endpoints (BerriAI#43962)
  fix(cost-map): raise baseten DeepSeek-V4.1-Flash max output to 262144 (BerriAI#43916)
  chore(cost-map): add deprecation date for anthropic claude-sonnet-4-5 (BerriAI#43898)
  chore(cost-map): add fireworks inkling priority prices from the prices api (BerriAI#43949)
  feat(guardrails): honor litellm_params.timeout in every HTTP guardrail (BerriAI#43134)
  test(e2e): typed per-test metadata for the e2e suite (BerriAI#42044)
  fix(caching): write the response-cache SET to Redis at once instead of on the post-call batch (BerriAI#43973)
  feat(ui): filter tags by name and description on the Tag Management page (BerriAI#42949)
  feat(providers): add Cortecs as an OpenAI-compatible provider (BerriAI#43872)
  feat(e2e): record each e2e test's steps, starting with ProxyClient (BerriAI#42393)
  test(ci): repair stale tests and move retired OpenAI text-completion fixtures (BerriAI#43958)
  feat(proxy): record in spend logs whether a request used a client-forwarded Anthropic OAuth token (BerriAI#43063)
  fix(azure_storage): keep the DataLakeServiceClient alive until its TTL elapses (BerriAI#43082)
  chore(deps): bump gitpython and tornado, extend diskcache osv ignore to Nov 1 (BerriAI#43961)
  fix(guardrails): treat an unknown straiker api_version as unset instead of skipping the guardrail (BerriAI#43956)
  fix(azure_storage): name Data Lake objects without base64 padding or slashes (BerriAI#43914)
  fix(grayswan): send request conversation and tool calls to post-call monitor (BerriAI#43770)
  chore(cost-map): sync openrouter prices from the models API (BerriAI#43950)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants