Skip to content

feat(azure_storage): upload audit logs to Azure Storage via audit_log_callbacks - #43838

Merged
yassin-berriai merged 1 commit into
mainfrom
litellm_azure_storage_audit_logs
Oct 10, 2026
Merged

yassin-berriai merged 1 commit into
mainfrom
litellm_azure_storage_audit_logs

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • audit_log_callbacks: ["azure_storage"] loads the logger but writes nothing
  • The Azure logger never overrode the audit callback, so events hit the base no-op

How it solves it:

  • AzureBlobStorageLogger.async_log_audit_log_event uploads each audit event as JSON
  • Files land under audit_logs/YYYY-MM-DD/HH-MM-SS_<audit id>.json in the configured file system
  • Reuses the existing account key and Entra ID (service principal) upload paths

User Flow

Before: an admin who routes audit logs to Azure Storage finds the container empty

  1. The admin sets store_audit_logs: true and audit_log_callbacks: ["azure_storage"] and restarts the proxy
  2. They create, update and regenerate a key and create a team with POST http://localhost:4000/key/generate, /key/update, /key/{key}/regenerate, /team/new, all 200
  3. GET http://localhost:4000/audit and http://localhost:4000/ui/logs/ (Audit Logs tab) list the four audit rows
  4. The Azure Storage file system has no audit files

After: the same actions produce one JSON file per audit event in Azure Storage

  1. The admin sets the same config and restarts the proxy
  2. They run the same four management calls, all 200
  3. GET http://localhost:4000/audit and the Audit Logs tab list the four audit rows
  4. The file system now has audit_logs/<today>/<time>_<audit id>.json for each of the four rows, with the key value masked the same way the DB row is

Linear ticket

Resolves LIT-6128

Files changed

File What changed and why
litellm/integrations/azure_storage/azure_storage.py Adds async_log_audit_log_event plus _upload_json_to_file_path, which writes one file through the account key SDK client or the Entra ID DFS REST calls already used for request logs. The account key SDK sequence (create_file, append_data, flush_data) moves into _upload_bytes_with_account_key so the request log path and the audit path share it instead of duplicating it

Audit events are uploaded one file per event straight from the audit dispatcher task, not through the request log queue, so request log batches keep their current shape and file naming. Upload failures propagate to the dispatcher's task done callback, which already logs them. The audit payload reaching the callback is the LiteLLM_AuditLogs model after its mask_api_keys validator ran, so key fields arrive as sk-r****yFFQ and never as the raw virtual key

flowchart LR
  A[key/team/user change] --> B[create_audit_log_for_update]
  B --> C[_dispatch_audit_log_to_callbacks]
  C -->|asyncio task| D[AzureBlobStorageLogger.async_log_audit_log_event]
  D -->|account key| E[_upload_bytes_with_account_key]
  D -->|Entra ID| F[DFS REST PUT/PATCH/PATCH]
  E --> G[audit_logs/YYYY-MM-DD/HH-MM-SS_id.json]
  F --> G
Loading

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Regime: a real Azure Storage account (ADLS Gen2, HNS on, core.windows.net), real Postgres, the proxy started from litellm/proxy/proxy_cli.py with PYTHONPATH set to the checkout under test. No mocks on the audit path (the only model is a mock_response model, used once to show request logs still upload). Before and After use the same config and the same four management calls against a fresh, empty file system per leg. Secrets live only in the proxy's environment, the curl lines below omit the Authorization header and the raw generated key is shown as <new key>

Shared config:

model_list:
  - model_name: mock-gpt
    litellm_params:
      model: openai/mock-gpt
      api_key: fake-key
      mock_response: hello from mock
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  store_model_in_db: true
litellm_settings:
  callbacks: ["azure_storage"]
  store_audit_logs: true
  audit_log_callbacks: ["azure_storage"]

Shared commands ($H is -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json"):

curl -s $H localhost:$PORT/key/generate -d '{"key_alias":"<leg>-key","max_budget":10}'
curl -s $H localhost:$PORT/key/update -d '{"key":"<new key>","max_budget":20}'
curl -s $H localhost:$PORT/key/<new key>/regenerate -d '{}'
curl -s $H localhost:$PORT/team/new -d '{"team_alias":"<leg>-team"}'

Before (6fcf9ef, main without this PR)

Account key

  1. Proxy startup log prints Initialized Audit Log Callbacks - ['azure_storage']
  2. The four calls return 200 OK (POST /key/generate, POST /key/update, POST /key/<new key>/regenerate, POST /team/new in the access log)
  3. Listing the file system lit6128-main-key with the DFS REST API (?resource=filesystem&recursive=true) returns no paths. The audit rows exist in Postgres, nothing reaches Azure

After (ee4c659, the same single commit as 3b73c6c rebased onto main after #45845, the proof below was captured at 3b73c6c)

Account key

  1. Proxy on port 4002 with AZURE_STORAGE_ACCOUNT_KEY set, startup log prints Initialized Audit Log Callbacks - ['azure_storage']
  2. The four calls
$ curl -s localhost:4002/key/generate -d '{"key_alias":"pr2-key-key","max_budget":10}'
{"key_alias": "pr2-key-key", "token_id": "1dd928d2744622ce4553d8214dbe62315cb85d7eaab640357d84bee2906c4754", "max_budget": 10.0}
$ curl -s localhost:4002/key/update -d '{"key":"<new key>","max_budget":20}'
{"max_budget": 20.0}
$ curl -s localhost:4002/key/<new key>/regenerate -d '{}'
{"key_alias": "pr2-key-key", "token_id": "4bfba28d77eae4e80c7ff4fe11bac1a3dffe4cb460aa36e74c0fe3e287cb0948"}
$ curl -s localhost:4002/team/new -d '{"team_alias":"pr2-key-team"}'
{"team_alias": "pr2-key-team", "team_id": "75053a51-e011-48f6-ac1f-4e032e85a077"}
  1. Listing lit6128-pr2-key shows one file per audit row
audit_logs/2026-10-10/19-16-45_30f941d4-6150-4d97-a861-4b6301d13a6c.json
audit_logs/2026-10-10/19-16-45_3918924e-a194-492b-8faa-4ce6b7432b55.json
audit_logs/2026-10-10/19-16-45_a3b51904-fbc8-4f1e-8c45-30696e4d1d7d.json
audit_logs/2026-10-10/19-16-45_bf43ab7d-9614-4ce9-a783-7a5037b6c494.json
  1. The created file holds the masked key, not the raw one
{
  "id": "30f941d4-6150-4d97-a861-4b6301d13a6c",
  "action": "created",
  "table_name": "LiteLLM_VerificationToken",
  "object_id": "1dd928d2744622ce4553d8214dbe62315cb85d7eaab640357d84bee2906c4754",
  "updated_values": {"key": "sk-r*****************yFFQ", "key_alias": "pr2-***-key", "team_id": null}
}
  1. grep -c "Traceback" on the proxy log for this run returns 0

Service principal (Entra ID, no account key set)

  1. Proxy on port 4003 with AZURE_STORAGE_TENANT_ID, AZURE_STORAGE_CLIENT_ID, AZURE_STORAGE_CLIENT_SECRET and no AZURE_STORAGE_ACCOUNT_KEY, so uploads go through the DFS REST path
  2. The four calls
$ curl -s localhost:4003/key/generate -d '{"key_alias":"pr2-sp-key","max_budget":10}'
{"key_alias": "pr2-sp-key", "token_id": "f9664f4ec146db1eb0df9c88941bb937baaba6838cec104ed7d0e53c03f8c91d", "max_budget": 10.0}
$ curl -s localhost:4003/key/update -d '{"key":"<new key>","max_budget":20}'
{"max_budget": 20.0}
$ curl -s localhost:4003/key/<new key>/regenerate -d '{}'
{"key_alias": "pr2-sp-key", "token_id": "9485282636795190575a06a86f55b6a5a09f3e6a6a03875f2dabcaa3a9fb633c"}
$ curl -s localhost:4003/team/new -d '{"team_alias":"pr2-sp-team"}'
{"team_alias": "pr2-sp-team", "team_id": "fcf5d057-6675-4150-8b93-bcf05f684236"}
  1. Listing lit6128-pr2-sp shows one file per audit row
audit_logs/2026-10-10/19-16-46_8d2b2111-985d-4756-9ea2-75cb9c21eb45.json
audit_logs/2026-10-10/19-16-46_c12dcdc4-f9c1-4023-83be-f5f30c2ab16c.json
audit_logs/2026-10-10/19-16-46_d2c38ed7-2d45-466b-847d-a5c972a6e3fe.json
audit_logs/2026-10-10/19-16-47_6f6d6050-6b4d-4364-bada-dfbbd023fe79.json
  1. The team file
{"id": "6f6d6050-6b4d-4364-bada-dfbbd023fe79", "action": "created", "table_name": "LiteLLM_TeamTable", "changed_by_api_key": "litellm_proxy_master_key"}
  1. grep -c "Traceback" on the proxy log returns 0

Both listings and both payload reads in one terminal capture, with the count-only secret scrub of the proof text and both proxy logs at the bottom (account key, client secret, master key and every raw generated key all 0 hits):

Azure listings for the account key and Entra ID legs, masked payloads, secret scrub all zero

Request logs keep working through the shared account key helper

  1. On the port 4002 proxy, curl -s $H localhost:4002/chat/completions -d '{"model":"mock-gpt","messages":[{"role":"user","content":"hi"}]}' returns {"id": "chatcmpl-384c5fbb-e5f3-4654-a316-3e2f7cedbfad", "content": "hello from mock"}
  2. Proxy log: AzureBlobStorageLogger - about to flush 1 events then Successfully uploaded and wrote to 2026-10-10/chatcmpl-384c5fbb-e5f3-4654-a316-3e2f7cedbfad.json
  3. Reading that path from lit6128-pr2-key returns {"id": "chatcmpl-384c5fbb-e5f3-4654-a316-3e2f7cedbfad", "model": "openai/mock-gpt", "status": "success"}, so the request log file name and location are unchanged

Admin UI

  1. Open http://localhost:4002/ui/logs/ and click the Audit Logs tab
  2. The rows at 19:16:45 (Keys created, updated, rotated, Teams created, object id 1dd928d2...) are the four account key files above, the rows at 19:16:46 and 19:16:47 (object id f9664f4e...) are the four Entra ID files

Audit Logs tab listing the created, updated, rotated and team rows uploaded to Azure

The dashboard has no control for choosing audit log destinations for any callback (no reference to audit_log_callbacks under ui/litellm-dashboard/src, S3 audit export is also config only), so this PR adds none

Type

New Feature

Caveats (if any)

Low

  • One file per audit event, no batching, so a burst of N events is N uploads
  • The screenshots are Devin attachment links until the public media repo accepts a push from this box

Link to Devin session: https://app.devin.ai/sessions/373eb10be9964560bd136af82e162bd9
Open in Devin Desktop: https://app.devin.ai/desktop/session/373eb10be9964560bd136af82e162bd9?variant=devin
Requested by: @yassin-berriai

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@yassin-berriai

Copy link
Copy Markdown
Contributor

bugbot run

@yassin-berriai

Copy link
Copy Markdown
Contributor

@greptileai

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@greptile-apps

greptile-apps Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] This PR appears safe to merge; no new blocking issue was found.

Summary

Adds Azure Storage uploads for audit events and shares the account-key upload steps with request logs.

  • Azure Storage saves each audit event as a dated JSON file.

Reviews (5) · Last reviewed commit: "feat(azure_storage): upload audit logs t..." · Reviewed by Greptile

Comment thread litellm/integrations/azure_storage/azure_storage.py Outdated
@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_azure_storage_audit_logs branch from 1e97f3b to dc12de3 Compare September 30, 2026 09:24
@yassin-berriai

Copy link
Copy Markdown
Contributor

bugbot run

@yassin-berriai

Copy link
Copy Markdown
Contributor

@greptileai

Comment thread litellm/integrations/azure_storage/azure_storage.py

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed

codspeed Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

✅ 31 untouched benchmarks


Comparing litellm_azure_storage_audit_logs (ee4c659) with main (691ca03)

Open in CodSpeed

@yassin-berriai

Copy link
Copy Markdown
Contributor

@greptileai

@yassin-berriai
yassin-berriai requested review from yucheng-berri and removed request for yucheng-berri September 30, 2026 09:44
@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_azure_storage_audit_logs branch from dc12de3 to 56e21af Compare October 2, 2026 23:08
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai please review the rebased head 56e21af (single commit on current main; the client expiry change is gone from the diff because main already fixed it in #43082)

@yassin-berriai

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 56e21af. Configure here.

@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_azure_storage_audit_logs branch 2 times, most recently from e0d6ece to 3b73c6c Compare October 10, 2026 19:26
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai please review the current head 3b73c6c, the account key upload path now shares one helper with the request log path

…_callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot force-pushed the litellm_azure_storage_audit_logs branch from 3b73c6c to ee4c659 Compare October 10, 2026 20:23
@yassin-berriai
yassin-berriai merged commit 60f35e3 into main Oct 10, 2026
89 checks passed
@yassin-berriai
yassin-berriai deleted the litellm_azure_storage_audit_logs branch October 10, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants