fix(spend-tracking): keep batch spend keys joinable after v1.99 provenance gate

[mateo-berri]

TLDR

Problem this solves:

• Since v1.99, some spend rows store a double-hashed api_key
• Usage / BI can no longer join those rows to the virtual key
• Spend shows up as key-hash-... with null api_key_alias and user_email

How it solves it:

• Stamp user_api_key_hash on batch cost, legacy queue, and callback-log replay writers
• Shared reverse-hash recovery restores alias and email for historical dirty rows in Usage, CloudZero, and Focus
• Recovery hashes the token column inside Postgres (one lookup over active keys, one over deleted keys), so no key rows cross the wire and nothing pages
• Usage Key Activity now copies recovered user_email onto every key breakdown

User Flow

Before: a proxy admin looks at Usage Key Activity after a managed batch finishes and cannot tell which virtual key spent the money

1. Their app creates a managed batch with a named virtual key (for example alias batch-worker) owned by alice@example.com
2. The batch completes and the proxy bills its cost into daily spend
3. They open https://litellm-domain/ui/?page=usage Key Activity (or pull the same data into BI / CloudZero)
4. That spend appears under key-hash-<64 hex chars> with empty api_key_alias and empty user_email, so cost attribution looks unassigned

After: the same batch spend stays attached to the creating key's alias and user

1. Their app creates the same managed batch with alias batch-worker
2. The batch completes and the proxy bills its cost into daily spend
3. They open https://litellm-domain/ui/?page=usage Key Activity (or pull the same data into BI / CloudZero)
4. That spend appears under batch-worker with alice@example.com
5. Already-written dirty rows from the buggy window also resolve alias and email via reverse-hash / UserTable recovery

Relevant issues

Customer report: since 1.99, hashed API keys show with no user_email / api_key_alias, dirtying cost data and Usage Key Activity

Regression introduced by #30736 (security fix for #30670), shipped in v1.99.0. Already-hashed writer paths that omitted user_api_key_hash got hashed a second time

Linear ticket

Resolves LIT-6852

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Verdict: PASS

Topology: one proxy per leg, booted from that leg's commit with --num_workers 2 (two worker processes, each running the managed-batch cost poller with proxy_batch_polling_interval: 15), sharing one local Postgres database lit6852_qa, enterprise license loaded for managed batches, real OpenAI calls to gpt-5.4-nano. Before ran at the merge base ab0478f068 (port 44948, exports on a second base proxy on 24381). After ran at the tip b93cf2e20a (port 39869) with every base proxy stopped first, so only tip pollers bill batches. $LITELLM_MASTER_KEY, $BATCH_WORKER_KEY and $QUEUE_WORKER_KEY stand in for the real keys. The v1.99 double hash of the batch-worker token d4cdba72... is b5e1826d..., and of the queue-worker token 184af3b2... is 085604bc...

Shared setup (run once against the base proxy, both legs reuse the keys)

$ curl -s http://127.0.0.1:44948/user/new -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"user_email":"alice-lit6852@example.com","user_role":"internal_user"}' | jq "{user_id, user_email}"
{
  "user_id": "0c0ba4ba-4e37-4c3f-a89f-2419b104566b",
  "user_email": "alice-lit6852@example.com"
}

$ curl -s http://127.0.0.1:44948/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"key_alias":"batch-worker","user_id":"0c0ba4ba-4e37-4c3f-a89f-2419b104566b","models":["gpt-5.4-nano"]}' | jq "{key_alias, user_id, token_id}"
{
  "key_alias": "batch-worker",
  "user_id": "0c0ba4ba-4e37-4c3f-a89f-2419b104566b",
  "token_id": "d4cdba721cfc67d3d38ebe975cf6820e4b65a2c3b5f6baf4b338607f7bc65815"
}

$ curl -s http://127.0.0.1:44948/user/new -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"user_email":"queue-admin-lit6852@example.com","user_role":"proxy_admin"}' | jq "{user_id, user_email}"
{
  "user_id": "1131c480-d107-4c12-884a-d9969394879a",
  "user_email": "queue-admin-lit6852@example.com"
}

$ curl -s http://127.0.0.1:44948/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"key_alias":"queue-worker","user_id":"1131c480-d107-4c12-884a-d9969394879a","models":["gpt-5.4-nano"]}' | jq "{key_alias, user_id, token_id}"
{
  "key_alias": "queue-worker",
  "user_id": "1131c480-d107-4c12-884a-d9969394879a",
  "token_id": "184af3b2937677e72f25cdb75a64f41f2c0e8228069cda1569a247271c247b4e"
}

Before (ab0478f)

The managed batch is created with the batch-worker key, the legacy queue call with the queue-worker key, then Usage Key Activity and the CloudZero and Vantage exports are read back. Both spends land under the double hash with no alias and no email

$ curl -s http://127.0.0.1:44948/v1/files -H "Authorization: Bearer $BATCH_WORKER_KEY" -F purpose=batch -F target_model_names=gpt-5.4-nano -F file=@batch.jsonl | jq "{id, purpose, status}"
{
  "id": "bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07dW5pZmllZF9pZCwyMTkzMDkxNS1kMDQ2LTQ1ZjYtYTliYS0zMzg2OWRlMjA5ZTg7dGFyZ2V0X21vZGVsX25hbWVzLGdwdC01LjQtbmFubztsbG1fb3V0cHV0X2ZpbGVfaWQsZmlsZS0xejRZQTVSVEZFUW9ublpwTlR0ZmNOO2xsbV9vdXRwdXRfZmlsZV9tb2RlbF9pZCxkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjlj",
  "purpose": "batch",
  "status": "uploaded"
}

$ curl -s http://127.0.0.1:44948/v1/batches -H "Authorization: Bearer $BATCH_WORKER_KEY" -H "Content-Type: application/json" -d '{"input_file_id":"bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07dW5pZmllZF9pZCwyMTkzMDkxNS1kMDQ2LTQ1ZjYtYTliYS0zMzg2OWRlMjA5ZTg7dGFyZ2V0X21vZGVsX25hbWVzLGdwdC01LjQtbmFubztsbG1fb3V0cHV0X2ZpbGVfaWQsZmlsZS0xejRZQTVSVEZFUW9ublpwTlR0ZmNOO2xsbV9vdXRwdXRfZmlsZV9tb2RlbF9pZCxkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjlj","endpoint":"/v1/chat/completions","completion_window":"24h"}' | jq "{id, status}"
{
  "id": "bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjljO2xsbV9iYXRjaF9pZDpiYXRjaF82YTlhMWRlZjMyYWM4MTkwODUzYjJlMzBjMTA1ZWY4ZA",
  "status": "validating"
}

$ until [ "$(curl -s http://127.0.0.1:44948/v1/batches/bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjljO2xsbV9iYXRjaF9pZDpiYXRjaF82YTlhMWRlZjMyYWM4MTkwODUzYjJlMzBjMTA1ZWY4ZA -H "Authorization: Bearer $BATCH_WORKER_KEY" | jq -r .status)" = completed ]; do sleep 30; done; curl -s ... | jq "{id, status, request_counts}"
{
  "id": "bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjljO2xsbV9iYXRjaF9pZDpiYXRjaF82YTlhMWRlZjMyYWM4MTkwODUzYjJlMzBjMTA1ZWY4ZA",
  "status": "completed",
  "request_counts": {
    "completed": 1,
    "failed": 0,
    "total": 1
  }
}
batch-terminal 01:26:36Z

$ curl -s http://127.0.0.1:44948/queue/chat/completions -H "Authorization: Bearer $QUEUE_WORKER_KEY" -H "Content-Type: application/json" -d '{"model":"gpt-5.4-nano","messages":[{"role":"user","content":"Reply with the single word: queue-before"}],"priority":0}' | jq "{id, model, content: .choices[0].message.content}"
{
  "id": "chatcmpl-EKD3y3OR6f7fEcSm2zv89plPReFaZ",
  "model": "gpt-5.4-nano-2026-03-17",
  "content": "queue-before"
}

$ curl -s "http://127.0.0.1:44948/user/daily/activity?start_date=2026-09-03&end_date=2026-09-05" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq "[.results[].breakdown.api_keys | to_entries[] | {api_key: .key, spend: .value.metrics.spend, requests: .value.metrics.api_requests, metadata: .value.metadata}]"
[
  {
    "api_key": "085604bcc373b586e34f0e3e7f8cc49adfccfa3e606500e7c66efad883601671",
    "spend": 0.00000905,
    "requests": 1,
    "metadata": {
      "key_alias": null,
      "team_id": null
    }
  },
  {
    "api_key": "d4cdba721cfc67d3d38ebe975cf6820e4b65a2c3b5f6baf4b338607f7bc65815",
    "spend": 0.0,
    "requests": 2,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null
    }
  },
  {
    "api_key": "b5e1826d5155b8b40e54347031531af039f2fe526aba27751650d45bbdb58b5a",
    "spend": 0.0000038,
    "requests": 1,
    "metadata": {
      "key_alias": null,
      "team_id": null
    }
  }
]

$ curl -s -X POST http://127.0.0.1:24381/cloudzero/dry-run -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"limit": 50}' | jq '{status: .status, rows: (.dry_run_data.usage_data | length), keys: [.dry_run_data.usage_data[] | {api_key: .api_key[0:12], api_key_alias, user_email, spend}] | unique}'
{
  "status": "success",
  "rows": 7,
  "keys": [
    {
      "api_key": "085604bcc373",
      "api_key_alias": null,
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "184af3b29376",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "b5e1826d5155",
      "api_key_alias": null,
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0000038
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0000038
    }
  ]
}

$ curl -s -X POST http://127.0.0.1:24381/vantage/dry-run -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"limit": 50}' | jq '{status: .status, rows: (.dry_run_data.usage_data | length), keys: [.dry_run_data.usage_data[] | {api_key: .api_key[0:12], api_key_alias, user_email, spend}] | unique}'
{
  "status": "success",
  "rows": 7,
  "keys": [
    {
      "api_key": "085604bcc373",
      "api_key_alias": null,
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "184af3b29376",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "b5e1826d5155",
      "api_key_alias": null,
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0000038
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0000038
    }
  ]
}

After (b93cf2e)

Key Activity on the tip proxy right before the After leg, for the deltas below

$ curl -s "http://127.0.0.1:39869/user/daily/activity?start_date=2026-09-03&end_date=2026-09-05" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq "[.results[].breakdown.api_keys | to_entries[] | {api_key: .key, spend: .value.metrics.spend, requests: .value.metrics.api_requests, metadata: .value.metadata}]"
[
  {
    "api_key": "085604bcc373b586e34f0e3e7f8cc49adfccfa3e606500e7c66efad883601671",
    "spend": 0.00000905,
    "requests": 1,
    "metadata": {
      "key_alias": "queue-worker",
      "team_id": null,
      "user_email": "queue-admin-lit6852@example.com"
    }
  },
  {
    "api_key": "184af3b2937677e72f25cdb75a64f41f2c0e8228069cda1569a247271c247b4e",
    "spend": 0.00001955,
    "requests": 2,
    "metadata": {
      "key_alias": "queue-worker",
      "team_id": null,
      "user_email": "queue-admin-lit6852@example.com"
    }
  },
  {
    "api_key": "d4cdba721cfc67d3d38ebe975cf6820e4b65a2c3b5f6baf4b338607f7bc65815",
    "spend": 0.0000038,
    "requests": 8,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null,
      "user_email": "alice-lit6852@example.com"
    }
  },
  {
    "api_key": "b5e1826d5155b8b40e54347031531af039f2fe526aba27751650d45bbdb58b5a",
    "spend": 0.000008324999999999999,
    "requests": 2,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null,
      "user_email": "alice-lit6852@example.com"
    }
  }
]

Same flow against the tip proxy. The batch cost now lands on the real batch-worker hash d4cdba72... (11 requests, spend up by the batch cost from 0.0000038 to 0.000008325) and the queue call on the real queue-worker hash 184af3b2.... The historical double-hash rows written before the fix (b5e1826d..., 085604bc...) keep their spend but now resolve to the alias and the owner email, in Key Activity and in both exports

$ curl -s http://127.0.0.1:39869/v1/files -H "Authorization: Bearer $BATCH_WORKER_KEY" -F purpose=batch -F target_model_names=gpt-5.4-nano -F file=@batch.jsonl | jq "{id, purpose, status}"
{
  "id": "bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07dW5pZmllZF9pZCxiYWM2ZjVjYS0yMWNhLTRlODAtODMwMi1mYTFhMWMzNTA2MGI7dGFyZ2V0X21vZGVsX25hbWVzLGdwdC01LjQtbmFubztsbG1fb3V0cHV0X2ZpbGVfaWQsZmlsZS1IQzJ6bTdmWlRFWERnR0tBUEtVcUZtO2xsbV9vdXRwdXRfZmlsZV9tb2RlbF9pZCxkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjlj",
  "purpose": "batch",
  "status": "uploaded"
}

$ curl -s http://127.0.0.1:39869/v1/batches -H "Authorization: Bearer $BATCH_WORKER_KEY" -H "Content-Type: application/json" -d '{"input_file_id":"bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07dW5pZmllZF9pZCxiYWM2ZjVjYS0yMWNhLTRlODAtODMwMi1mYTFhMWMzNTA2MGI7dGFyZ2V0X21vZGVsX25hbWVzLGdwdC01LjQtbmFubztsbG1fb3V0cHV0X2ZpbGVfaWQsZmlsZS1IQzJ6bTdmWlRFWERnR0tBUEtVcUZtO2xsbV9vdXRwdXRfZmlsZV9tb2RlbF9pZCxkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjlj","endpoint":"/v1/chat/completions","completion_window":"24h"}' | jq "{id, status}"
{
  "id": "bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjljO2xsbV9iYXRjaF9pZDpiYXRjaF82YTlhZWY2ZmJkMDQ4MTkwODY3OGNiMzEwNjUyNWM0YQ",
  "status": "validating"
}

$ curl -s http://127.0.0.1:39869/queue/chat/completions -H "Authorization: Bearer $QUEUE_WORKER_KEY" -H "Content-Type: application/json" -d '{"model":"gpt-5.4-nano","messages":[{"role":"user","content":"Reply with the single word: queue-after3"}],"priority":0}' | jq "{id, model, content: .choices[0].message.content}"
{
  "id": "chatcmpl-EKQuwnZ2DZ0qLjnXFs0KBYFgux5uI",
  "model": "gpt-5.4-nano-2026-03-17",
  "content": "queue-after3"
}

$ until [ "$(curl -s http://127.0.0.1:39869/v1/batches/bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjljO2xsbV9iYXRjaF9pZDpiYXRjaF82YTlhZWY2ZmJkMDQ4MTkwODY3OGNiMzEwNjUyNWM0YQ -H "Authorization: Bearer $BATCH_WORKER_KEY" | jq -r .status)" = completed ]; do sleep 30; done; curl -s ... | jq "{id, status, request_counts}"
{
  "id": "bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpkZjZhOGE3NmE1YTdmM2I2ZWI1YWMxZDM5M2I4MmExMjY2YzUyYjc4ZWUyZGE5NjkyYjhhZmZlNmQxZjZiYjljO2xsbV9iYXRjaF9pZDpiYXRjaF82YTlhZWY2ZmJkMDQ4MTkwODY3OGNiMzEwNjUyNWM0YQ",
  "status": "completed",
  "request_counts": {
    "completed": 1,
    "failed": 0,
    "total": 1
  }
}
batch-terminal 16:24:01Z

sleep 120  # give the batch cost poller (proxy_batch_polling_interval: 15) time to bill the completed batch

$ curl -s "http://127.0.0.1:39869/user/daily/activity?start_date=2026-09-03&end_date=2026-09-05" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq "[.results[].breakdown.api_keys | to_entries[] | {api_key: .key, spend: .value.metrics.spend, requests: .value.metrics.api_requests, metadata: .value.metadata}]"
[
  {
    "api_key": "085604bcc373b586e34f0e3e7f8cc49adfccfa3e606500e7c66efad883601671",
    "spend": 0.00000905,
    "requests": 1,
    "metadata": {
      "key_alias": "queue-worker",
      "team_id": null,
      "user_email": "queue-admin-lit6852@example.com"
    }
  },
  {
    "api_key": "184af3b2937677e72f25cdb75a64f41f2c0e8228069cda1569a247271c247b4e",
    "spend": 0.00003005,
    "requests": 3,
    "metadata": {
      "key_alias": "queue-worker",
      "team_id": null,
      "user_email": "queue-admin-lit6852@example.com"
    }
  },
  {
    "api_key": "d4cdba721cfc67d3d38ebe975cf6820e4b65a2c3b5f6baf4b338607f7bc65815",
    "spend": 0.000008324999999999999,
    "requests": 11,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null,
      "user_email": "alice-lit6852@example.com"
    }
  },
  {
    "api_key": "b5e1826d5155b8b40e54347031531af039f2fe526aba27751650d45bbdb58b5a",
    "spend": 0.000008324999999999999,
    "requests": 2,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null,
      "user_email": "alice-lit6852@example.com"
    }
  }
]
leg-done

$ curl -s -X POST http://127.0.0.1:39869/cloudzero/dry-run -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"limit": 50}' | jq '{status: .status, rows: (.dry_run_data.usage_data | length), keys: [.dry_run_data.usage_data[] | {api_key: .api_key[0:12], api_key_alias, user_email, spend}] | unique}'
{
  "status": "success",
  "rows": 7,
  "keys": [
    {
      "api_key": "085604bcc373",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "184af3b29376",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00003005
    },
    {
      "api_key": "b5e1826d5155",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    }
  ]
}

$ curl -s -X POST http://127.0.0.1:39869/vantage/dry-run -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"limit": 50}' | jq '{status: .status, rows: (.dry_run_data.usage_data | length), keys: [.dry_run_data.usage_data[] | {api_key: .api_key[0:12], api_key_alias, user_email, spend}] | unique}'
{
  "status": "success",
  "rows": 7,
  "keys": [
    {
      "api_key": "085604bcc373",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "184af3b29376",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00003005
    },
    {
      "api_key": "b5e1826d5155",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    }
  ]
}

Deleted-key recovery path: delete the batch-worker key and read the same surfaces again. Key Activity still labels the double-hash row through the deleted-key lookup

$ curl -s -X POST http://127.0.0.1:39869/key/delete -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"key_aliases":["batch-worker"]}'
{"deleted_keys":["batch-worker"]}

$ curl -s "http://127.0.0.1:39869/user/daily/activity?start_date=2026-09-03&end_date=2026-09-05" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq "[.results[].breakdown.api_keys | to_entries[] | {api_key: .key, spend: .value.metrics.spend, requests: .value.metrics.api_requests, metadata: .value.metadata}]"
[
  {
    "api_key": "085604bcc373b586e34f0e3e7f8cc49adfccfa3e606500e7c66efad883601671",
    "spend": 0.00000905,
    "requests": 1,
    "metadata": {
      "key_alias": "queue-worker",
      "team_id": null,
      "user_email": "queue-admin-lit6852@example.com"
    }
  },
  {
    "api_key": "184af3b2937677e72f25cdb75a64f41f2c0e8228069cda1569a247271c247b4e",
    "spend": 0.00003005,
    "requests": 3,
    "metadata": {
      "key_alias": "queue-worker",
      "team_id": null,
      "user_email": "queue-admin-lit6852@example.com"
    }
  },
  {
    "api_key": "d4cdba721cfc67d3d38ebe975cf6820e4b65a2c3b5f6baf4b338607f7bc65815",
    "spend": 0.000008324999999999999,
    "requests": 11,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null,
      "user_email": "alice-lit6852@example.com"
    }
  },
  {
    "api_key": "b5e1826d5155b8b40e54347031531af039f2fe526aba27751650d45bbdb58b5a",
    "spend": 0.000008324999999999999,
    "requests": 2,
    "metadata": {
      "key_alias": "batch-worker",
      "team_id": null,
      "user_email": "alice-lit6852@example.com"
    }
  }
]

$ curl -s -X POST http://127.0.0.1:39869/cloudzero/dry-run -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"limit": 50}' | jq '{status: .status, rows: (.dry_run_data.usage_data | length), keys: [.dry_run_data.usage_data[] | {api_key: .api_key[0:12], api_key_alias, user_email, spend}] | unique}'
{
  "status": "success",
  "rows": 7,
  "keys": [
    {
      "api_key": "085604bcc373",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "184af3b29376",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00003005
    },
    {
      "api_key": "b5e1826d5155",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": null,
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": null,
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    }
  ]
}

$ curl -s -X POST http://127.0.0.1:39869/vantage/dry-run -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"limit": 50}' | jq '{status: .status, rows: (.dry_run_data.usage_data | length), keys: [.dry_run_data.usage_data[] | {api_key: .api_key[0:12], api_key_alias, user_email, spend}] | unique}'
{
  "status": "success",
  "rows": 7,
  "keys": [
    {
      "api_key": "085604bcc373",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00000905
    },
    {
      "api_key": "184af3b29376",
      "api_key_alias": "queue-worker",
      "user_email": "queue-admin-lit6852@example.com",
      "spend": 0.00003005
    },
    {
      "api_key": "b5e1826d5155",
      "api_key_alias": "batch-worker",
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": null,
      "user_email": "alice-lit6852@example.com",
      "spend": 0.0
    },
    {
      "api_key": "d4cdba721cfc",
      "api_key_alias": null,
      "user_email": "alice-lit6852@example.com",
      "spend": 0.000008324999999999999
    }
  ]
}

Observations from the run, each with what this PR does about it

• Pre-fix pods sharing the DB still bill the double hash. This PR leaves it alone; recovery on read labels those rows, so a mixed-version fleet mid-rollout keeps correct attribution
• One key shows as two Key Activity rows for the buggy window. This PR leaves it alone; both rows carry the alias and email, and grouping by alias sums correctly
• Deleted key: Key Activity keeps aliases, exports drop the real-hash alias. Pre-existing, this PR leaves it alone; exports joined active keys only before this PR too
• Exports already carried user_email before; only the alias was null. Not caused by this PR

Admin UI walkthrough against the tip proxy (npm run dev in ui/litellm-dashboard, proxy on 39869)

1. Open http://localhost:3000/ui/?page=usage, sign in with the master key, and pick the Key Activity view for 2026-09-03 to 2026-09-05
2. Every row for the batch and queue spend reads batch-worker or queue-worker with the owner email under it, and no row reads key-hash-...
3. Use the export button on that view: the CSV carries the alias and email columns filled for the same rows

Type

🐛 Bug Fix

Caveats (if any)

Low

• Old dirty rows keep the double hash in storage, so the buggy window shows a second Key Activity row per key with the same alias and email. A rewrite is a data migration on the spend tables, which must not run at boot, and a read-side merge would touch the aggregation every Usage call runs, so it stays as is
• Keys with no owning user have no email to recover and keep the alias as their label. Keys created with no key_alias still render as key-hash-<token> in Usage. Both by design
• A permanent join miss (master key spend, purged keys) makes Postgres hash every token row once per Usage lookup, bounded by the token table size. An expression index needs a migration and a cache adds per-process state, neither worth it for this regression fix
• CloudZero and Focus exports fill aliases from active keys only, as before this PR, so a deleted key's correctly hashed rows export with a null alias while its double-hashed rows recover through the deleted-key lookup. Pre-existing and unchanged
• Daily activity key metadata now carries the owner user_email for every scope that already returns the key (user, team, tag, org). The same audiences already see member emails in team and user views

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• b93cf2e passes /live-pr-risk

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix-batch-spend-key-double-hash-bcae (b93cf2e) with litellm_internal_staging (16db51e)}

[codecov]

Codecov Report

❌ Patch coverage is 97.19626% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ellm/proxy/spend_tracking/key_metadata_recovery.py	96.38%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cursor]

@greptileai please rescore this PR now that required CI is green on tip 4db370e

[greptile-apps]

Greptile Summary

The PR preserves virtual-key provenance on batch, queued, and callback-replayed spend writes, then recovers metadata for historical double-hashed rows

• Adds shared digest-based metadata recovery for Usage, CloudZero, and Focus
• Propagates recovered user email through activity responses and dashboard labels
• Adds regression coverage for write provenance and historical attribution

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/proxy/spend_tracking/key_metadata_recovery.py	Adds shared active/deleted-key digest recovery and owner-email enrichment
litellm/proxy/management_endpoints/common_daily_activity.py	Integrates recovered key metadata and user email into daily activity breakdowns
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Stamps the existing virtual-key hash as provenance for batch cost writes
litellm/integrations/cloudzero/database.py	Enriches CloudZero usage rows with recovered key metadata before frame construction
litellm/integrations/focus/database.py	Enriches Focus usage rows with recovered key metadata before frame construction
ui/litellm-dashboard/src/components/UsagePage/keyActivityLabel.ts	Centralizes key activity labels using alias, email, and hashed-key fallback data

_{Reviews (4): Last reviewed commit: "Merge branch 'litellm_fix-batch-spend-ke..." | Re-trigger Greptile}

[veria-ai]

Low: Uncached token scans enable denial of service

An authenticated caller with a legacy double-hashed spend row can repeatedly query their daily activity and make each request fetch and hash up to 10,000 active records, followed by another 10,000 deleted records. Avoid performing this reverse scan on the request path; backfill the affected rows or use a bounded, persistent digest lookup with positive and negative caching.

[cursor]

We recover on read so we do not rewrite spend rows. The scan stops when the digest matches and only runs on join misses.

[veria-ai]

PR overview

This pull request updates spend-tracking key metadata recovery so batch spend keys remain joinable after the v1.99 provenance gate, including for legacy double-hashed spend rows.

One denial-of-service concern remains: an authenticated caller with affected legacy data can repeatedly trigger large uncached scans and hashing work during daily activity queries. Exploitation requires a legacy double-hashed row, but repeated requests could impose substantial database and CPU load.

Open issues (1)

• Low: Uncached token scans enable denial of service — litellm/proxy/spend_tracking/key_metadata_recovery.py:116

Fixed/addressed: 0 · PR risk: 4/10

[cursor]

@greptileai please rescore after the pagination fix. Recovery now reads every token page until the digest matches.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Bugbot: export recovery was overwriting spend-user email. Alias still fills; existing email now stays.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Recovered export rows break DataFrame build
  • _row_with_recovered_fields now returns a plain dict so polars.DataFrame consumes it as a dict record like the untouched to_dicts() rows.
• ✅ Fixed: Export recovery scans on missing email
  • The fill_missing_api_key_aliases trigger now only fires on missing alias, so team/service-key rows with no user_email no longer force a full-table SHA-256 scan.

Or push these changes by commenting:

@cursor push 527a8a930f

Preview (527a8a930f)

diff --git a/litellm/proxy/spend_tracking/key_metadata_recovery.py b/litellm/proxy/spend_tracking/key_metadata_recovery.py
--- a/litellm/proxy/spend_tracking/key_metadata_recovery.py
+++ b/litellm/proxy/spend_tracking/key_metadata_recovery.py
@@ -181,14 +181,12 @@
     if not isinstance(api_key, str) or api_key not in recovered:
         return row
     meta: Final = recovered[api_key]
-    return MappingProxyType(
-        {
-            **row,
-            alias_field: meta.get("key_alias") or row.get(alias_field),
-            team_id_field: meta.get("team_id") or row.get(team_id_field),
-            user_email_field: row.get(user_email_field) or meta.get("user_email"),
-        }
-    )
+    return {  # mutable-ok: polars.DataFrame row construction dispatches on dict, not Mapping
+        **row,
+        alias_field: meta.get("key_alias") or row.get(alias_field),
+        team_id_field: meta.get("team_id") or row.get(team_id_field),
+        user_email_field: row.get(user_email_field) or meta.get("user_email"),
+    }
 
 
 async def fill_missing_api_key_aliases(
@@ -211,9 +209,7 @@
         key
         for row in rows
         for key in (row.get(api_key_field),)
-        if isinstance(key, str)
-        and key
-        and (row.get(alias_field) in (None, "") or row.get(user_email_field) in (None, ""))
+        if isinstance(key, str) and key and row.get(alias_field) in (None, "")
     )
     if not missing_keys:
         return tuple(rows)

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Bugbot on tip b93cf2e found no new bugs in spend recovery or export attribution.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b93cf2e. Configure here.}

[tin-berri]

LGTM