Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions deploy/lens/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ RUN apk add --no-cache rust build-base cmake perl pkgconf openssl-dev libseccomp
WORKDIR /src
COPY .cargo/ .cargo/
COPY litellm-rust/ litellm-rust/
COPY model_prices_and_context_window.json model_prices_and_context_window.json
COPY litellm/integrations/otel/model/providers.json litellm/integrations/otel/model/providers.json
COPY litellm/proxy/lens/prompts/ litellm/proxy/lens/prompts/
WORKDIR /src/litellm-rust
ENV CARGO_PROFILE_RELEASE_DEBUG=0 CARGO_PROFILE_RELEASE_STRIP=symbols
Expand Down
6 changes: 6 additions & 0 deletions deploy/lens/Dockerfile.dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,9 @@
!litellm-rust/
!litellm-rust/**
litellm-rust/target/

!model_prices_and_context_window.json
!litellm/integrations/
!litellm/integrations/otel/
!litellm/integrations/otel/model/
!litellm/integrations/otel/model/providers.json
40 changes: 40 additions & 0 deletions litellm-rust/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions litellm-rust/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ license = "MIT"
repository = "https://github.com/BerriAI/litellm"

[workspace.dependencies]
litellm-cost = { path = "crates/cost" }
litellm-model-catalog = { path = "crates/model-catalog" }
litellm-config = { path = "crates/config" }
litellm-router = { path = "crates/router" }
litellm-tracing = { path = "crates/tracing" }
Expand Down
5 changes: 5 additions & 0 deletions litellm-rust/crates/cost/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ edition.workspace = true
license.workspace = true
repository.workspace = true

[dependencies]
serde_json.workspace = true
chrono = { version = "0.4", default-features = false, features = ["std"] }
chrono-tz = "0.10.4"

[dev-dependencies]
criterion.workspace = true
proptest.workspace = true
Expand Down
10 changes: 7 additions & 3 deletions litellm-rust/crates/cost/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# litellm-cost

This crate calculates text token charges from rates and usage supplied by its caller. It is standalone and has no Python bridge or proxy integration
This crate calculates text token charges from rates and usage supplied by its caller. It has no Python dependency and is used by standalone Lens for display estimates

Call `compile(&pricing)` once for an immutable plan, then `plan.calculate(&request)` for each supported request. `calculate(&pricing, &request)` compiles on each call. A successful result exposes pre-multiplier component costs, selected rates, the multiplier, and derived `input()`, `output()`, and `total()` values

The caller states whether `prompt_tokens` includes cache tokens. Threshold selection uses total input tokens for either convention and selects one rate for the whole request. Thresholds are sorted when compiled, and duplicate thresholds or tier overrides fail deterministically. `Fast` selects priority rates; unknown tiers use standard rates

`Rate::Missing`, `Rate::Null`, and `Rate::Value(0.0)` remain distinct. Missing cache rates fall back to the selected input rate, and an absent one-hour write rate falls back to the selected write rate. Missing input or output rates return typed errors, including for zero usage. Python's sparse-entry behavior remains outside this native contract
`Rate::Missing`, `Rate::Null`, and `Rate::Value(0.0)` remain distinct. Missing cache rates fall back to the selected input rate, and an absent one-hour write rate falls back to the selected write rate. Missing input or output rates return typed errors, including for zero usage. These supplied-rate API semantics also remain unchanged when Lens uses catalog estimation

The supported off-peak shape is one non-wrapping UTC daily window. The caller supplies the applicable regional multiplier after provider-specific selection. Negative or non-finite rates, ambiguous rules, inconsistent cache counts, incomplete write splits, invalid windows and overflow return errors. Callers must decline unsupported inputs before native execution if their public contract accepts those shapes

This crate does not select models, read catalogs, fetch provider prices, normalize multimodal usage, process provider-reported costs, or calculate non-token charges. It does not change proxy behavior. The reference fixture was generated by `tests/generate_python_reference.py` against the Python implementation at the commit recorded in `tests/python_reference.tsv`, using synthetic rates and fixed usage
`catalog::estimate` selects effective rates from a supplied raw catalog row. It supports structured tiers, service tiers, provider thresholds and cache defaults, inclusive reasoning output, and captured-time off-peak schedules with weekday timezones. It validates selected input/output rates even at zero usage, and cache/reasoning rates only when they affect the amount. Missing context or invalid applicable prices return `None`; explicit zero remains a known price. Both APIs use the same component arithmetic

This crate does not select models, fetch provider prices, normalize multimodal usage, process provider-reported costs, or calculate non-token charges. `litellm-model-catalog` owns offline identity and alias resolution, and `litellm-traces` owns GenAI usage admission. Lens reads `supports_token_only_pricing: false` from the resolved catalog row to decline models whose costs cannot be represented by ordinary token charges. Set this flag to false for new dedicated search, agentic or router models, including separately hosted catalog entries; model names never determine eligibility. The same flag is exposed through Python `get_model_info`. Missing, null or true retains ordinary token eligibility; malformed values remain unknown. Known requested-model requirements are preserved when the served model differs. Uncovered request fees and reported search, modality, citation, computer-use and code-execution usage also remain unknown. Optional search capability alone does not disable ordinary token estimates

Lens bundles the repository catalog into its release, so its labeled estimates use deployed catalog prices. Matched gateway spend remains authoritative. The reference fixture was generated by `tests/generate_python_reference.py` against the Python implementation at the commit recorded in `tests/python_reference.tsv`, using synthetic rates and fixed usage
Loading
Loading