refactor(sdk): separate core AWS and tokenizer dependencies

[joshua-berri]

TLDR

Problem this solves:

• Core installs unnecessarily require AWS and Hugging Face packages
• Eager imports prevent using core without those packages

How it solves it:

• Remove three dependencies from core's standalone manifest
• Load optional packages at their consuming operations
• Preserve native/Python tokenizers and existing AWS signing

User Flow

Before: core installs AWS and Hugging Face dependencies for every application

1. Create a fresh environment and install the source-built litellm-core wheel
2. Inspect installed packages: boto3, tokenizers, and huggingface-hub are present
3. Run chat completions and token counting

After: core works without those packages, with explicit installation for Python features that require them

1. Create a fresh environment and install the source-built litellm-core wheel
2. Inspect installed packages: boto3, tokenizers, and huggingface-hub are absent
3. Run chat completions and token counting; approximate fallback counts warn
4. Install boto3 for AWS signing/event decoding and tokenizers huggingface-hub for Python Hugging Face tokenization

Relevant issues

Second of two existing PRs, depending on #44340. Head litellm_optional_aws_tokenizers targets litellm_lean_sdk_dependencies at fb233676cc22a066918306819b34b8a717745048

Replayed only the child commits onto the updated parent, preserving upstream tests and SDK regression tests through the four append conflicts. The final child contains sixteen commits beyond the recorded parent boundary, including the scoped test-helper and Bedrock request-typing fixes

Release integration remains separate in project-releaser #293

Implementation

Core removes boto3, tokenizers, and huggingface-hub from mandatory dependencies and declares python-dateutil directly. jsonschema remains mandatory. Shared optional-import handling preserves actionable missing-package errors without masking unrelated import failures or retrying installation errors. Bedrock bearer preparation works without botocore; credential resolution, SigV4 signing and event decoding retain their existing implementations. Python tokenizer loading is deferred while native execution, runtime aliases, added-token decoding and sanitized fallback warnings remain supported. Legacy dependencies and lockfiles are unchanged

Squash and merge order

Both PRs remain open: #44340, head litellm_lean_sdk_dependencies, targets main. #44447, head litellm_optional_aws_tokenizers, targets litellm_lean_sdk_dependencies

Squash-merge #44340 first when approved. Record its final head before merging. Then fetch main, replay only the child commits with git rebase --onto origin/main fb233676cc22a066918306819b34b8a717745048 <child-branch>, and retarget #44447 to main. That boundary must be updated if the parent changes again. Inspect the child-only diff and refresh verification afterward

Local squash rehearsal replayed the child commits cleanly and preserved the final file tree. Backup refs retain both pre-restack tips. No PR has been merged

Current verification

Current tip: 034f2f0b7ff685ff33d228e150bb388c00096f75. Parent targets main at 591637d47571f82c8181ec020c07d68c890ed226; child targets parent fb233676cc22a066918306819b34b8a717745048. All ten parent patches and sixteen child patches remain identical. Both refreshed trees exactly match their previous tips combined with the three new main commits

The refresh contains Akto logging-only support, OpenRouter pricing data and an encrypted-reasoning Router fallback fix. No Rust, packaging, dependency, AWS-auth or tokenizer implementation changed. Focused router/fallback checks passed 89 tests; the local proxy cases could not collect because the reused environment lacks APScheduler and remain covered by the hosted proxy suite

Whole Ruff, test-tree lint, formatting, strict-rule, type-discipline, test-quality, basedpyright, import and lock gates pass for both refreshed tips. CI and bot reviews are refreshing after the push

The completed independent source-build report covers parent e8bd3e51ab12075f2a13c927eded80fd8f67f527 and child 614e1f175eef4874cf1aca7063270ea94bdefdd8 against main 1a42c5abde9a44c225e6526b589df67a782b826b. All ten source artifacts built successfully, parent and child standalone sdist rebuilds matched their wheels, installed core/legacy and co-install guard checks passed, and real proxy-to-OpenAI normal/streaming calls matched baseline with HTTP 200, pong and usage 13/4/17

That report found no regressions. Its 49 native trace failures/errors were identical on baseline and child and were blocked by Docker Hub rate limiting for the ClickHouse test image. The existing hosted Rust import failure remains separately classified as unrelated

The agreed refresh scope preserves that evidence for the unchanged SDK patches and uses targeted router checks for the new interaction. No full Devin rerun or fresh artifact-build claim is made for the latest three-commit refresh. Historical reports below retain their exact tested SHAs

Pre-Submission checklist

• Conflict resolution preserves both upstream and SDK regressions
• Parent/child boundaries recorded and squash replay rehearsed
• Final-tip local verification and independent aggregate execution recorded below
• Refreshed final-tip hosted Codecov and core-distribution checks
• Greptile confirmed the previous-tip live-QA finding resolved
• Refreshed final-tip independent verification and bot reviews
• Contributor agreement evidence available

Screenshots / Proof of Fix

Independent verification completed on October 8 in the Devin session, using fresh wheels and source archives from these exact commits

Source	Commit
Legacy baseline	5d207d85ee547bc323d704bc6aaab901aa0be8b6
Parent #44340	dd1fd3b42c12b841cdfc5e87f6afdedba8268b67
This PR	029a626c99ec8c6012512b18ecea075a90db9dcd
Releaser baseline / #293	5ed9ef148c612a9ce70b8a5c6649415bb41305ae / 3e2229f5d14f894a4c981fab382241243421ed81

Before (5d207d8 and parent dd1fd3b)

1. Build and install each source wheel in an isolated environment, verify the imported package path, and run real OpenAI chat, async chat, streaming with usage, embeddings, Router sync/async calls and invalid-key controls
2. Run real Anthropic chat/streaming/authentication controls and Bedrock bearer controls using the installed SDK
3. Start the installed legacy proxy and inspect /ui/, /ui/login/, favicon, JavaScript/CSS assets, custom UI, non-root fallback and the configured /litellm root path

After (029a626)

1. Repeat the same real-provider requests using the child legacy wheel, bare core wheel and optional-dependency profiles. OpenAI passed 42/42 controls across six baseline/parent/child environments; Anthropic passed 35/35 across seven environments
2. Bedrock bearer passed 36/36 across six environments with the required dependencies. Bare child core passed four controls; its two streaming controls returned the expected actionable botocore installation error. Live SigV4 was blocked by missing AWS access keys, while synthetic signing and missing-dependency cases were exercised separately
3. Repeat the installed proxy checks against baseline and child. CLI entry points, UI routes, assets, custom UI and root-path behavior matched. Upgrade, reinstall and rollback passed with pip check clean and all 714 installed dashboard files matching their wheel

Real proxy to real provider, October 9

The remaining proxy-path check passed on baseline 5d207d85ee547bc323d704bc6aaab901aa0be8b6 and final child 029a626c99ec8c6012512b18ecea075a90db9dcd. Each proxy used its source-built legacy wheel, with installed files checked against that wheel: 3,690 baseline files and 3,691 child files, zero mismatches

Both proxies used this configuration, resolving credentials from environment variables

model_list:
  - model_name: r9-smoke
    litellm_params:
      model: openai/gpt-6.1-sol
      api_key: os.environ/OPENAI_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Baseline listened on port 4131, child on 4132. The same request was sent through each proxy to the real OpenAI provider

curl -sS http://127.0.0.1:4131/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"r9-smoke","messages":[{"role":"user","content":"Reply with the single word pong."}],"max_completion_tokens":256,"reasoning_effort":"low"}'

Repeat with port 4132 for the child. Streaming used the same body with "stream":true,"stream_options":{"include_usage":true}

Observed result	Baseline	Final child
Normal request	HTTP 200, pong, finish reason stop	HTTP 200, pong, finish reason stop
Token usage	13 prompt, 4 completion, 17 total	13 prompt, 4 completion, 17 total
Streaming request	HTTP 200, 4 chunks plus [DONE], pong, usage 13/4/17	HTTP 200, 4 chunks plus [DONE], pong, usage 13/4/17

Proxy access logs recorded HTTP 200 for both requests on both proxies. The secret scan found no key-pattern matches. Both task-owned proxies were stopped and their ports closed afterward; no source changes or rebuilds were needed

Redacted proxy smoke evidence, executed script, baseline response, child response

Authenticated dashboard coverage remains outside the executed checks

Additional gate	Measured result
Original release harness	39/39 passed, plus 384 routing combinations with zero mismatches
Parent and child source distributions	12/12 tests each, including independent extracted-sdist rebuilds with no deselections
Installed distributions	11 isolated installation/HTTP cases passed
Co-installation	Both installation orders for parent and child rejected imports as designed
AWS/tokenizer probes	Nine environments; matching dependency profiles preserved baseline behavior
Independent source regressions	2,391 passed initially; four setup failures passed after correcting import paths and restoring the CI dependency; six skipped

Evidence: case map, commands and run record, 39-case release output, proxy/UI output, AWS/tokenizer output, live-provider output. Attachments may require organization access

Coverage limits: no aarch64, musllinux, macOS or Windows builds, no manylinux_2_28 Docker leg, and no real publishing, OIDC or dispatch. The platform completeness negative correctly rejected 30 missing cells. Router calls were exercised, but Router-level retry behavior was not

Two behaviors matched baseline and are outside this change: optional MCP/Tenacity requirements for Responses/retries, and root-path UI file rewriting. The UI result above uses the clean copy-mode rerun, replacing an earlier run affected by shared hardlinks

Type

Refactoring, infrastructure and tests

Caveats

Severe

• Core and legacy cannot be co-installed

Medium

• Verification limits are listed with the evidence above
• Contributor-agreement evidence remains unavailable

Final Attestation

• Final-tip verification and review requirements completed

---

Note

Medium Risk
Touches Bedrock/SageMaker auth, token counting fallbacks, and core package dependencies—behavior changes when optional packages are missing, though tests and CI profiles aim to lock that in.

Overview
Makes litellm-core leaner by dropping mandatory boto3, tokenizers, and huggingface-hub from packaging/litellm-core/pyproject.toml (adds python-dateutil explicitly; jsonschema stays required).

Optional dependencies load on demand. New ensure_optional_import centralizes actionable pip install … errors. AWS paths (Bedrock, SageMaker, Polly, secrets manager, S3 Focus upload) import boto3/botocore only when signing or uploading. Bearer-token Bedrock requests use a BearerPreparedRequest path that does not need botocore. Tokenizer code defers tokenizers imports, keeps native/Rust paths, falls back to tiktoken with redacted warnings, and exposes runtime type aliases via __getattr__.

Error handling tightens. ImportError/ModuleNotFoundError for boto3/botocore (and Bedrock providers) pass through exception mapping unchanged; async retries skip ImportError.

Verification expands. Base SDK smoke checks gain --profile modes (core vs dependencies), a loopback HTTP harness (check_sdk_http.py), and CI runs core installs without optional packages before installing AWS/tokenizer deps for fuller checks.

^{Reviewed by Cursor Bugbot for commit 034f2f0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[joshua-berri]

bugbot run

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge, with the existing non-blocking tokenizer type-inspection issue still open

Findings

1.  Tokenizer types cannot be inspected ▶

Summary

Makes AWS and Python Hugging Face packages optional for litellm-core, while leaving legacy dependencies unchanged

• Core installs no longer pull in AWS and Python Hugging Face packages.
• AWS packages load when an AWS operation needs them.
• Token counting keeps working when Python Hugging Face packages are absent.

The previous commit is not an ancestor of the rebased head, and the target merge bases differ. This assessment therefore uses the full PR diff

Prior dismissals remain honored: joshua-berri accepted temporary callback changes to test the public registration path, warning suppression under caller-selected ERROR logging, and shared AWS dependency classification in the central mapper. Callback restoration now uses patch.object. The documented real proxy-to-provider checks address the previous live-QA concern

No tests or runtime checks were executed during this review

_{Reviews (26) · Last reviewed commit: "fix(sdk): preserve bearer request typing..." · Reviewed by Greptile}

[cursor]

Stale Bugbot comment from a previous run.

[joshua-berri]

@greptileai Please review the child-only diff against litellm_lean_sdk_dependencies at e83dac0. Current head is ee53ccc. This PR makes AWS, Hugging Face and validation dependencies optional; the parent owns UI/CLI extraction. Independent source verification passed; CI is running.

[codecov]

Codecov Report

❌ Patch coverage is 94.85714% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tests/base_sdk_tests/check_base_sdk_install.py	76.92%	6 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

[joshua-berri]

@greptileai @veria-ai Please review current tip d5f7570. The runtime tokenizer aliases now support core tiktoken/native callers without Hugging Face; installed and broken-import cases have regressions. Added real Polly/SageMaker signing and missing-extra tests. Local canonical checks and clean wheel profiles passed; focused independent verification is running.

[joshua-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[joshua-berri]

@greptileai @veria-ai Please refresh review at 380ec95. The only change from d5f7570 is an empty Polly test-package init.py to satisfy the package-layout CI test. Runtime and wheel contents are unchanged; mapped tests and canonical lint pass.

[joshua-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[joshua-berri]

@greptileai @veria-ai Please review the rebased tip; scoped local checks pass and independent verification is refreshing. bugbot run

[joshua-berri]

@greptileai @veria-ai Please review the current tip. Callback configuration now uses a scoped standard-library patch; refreshed local checks pass. bugbot run

[joshua-berri]

bugbot run

Please review the current rebased tip against its immediate parent; local lint and installation checks pass.

[cursor]

Stale Bugbot comment from a previous run.

[joshua-berri]

bugbot run

Please review refreshed tip 614e1f175eef4874cf1aca7063270ea94bdefdd8 after the clean stack rebase; SDK patches are unchanged and verification is running

[cursor]

Stale Bugbot comment from a previous run.

[greptile-apps]

Tokenizer types cannot be inspected

Moving Encoding behind TYPE_CHECKING makes get_type_hints(openai_tokenizer_encoding) raise NameError instead of returning its types. The same problem affects tokenizer functions in litellm/rust_bridge/tokenizer.py. Keep the annotation names available at runtime without eagerly loading optional packages

[joshua-berri]

@greptileai @veria-ai Please review 034f2f0b7ff685ff33d228e150bb388c00096f75 after the clean rebase; SDK patches are identical, prior evidence retained, focused router tests pass

[joshua-berri]

bugbot run

Please review 034f2f0b7ff685ff33d228e150bb388c00096f75 after the clean rebase; unchanged SDK patches retain prior verification, with focused router checks passing

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 034f2f0. Configure here.}