Litellm key update fix

.circleci/config.yml

@@ -686,6 +686,7 @@ jobs:
             pip install "pytest-retry==1.6.3"
             pip install "pytest-cov==5.0.0"
             pip install "pytest-asyncio==0.21.1"
+            pip install pytest-mock
             pip install "respx==0.21.1"
             pip install "google-generativeai==0.3.2"
             pip install "google-cloud-aiplatform==1.43.0"

deploy/charts/litellm-helm/templates/migrations-job.yaml

@@ -0,0 +1,30 @@
+# This job runs the prisma migrations for the LiteLLM DB.
+
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: {{ include "litellm.fullname" . }}-migrations
+  annotations:
+    argocd.argoproj.io/hook: PreSync
+    argocd.argoproj.io/hook-delete-policy: Never # keep this resource so we can debug status on ArgoCD
+    checksum/config: {{ toYaml .Values | sha256sum }}
+spec:
+  template:
+    spec:
+      containers:
+        - name: prisma-migrations
+          image: "ghcr.io/berriai/litellm:main-stable"
+          command: ["python", "litellm/proxy/prisma_migration.py"]
+          workingDir: "/app"
+          env:
+            {{- if .Values.db.deployStandalone }}
+            - name: DATABASE_URL
+              value: postgresql://{{ .Values.postgresql.auth.username }}:{{ .Values.postgresql.auth.password }}@{{ .Release.Name }}-postgresql/{{ .Values.postgresql.auth.database }}
+            {{- else if .Values.db.useExisting }}
+            - name: DATABASE_URL
+              value: {{ .Values.db.url | quote }}
+            {{- end }}
+            - name: DISABLE_SCHEMA_UPDATE
+              value: "{{ .Values.migrationJob.disableSchemaUpdate }}"
+      restartPolicy: OnFailure
+  backoffLimit: {{ .Values.migrationJob.backoffLimit }}

deploy/charts/litellm-helm/values.yaml

@@ -179,3 +179,12 @@ postgresql:
 redis:
   enabled: false
   architecture: standalone
+
+# Prisma migration job settings
+migrationJob:
+  enabled: true # Enable or disable the schema migration Job
+  retries: 3 # Number of retries for the Job in case of failure
+  backoffLimit: 4 # Backoff limit for Job restarts
+  disableSchemaUpdate: false # Skip schema migrations for specific environments. When True, the job will exit with code 0.
+
+

docs/my-website/.gitignore

@@ -18,3 +18,4 @@
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
+yarn.lock

docs/my-website/docs/benchmarks.md

@@ -0,0 +1,41 @@
+# Benchmarks
+
+Benchmarks for LiteLLM Gateway (Proxy Server)
+
+Locust Settings:
+- 2500 Users
+- 100 user Ramp Up
+
+
+## Basic Benchmarks
+
+Overhead when using a Deployed Proxy vs Direct to LLM
+- Latency overhead added by LiteLLM Proxy: 107ms
+
+| Metric | Direct to Fake Endpoint | Basic Litellm Proxy |
+|--------|------------------------|---------------------|
+| RPS | 1196 | 1133.2 |
+| Median Latency (ms) | 33 | 140 |
+
+
+## Logging Callbacks
+
+### [GCS Bucket Logging](https://docs.litellm.ai/docs/proxy/bucket)
+
+Using GCS Bucket has **no impact on latency, RPS compared to Basic Litellm Proxy**
+
+| Metric | Basic Litellm Proxy | LiteLLM Proxy with GCS Bucket Logging |
+|--------|------------------------|---------------------|
+| RPS | 1133.2 | 1137.3 |
+| Median Latency (ms) | 140 | 138 |
+
+
+### [LangSmith logging](https://docs.litellm.ai/docs/proxy/logging)
+
+Using LangSmith has **no impact on latency, RPS compared to Basic Litellm Proxy**
+
+| Metric | Basic Litellm Proxy | LiteLLM Proxy with LangSmith |
+|--------|------------------------|---------------------|
+| RPS | 1133.2 | 1135 |
+| Median Latency (ms) | 140 | 132 |
+

docs/my-website/docs/observability/opentelemetry_integration.md

@@ -49,9 +49,19 @@ OTEL_ENDPOINT="http://0.0.0.0:4317"
 
 </TabItem>
 
+<TabItem value="laminar" label="Log to Laminar">
+
+```shell
+OTEL_EXPORTER="otlp_grpc"
+OTEL_ENDPOINT="https://api.lmnr.ai:8443"
+OTEL_HEADERS="authorization=Bearer <project-api-key>"
+```
+
+</TabItem>
+
 </Tabs>
 
-Use just 2 lines of code, to instantly log your LLM responses **across all providers** with OpenTelemetry:
+Use just 1 line of code, to instantly log your LLM responses **across all providers** with OpenTelemetry:
 
 ```python
 litellm.callbacks = ["otel"]

docs/my-website/docs/providers/anthropic.md

@@ -864,3 +864,96 @@ Human: How do I boil water?
 Assistant:
 ```
 
+
+## Usage - PDF 
+
+Pass base64 encoded PDF files to Anthropic models using the `image_url` field.
+
+<Tabs>
+<TabItem value="sdk" label="SDK">
+
+### **using base64**
+```python
+from litellm import completion, supports_pdf_input
+import base64
+import requests
+
+# URL of the file
+url = "https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/2403.05530.pdf"
+
+# Download the file
+response = requests.get(url)
+file_data = response.content
+
+encoded_file = base64.b64encode(file_data).decode("utf-8")
+
+## check if model supports pdf input - (2024/11/11) only claude-3-5-haiku-20241022 supports it
+supports_pdf_input("anthropic/claude-3-5-haiku-20241022") # True
+
+response = completion(
+    model="anthropic/claude-3-5-haiku-20241022",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "You are a very professional document summarization specialist. Please summarize the given document."},
+                {
+                    "type": "image_url",
+                    "image_url": f"data:application/pdf;base64,{encoded_file}", # 👈 PDF
+                },
+            ],
+        }
+    ],
+    max_tokens=300,
+)
+
+print(response.choices[0])
+```
+</TabItem>
+<TabItem value="proxy" lable="PROXY">
+
+1. Add model to config 
+
+```yaml
+- model_name: claude-3-5-haiku-20241022
+  litellm_params:
+    model: anthropic/claude-3-5-haiku-20241022
+    api_key: os.environ/ANTHROPIC_API_KEY
+```
+
+2. Start Proxy
+
+```
+litellm --config /path/to/config.yaml
+```
+
+3. Test it! 
+
+```bash
+curl http://0.0.0.0:4000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
+  -d '{
+    "model": "claude-3-5-haiku-20241022",
+    "messages": [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "You are a very professional document summarization specialist. Please summarize the given document"
+          },
+          {
+                "type": "image_url",
+                "image_url": "data:application/pdf;base64,{encoded_file}" # 👈 PDF
+            }
+          }
+        ]
+      }
+    ],
+    "max_tokens": 300
+  }'
+
+```
+</TabItem>
+</Tabs>

docs/my-website/docs/proxy/bucket.md

@@ -9,7 +9,7 @@ LiteLLM Supports Logging to the following Cloud Buckets
 - (Enterprise) ✨ [Google Cloud Storage Buckets](#logging-proxy-inputoutput-to-google-cloud-storage-buckets)
 - (Free OSS) [Amazon s3 Buckets](#logging-proxy-inputoutput---s3-buckets) 
 
-## Logging Proxy Input/Output to Google Cloud Storage Buckets
+## Google Cloud Storage Buckets
 
 Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?hl=en)
 
@@ -20,6 +20,14 @@ Log LLM Logs to [Google Cloud Storage Buckets](https://cloud.google.com/storage?
 :::
 
 
+| Property | Details |
+|----------|---------|
+| Description | Log LLM Input/Output to cloud storage buckets |
+| Load Test Benchmarks | [Benchmarks](https://docs.litellm.ai/docs/benchmarks) |
+| Google Docs on Cloud Storage | [Google Cloud Storage](https://cloud.google.com/storage?hl=en) |
+
+
+
 ### Usage
 
 1. Add `gcs_bucket` to LiteLLM Config.yaml
@@ -85,7 +93,7 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
 6. Save the JSON file and add the path to `GCS_PATH_SERVICE_ACCOUNT`
 
 
-## Logging Proxy Input/Output - s3 Buckets
+## s3 Buckets
 
 We will use the `--config` to set 
 

docs/my-website/docs/proxy/caching.md

@@ -136,6 +136,7 @@ litellm_settings:
     type: "redis"
     service_name: "mymaster"
     sentinel_nodes: [["localhost", 26379]]
+    sentinel_password: "password" # [OPTIONAL]
 ```
 
 </TabItem>
@@ -149,6 +150,7 @@ You can configure redis sentinel in your .env by setting `REDIS_SENTINEL_NODES`
 ```env
 REDIS_SENTINEL_NODES='[["localhost", 26379]]'
 REDIS_SERVICE_NAME = "mymaster"
+REDIS_SENTINEL_PASSWORD = "password"
 ```
 
 :::note

docs/my-website/docs/proxy/configs.md

@@ -934,8 +934,8 @@ router_settings:
 | EMAIL_SUPPORT_CONTACT | Support contact email address
 | GCS_BUCKET_NAME | Name of the Google Cloud Storage bucket
 | GCS_PATH_SERVICE_ACCOUNT | Path to the Google Cloud service account JSON file
-| GCS_FLUSH_INTERVAL | Flush interval for GCS logging (in seconds). Specify how often you want a log to be sent to GCS.
-| GCS_BATCH_SIZE | Batch size for GCS logging. Specify after how many logs you want to flush to GCS. If `BATCH_SIZE` is set to 10, logs are flushed every 10 logs.
+| GCS_FLUSH_INTERVAL | Flush interval for GCS logging (in seconds). Specify how often you want a log to be sent to GCS. **Default is 20 seconds**
+| GCS_BATCH_SIZE | Batch size for GCS logging. Specify after how many logs you want to flush to GCS. If `BATCH_SIZE` is set to 10, logs are flushed every 10 logs. **Default is 2048**
 | GENERIC_AUTHORIZATION_ENDPOINT | Authorization endpoint for generic OAuth providers
 | GENERIC_CLIENT_ID | Client ID for generic OAuth providers
 | GENERIC_CLIENT_SECRET | Client secret for generic OAuth providers

docs/my-website/docs/proxy/deploy.md

@@ -688,8 +688,35 @@ Provide an ssl certificate when starting litellm proxy server
 
 Use this if you want to run the proxy with hypercorn to support http/2
 
-**Usage**
-Pass the `--run_hypercorn` flag when starting the proxy
+Step 1. Build your custom docker image with hypercorn
+
+```shell
+# Use the provided base image
+FROM ghcr.io/berriai/litellm:main-latest
+
+# Set the working directory to /app
+WORKDIR /app
+
+# Copy the configuration file into the container at /app
+COPY config.yaml .
+
+# Make sure your docker/entrypoint.sh is executable
+RUN chmod +x ./docker/entrypoint.sh
+
+# Expose the necessary port
+EXPOSE 4000/tcp
+
+# 👉 Key Change: Install hypercorn
+RUN pip install hypercorn
+
+# Override the CMD instruction with your desired command and arguments
+# WARNING: FOR PROD DO NOT USE `--detailed_debug` it slows down response times, instead use the following CMD
+# CMD ["--port", "4000", "--config", "config.yaml"]
+
+CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug"]
+```
+
+Step 2. Pass the `--run_hypercorn` flag when starting the proxy
 
 ```shell
 docker run \
@@ -699,7 +726,7 @@ docker run \
     -e SERVER_ROOT_PATH="/api/v1"\
     -e DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> \
     -e LITELLM_MASTER_KEY="sk-1234"\
-    ghcr.io/berriai/litellm:main-latest \
+    your_custom_docker_image \
     --config /app/config.yaml
     --run_hypercorn
 ```