[integration]: Deepeval logger

docs/my-website/docs/observability/deepeval_integration.md

@@ -0,0 +1,73 @@
+import Image from '@theme/IdealImage';
+
+# 🔭 DeepEval - Open-Source Evals with Tracing
+
+### What is DeepEval?
+[DeepEval](https://deepeval.com) is an open-source evaluation framework for LLMs ([Github](https://github.com/confident-ai/deepeval)). 
+
+### What is Confident AI?
+
+[Confident AI](https://documentation.confident-ai.com) (the ***deepeval*** platfrom) offers an Observatory for teams to trace and monitor LLM applications. Think Datadog for LLM apps. The observatory allows you to:
+
+- Detect and debug issues in your LLM applications in real-time
+- Search and analyze historical generation data with powerful filters
+- Collect human feedback on model responses
+- Run evaluations to measure and improve performance
+- Track costs and latency to optimize resource usage
+
+<Image img={require('../../img/deepeval_dashboard.png')} />
+
+
+### Pre-Requisites
+
+Install the dependencies
+```shell
+pip install -U deepeval litellm
+```
+
+### Quickstart
+
+```python
+import os
+import time
+import litellm
+
+
+os.environ['OPENAI_API_KEY']='<your-openai-api-key>'
+os.environ['CONFIDENT_API_KEY']='<your-confident-api-key>'
+
+litellm.success_callback = ["deepeval"]
+litellm.failure_callback = ["deepeval"]
+
+try:
+    response = litellm.completion(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "user", "content": "What's the weather like in San Francisco?"}
+        ],
+        metadata={
+            "deepeval_metrics": ["correctness", "coherence"]
+        }
+    )
+except Exception as e:
+    print(e)
+
+print(response)
+
+# wait for queue to send the trace
+time.sleep(5)
+```
+
+:::info
+You can obtain your `CONFIDENT_API_KEY` by logging into [Confident AI](https://app.confident-ai.com/project) platform. 
+:::
+
+
+`deepeval_metrics` A list of strings specifying the names of the online metrics you wish to run upon tracing to Confident AI. Learn more about using online metrics [here](https://documentation.confident-ai.com/llm-observability/online-metrics)
+
+## Support & Talk with Deepeval team
+- [Confident AI Docs 📝](https://documentation.confident-ai.com)
+- [Platform 🚀](https://confident-ai.com)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Support ✉️ support@confident-ai.com
+

docs/my-website/docs/proxy/logging.md

@@ -9,6 +9,7 @@ Log Proxy input, output, and exceptions using:
 - Langfuse
 - OpenTelemetry
 - GCS, s3, Azure (Blob) Buckets
+- Deepeval
 - Lunary
 - MLflow
 - Custom Callbacks
@@ -1182,7 +1183,64 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
 '
 ```
 
+## Deepeval
+LiteLLM supports logging on [Confidential AI](https://documentation.confident-ai.com/) (The Deepeval Platform):
 
+### Usage:
+1. Add `deepeval` in the LiteLLM `config.yaml`
+
+```yaml
+model_list:
+  - model_name: gpt-4o
+    litellm_params:
+      model: gpt-4o
+
+litellm_settings:
+  success_callback: ["deepeval"]
+  failure_callback: ["deepeval"]
+```
+
+2. Set your environment variables in `.env` file. 
+```shell
+CONFIDENT_API_KEY=<your-api-key>
+```
+:::info
+You can obtain your `CONFIDENT_API_KEY` by logging into [Confident AI](https://app.confident-ai.com/project) platform. 
+:::
+
+3. Start your proxy server:
+```shell
+litellm --config config.yaml --debug
+```
+
+4. Make a request:
+```shell
+curl -X POST 'http://0.0.0.0:4000/chat/completions' \
+-H 'Content-Type: application/json' \
+-H 'Authorization: Bearer sk-1234' \
+-d '{
+    "model": "gpt-3.5-turbo",
+    "messages": [
+      {
+        "role": "system",
+        "content": "You are a helpful math tutor. Guide the user through the solution step by step."
+      },
+      {
+        "role": "user",
+        "content": "how can I solve 8x + 7 = -23"
+      }
+    ],
+    "metadata": {
+      "deepeval_metrics": ["correctness", "coherence"]
+    }
+}'
+```
+:::info
+`deepeval_metrics` A list of strings specifying the names of the online metrics you wish to run upon tracing to Confident AI. Learn more about using online metrics [here](https://documentation.confident-ai.com/llm-observability/online-metrics)
+:::
+
+5. Check trace on platform: 
+<Image img={require('../../img/deepeval_visible_trace.png')} />
 
 ## s3 Buckets
 
@@ -1559,8 +1617,6 @@ Run the following command to start MLflow UI and review recorded traces.
 mlflow ui
 ```
 
-
-
 ## Custom Callback Class [Async]
 
 Use this when you want to run custom callbacks in `python`

docs/my-website/sidebars.js

@@ -422,6 +422,7 @@ const sidebars = {
         "observability/agentops_integration",
         "observability/langfuse_integration",
         "observability/lunary_integration",
+        "observability/deepeval_integration",
         "observability/mlflow",
         "observability/gcs_bucket_integration",
         "observability/langsmith_integration",

litellm/__init__.py

@@ -116,6 +116,7 @@
     "agentops",
     "anthropic_cache_control_hook",
     "bedrock_knowledgebase_hook",
+    "deepeval",
 ]
 logged_real_time_event_types: Optional[Union[List[str], Literal["*"]]] = None
 _known_custom_logger_compatible_callbacks: List = list(