Add blog post about litellm-observatory#20622
Merged
AlexsanderHamir merged 1 commit intomainfrom Feb 7, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile OverviewGreptile Summary
Confidence Score: 4/5
|
| Filename | Overview |
|---|---|
| docs/my-website/blog/litellm_observatory/index.md | Adds a new blog post describing LiteLLM Observatory; file has a markdown rendering bug due to an unmatched trailing triple-backtick at the end of the document. |
Sequence Diagram
sequenceDiagram
autonumber
participant User as User/CI
participant ObsAPI as Observatory API
participant Queue as Test Queue
participant Runner as Test Runner
participant LiteLLM as LiteLLM Deployment
participant Slack as Slack
User->>ObsAPI: POST /tests {deployment_url, api_key, test_name, settings}
ObsAPI->>Queue: Enqueue test if not duplicate
alt Duplicate running/queued
Queue-->>ObsAPI: Reject duplicate
ObsAPI-->>User: 409/4xx Duplicate test
else Accepted
Queue-->>ObsAPI: Accepted
ObsAPI-->>User: 202 Accepted (test_id)
Queue-->>Runner: Dequeue when capacity available
loop For test duration
Runner->>LiteLLM: Send model requests
LiteLLM-->>Runner: Responses / errors
Runner->>Runner: Track failure rate & thresholds
end
Runner->>Slack: Post results summary
end
Comment on lines
+134
to
+136
| We’ll continue to share those improvements openly as we go. | ||
| ``` | ||
|
|
Contributor
There was a problem hiding this comment.
Unmatched closing code fence
The post ends with a lone closing triple-backtick (`````), but there’s no corresponding opening fence at the end of the article. This will render the remainder of the page (or previous section) as a code block in many markdown renderers. Remove the trailing fence (or add the missing opening fence if intentional).
krrishdholakia
added a commit
that referenced
this pull request
Feb 7, 2026
…ng format for Claude Code (#20631) * Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(mypy): resolve missing return statements and type casting issues (#20618) * fix(mypy): resolve missing return statements and type casting issues * fix(pangea): use elif to prevent UnboundLocalError and handle None messages Address Greptile review feedback: - Make branches mutually exclusive using elif to prevent input_messages from being overwritten - Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API --------- Co-authored-by: Shin <shin@openclaw.ai> * [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * fixing user_id * [Feat] IP-Based Access Control for MCP Servers (#20620) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * test fix * fix linting * inint ui types * add ui for managing MCP private/public * add ui * fixes * add to schema * add types * fix endpoint * add endpoint * update manager * test mcp * dont use external party for ip address * Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622) * docs (#20626) * docs * fix(mypy): resolve type checking errors in 5 files (#20627) - a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment - caching/redis_cache.py: Add type ignore for async ping return type - caching/redis_cluster_cache.py: Add type ignore for async ping return type - llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text - proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument All changes add appropriate type: ignore comments to handle library typing inconsistencies. * fix(test): update deprecated gemini embedding model (#20621) Replace text-embedding-004 with gemini-embedding-001. The old model was deprecated and returns 404: 'models/text-embedding-004 is not found for API version v1beta' Co-authored-by: Shin <shin@openclaw.ai> * ui new buil * fix(websearch_interception): convert agentic loop response to streaming format when original request was streaming Fixes #20187 - When using websearch_interception in Bedrock with Claude Code: 1. Output tokens were showing as 0 because the agentic loop response wasn't being converted back to streaming format 2. The response from the agentic loop (follow-up request) was returned as a non-streaming dict, but Claude Code expects a streaming response This fix adds streaming format conversion for the agentic loop response when the original request was streaming (detected via the websearch_interception_converted_stream flag in logging_obj). The fix ensures: - Output tokens are correctly included in the message_delta event - stop_reason is properly preserved - The response format matches what Claude Code expects --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shin <shin@openclaw.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
krrishdholakia
added a commit
that referenced
this pull request
Feb 7, 2026
…iohttp tracing (#20630) * Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(mypy): resolve missing return statements and type casting issues (#20618) * fix(mypy): resolve missing return statements and type casting issues * fix(pangea): use elif to prevent UnboundLocalError and handle None messages Address Greptile review feedback: - Make branches mutually exclusive using elif to prevent input_messages from being overwritten - Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API --------- Co-authored-by: Shin <shin@openclaw.ai> * [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * fixing user_id * [Feat] IP-Based Access Control for MCP Servers (#20620) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * test fix * fix linting * inint ui types * add ui for managing MCP private/public * add ui * fixes * add to schema * add types * fix endpoint * add endpoint * update manager * test mcp * dont use external party for ip address * Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622) * docs (#20626) * docs * fix(mypy): resolve type checking errors in 5 files (#20627) - a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment - caching/redis_cache.py: Add type ignore for async ping return type - caching/redis_cluster_cache.py: Add type ignore for async ping return type - llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text - proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument All changes add appropriate type: ignore comments to handle library typing inconsistencies. * fix(test): update deprecated gemini embedding model (#20621) Replace text-embedding-004 with gemini-embedding-001. The old model was deprecated and returns 404: 'models/text-embedding-004 is not found for API version v1beta' Co-authored-by: Shin <shin@openclaw.ai> * ui new buil * fix(http_handler): bypass cache when shared_session is provided for aiohttp tracing When users pass a shared_session with trace_configs to acompletion(), the get_async_httpx_client() function was ignoring it and returning a cached client without the user's tracing configuration. This fix bypasses the cache when shared_session is provided, ensuring the user's ClientSession (with its trace_configs, connector settings, etc.) is actually used for the request. Fixes #20174 --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shin <shin@openclaw.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
Sameerlite
pushed a commit
that referenced
this pull request
Feb 12, 2026
…iohttp tracing (#20630) * Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(mypy): resolve missing return statements and type casting issues (#20618) * fix(mypy): resolve missing return statements and type casting issues * fix(pangea): use elif to prevent UnboundLocalError and handle None messages Address Greptile review feedback: - Make branches mutually exclusive using elif to prevent input_messages from being overwritten - Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API --------- Co-authored-by: Shin <shin@openclaw.ai> * [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * fixing user_id * [Feat] IP-Based Access Control for MCP Servers (#20620) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * test fix * fix linting * inint ui types * add ui for managing MCP private/public * add ui * fixes * add to schema * add types * fix endpoint * add endpoint * update manager * test mcp * dont use external party for ip address * Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622) * docs (#20626) * docs * fix(mypy): resolve type checking errors in 5 files (#20627) - a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment - caching/redis_cache.py: Add type ignore for async ping return type - caching/redis_cluster_cache.py: Add type ignore for async ping return type - llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text - proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument All changes add appropriate type: ignore comments to handle library typing inconsistencies. * fix(test): update deprecated gemini embedding model (#20621) Replace text-embedding-004 with gemini-embedding-001. The old model was deprecated and returns 404: 'models/text-embedding-004 is not found for API version v1beta' Co-authored-by: Shin <shin@openclaw.ai> * ui new buil * fix(http_handler): bypass cache when shared_session is provided for aiohttp tracing When users pass a shared_session with trace_configs to acompletion(), the get_async_httpx_client() function was ignoring it and returning a cached client without the user's tracing configuration. This fix bypasses the cache when shared_session is provided, ensuring the user's ClientSession (with its trace_configs, connector settings, etc.) is actually used for the request. Fixes #20174 --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shin <shin@openclaw.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes