Repository navigation
fix(otari): preserve cache_control via native /messages pass-through - #1121
Conversation
OtariProvider inherited the base _amessages(), which converts Anthropic Messages to Chat Completions and silently drops Anthropic-only features (cache_control on system blocks, thinking config). otari's gateway serves /messages natively, and otari SDK 0.1.0 now exposes AsyncOtariClient.message(). Override _amessages() to delegate to otari_client.message(), mirroring how _acompletion() delegates to otari_client.completion(). Non-streaming results validate into MessageResponse; streaming raw event dicts are mapped to typed MessageStreamEvent models (unknown types like ping are skipped). The deprecated GatewayProvider subclass inherits the fix via the shared client. This supersedes the raw-httpx approach in #1111 by keeping transport in the SDK. Fixes #1110 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GitHub Actions
Inactive
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughOtariProvider now overrides _amessages() to send Anthropic-format requests directly to otari's /messages endpoint. Non-streaming calls use otari_client.message and validate into MessageResponse; streaming calls consume otari SSE, convert raw event dicts to typed MessageStreamEvent instances, and yield only known event types. ChangesOtari native messages endpoint
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 37 files with indirect coverage changes 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/any_llm/providers/otari/otari.py`:
- Around line 286-292: In _stream_messages_async, make the stream resilient by
tracking whether a MessageStreamEvent with type "message_stop" (or equivalent
finish event) was yielded: set a local flag (e.g., message_stop_seen) before the
async for loop, update it when converted.type indicates stop, and after the loop
if the flag is false emit/yield a synthetic message_stop MessageStreamEvent
constructed via the same helper used for conversion
(_message_stream_event_from_dict or a minimal MessageStreamEvent instance) so
callers always receive a termination event even if the otari client or network
truncates the stream.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: d0446f26-68f8-4840-baf4-c3d7d12e03e3
📒 Files selected for processing (2)
src/any_llm/providers/otari/otari.pytests/unit/providers/test_otari_provider.py
| async def _stream_messages_async(self, **api_kwargs: Any) -> AsyncIterator[MessageStreamEvent]: | ||
| """Stream otari's /messages endpoint, yielding typed any-llm event models.""" | ||
| stream = await self.otari_client.message(stream=True, **api_kwargs) | ||
| async for event in stream: | ||
| converted = _message_stream_event_from_dict(_as_plain_dict(event)) | ||
| if converted is not None: | ||
| yield converted |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Consider defensive stream termination for resilience.
The implementation trusts the otari SDK to always send a message_stop event. The base provider implementation (in any_llm.py) includes fallback logic to ensure message_stop is emitted even when the upstream provider doesn't send a final chunk with finish_reason.
Whilst this follows the existing pattern in _acompletion (lines 245–263) and tests confirm otari sends the termination event, adding a fallback (e.g., tracking whether message_stop was seen and emitting it if missing) would improve resilience against future changes in otari SDK behaviour or network truncation.
🧰 Tools
🪛 Ruff (0.15.15)
[warning] 286-286: Dynamically typed expressions (typing.Any) are disallowed in **api_kwargs
(ANN401)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/any_llm/providers/otari/otari.py` around lines 286 - 292, In
_stream_messages_async, make the stream resilient by tracking whether a
MessageStreamEvent with type "message_stop" (or equivalent finish event) was
yielded: set a local flag (e.g., message_stop_seen) before the async for loop,
update it when converted.type indicates stop, and after the loop if the flag is
false emit/yield a synthetic message_stop MessageStreamEvent constructed via the
same helper used for conversion (_message_stream_event_from_dict or a minimal
MessageStreamEvent instance) so callers always receive a termination event even
if the otari client or network truncates the stream.
There was a problem hiding this comment.
Pull request overview
This PR fixes Otari’s amessages() behavior by bypassing the default Messages→Chat Completions conversion and delegating directly to otari’s native /messages support via the otari SDK, preserving Anthropic-only fields like cache_control and thinking.
Changes:
- Override
OtariProvider._amessages()to callotari_client.message()for native Messages API support (streaming + non-streaming). - Add SSE event mapping from raw otari
/messagesstream events into typedMessageStreamEventmodels (skipping unknown event types). - Add unit tests validating pass-through of Anthropic-specific fields, omission of
Noneoptionals, and typed streaming behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/any_llm/providers/otari/otari.py |
Overrides _amessages() to use otari SDK native /messages, plus stream event dict→model conversion. |
tests/unit/providers/test_otari_provider.py |
Adds tests for native _amessages() delegation, streaming event typing, and parameter filtering. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
GitHub Actions
Inactive
Description
OtariProviderinherited the base_amessages(), which converts Anthropic Messages to Chat Completions and silently drops Anthropic-only features (cache_controlon system blocks,thinkingconfig). For applications relying on Anthropic prompt caching, this meant every request through otari lost caching, raising latency and cost.otari's gateway serves
/messagesnatively, and otari SDK 0.1.0 now exposesAsyncOtariClient.message(). This PR overrides_amessages()to delegate tootari_client.message(), mirroring how_acompletion()delegates tootari_client.completion(). The transport (auth, URL, SSE parsing) stays inside the otari SDK where it belongs.MessageResponse./messagesSSE stream are mapped to typedMessageStreamEventmodels; unknown event types (e.g.ping) are skipped.GatewayProvidersubclass inherits the fix via the shared client.This supersedes the raw-httpx approach in #1111 (closed) by keeping transport in the SDK rather than reimplementing it in any-llm.
PR Type
Relevant issues
Fixes #1110
Checklist
AI Usage Information
AI Model used: Claude Opus 4.8
AI Developer Tool used: Claude Code
Any other info you'd like to share: Authored by Claude via back-and-forth with @njbrake. The reasoning and decisions are his.
I am an AI Agent filling out this form (check box if true)
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests