From 75474ac7b938ea6d7dfe032e9c128e03a27911c8 Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Sun, 11 Jan 2026 00:05:29 -0700 Subject: [PATCH 1/5] Update CLAUDE.md with qwen3 tool_calls bug fix instructions (#18922) --- CLAUDE.md | 245 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 137 insertions(+), 108 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 23a0e97eaeec..7a80f206015b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,108 +1,137 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Development Commands - -### Installation -- `make install-dev` - Install core development dependencies -- `make install-proxy-dev` - Install proxy development dependencies with full feature set -- `make install-test-deps` - Install all test dependencies - -### Testing -- `make test` - Run all tests -- `make test-unit` - Run unit tests (tests/test_litellm) with 4 parallel workers -- `make test-integration` - Run integration tests (excludes unit tests) -- `pytest tests/` - Direct pytest execution - -### Code Quality -- `make lint` - Run all linting (Ruff, MyPy, Black, circular imports, import safety) -- `make format` - Apply Black code formatting -- `make lint-ruff` - Run Ruff linting only -- `make lint-mypy` - Run MyPy type checking only - -### Single Test Files -- `poetry run pytest tests/path/to/test_file.py -v` - Run specific test file -- `poetry run pytest tests/path/to/test_file.py::test_function -v` - Run specific test - -### Running Scripts -- `poetry run python script.py` - Run Python scripts (use for non-test files) - -### GitHub Issue & PR Templates -When contributing to the project, use the appropriate templates: - -**Bug Reports** (`.github/ISSUE_TEMPLATE/bug_report.yml`): -- Describe what happened vs. what you expected -- Include relevant log output -- Specify your LiteLLM version - -**Feature Requests** (`.github/ISSUE_TEMPLATE/feature_request.yml`): -- Describe the feature clearly -- Explain the motivation and use case - -**Pull Requests** (`.github/pull_request_template.md`): -- Add at least 1 test in `tests/litellm/` -- Ensure `make test-unit` passes - -## Architecture Overview - -LiteLLM is a unified interface for 100+ LLM providers with two main components: - -### Core Library (`litellm/`) -- **Main entry point**: `litellm/main.py` - Contains core completion() function -- **Provider implementations**: `litellm/llms/` - Each provider has its own subdirectory -- **Router system**: `litellm/router.py` + `litellm/router_utils/` - Load balancing and fallback logic -- **Type definitions**: `litellm/types/` - Pydantic models and type hints -- **Integrations**: `litellm/integrations/` - Third-party observability, caching, logging -- **Caching**: `litellm/caching/` - Multiple cache backends (Redis, in-memory, S3, etc.) - -### Proxy Server (`litellm/proxy/`) -- **Main server**: `proxy_server.py` - FastAPI application -- **Authentication**: `auth/` - API key management, JWT, OAuth2 -- **Database**: `db/` - Prisma ORM with PostgreSQL/SQLite support -- **Management endpoints**: `management_endpoints/` - Admin APIs for keys, teams, models -- **Pass-through endpoints**: `pass_through_endpoints/` - Provider-specific API forwarding -- **Guardrails**: `guardrails/` - Safety and content filtering hooks -- **UI Dashboard**: Served from `_experimental/out/` (Next.js build) - -## Key Patterns - -### Provider Implementation -- Providers inherit from base classes in `litellm/llms/base.py` -- Each provider has transformation functions for input/output formatting -- Support both sync and async operations -- Handle streaming responses and function calling - -### Error Handling -- Provider-specific exceptions mapped to OpenAI-compatible errors -- Fallback logic handled by Router system -- Comprehensive logging through `litellm/_logging.py` - -### Configuration -- YAML config files for proxy server (see `proxy/example_config_yaml/`) -- Environment variables for API keys and settings -- Database schema managed via Prisma (`proxy/schema.prisma`) - -## Development Notes - -### Code Style -- Uses Black formatter, Ruff linter, MyPy type checker -- Pydantic v2 for data validation -- Async/await patterns throughout -- Type hints required for all public APIs - -### Testing Strategy -- Unit tests in `tests/test_litellm/` -- Integration tests for each provider in `tests/llm_translation/` -- Proxy tests in `tests/proxy_unit_tests/` -- Load tests in `tests/load_tests/` - -### Database Migrations -- Prisma handles schema migrations -- Migration files auto-generated with `prisma migrate dev` -- Always test migrations against both PostgreSQL and SQLite - -### Enterprise Features -- Enterprise-specific code in `enterprise/` directory -- Optional features enabled via environment variables -- Separate licensing and authentication for enterprise features \ No newline at end of file +# LiteLLM Bug Fix: Qwen3 Tool Calls Dropped + +## Issue +https://github.com/BerriAI/litellm/issues/18922 + +## Problem Summary +When using qwen3 models through LiteLLM's Ollama provider, `tool_calls` are dropped from the response. The response contains only `content: "{}"` while valid `tool_calls` from Ollama are lost. + +**Root Cause**: Qwen3 includes a `thinking` field in its responses that qwen2.5 does not. The Ollama response handler doesn't properly handle responses that have both `thinking` and `tool_calls`. + +## Files to Investigate + +1. **`litellm/llms/ollama/completion/transformation.py`** - Ollama response transformation +2. **`litellm/llms/ollama_chat.py`** - Ollama chat handler (legacy) +3. **`litellm/llms/ollama/chat/transformation.py`** - Ollama chat transformation + +## Expected Fix Location + +Look for where the Ollama response message is parsed. The code likely does something like: +```python +content = message.get("content", "") +``` + +But doesn't extract: +```python +tool_calls = message.get("tool_calls", []) +thinking = message.get("thinking", "") # qwen3 specific +``` + +## Test Cases + +### Test 1: Qwen3 with tool_calls should work + +```python +def test_ollama_qwen3_tool_calls(): + """Test that qwen3 tool_calls are properly forwarded.""" + import litellm + + response = litellm.completion( + model="ollama/qwen3:14b", + messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], + tools=[{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather for a location", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string"}}, + "required": ["location"] + } + } + }], + api_base="http://localhost:11434" + ) + + # This currently fails - tool_calls is None + assert response.choices[0].message.tool_calls is not None + assert len(response.choices[0].message.tool_calls) > 0 + assert response.choices[0].message.tool_calls[0].function.name == "get_weather" +``` + +### Test 2: Mock Ollama response with thinking field + +```python +def test_ollama_response_with_thinking_field(): + """Test that responses with 'thinking' field preserve tool_calls.""" + from litellm.llms.ollama.chat.transformation import OllamaChatConfig + + # Simulated Ollama response (what qwen3 returns) + mock_ollama_response = { + "message": { + "role": "assistant", + "content": "", + "thinking": "Let me check the weather function...", + "tool_calls": [{ + "id": "call_abc123", + "function": { + "name": "get_weather", + "arguments": {"location": "Tokyo"} + } + }] + }, + "done": True + } + + # Transform to OpenAI format + # The fix should ensure tool_calls are preserved + result = transform_ollama_response(mock_ollama_response) + + assert result["choices"][0]["message"]["tool_calls"] is not None + assert result["choices"][0]["message"]["tool_calls"][0]["function"]["name"] == "get_weather" +``` + +### Test 3: Arguments should be JSON string (OpenAI format) + +```python +def test_ollama_tool_call_arguments_are_stringified(): + """Ollama returns arguments as dict, OpenAI expects JSON string.""" + # Ollama returns: {"arguments": {"location": "Tokyo"}} + # OpenAI expects: {"arguments": "{\"location\": \"Tokyo\"}"} + + # The fix should stringify the arguments + assert isinstance( + result["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"], + str + ) +``` + +## Reproduction Commands + +```bash +# Direct Ollama (works) +curl -s http://localhost:11434/api/chat -d '{ + "model": "qwen3:14b", + "messages": [{"role": "user", "content": "Weather in Tokyo?"}], + "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], + "stream": false +}' | jq '.message.tool_calls' +# Returns: [{"function": {"name": "get_weather", "arguments": {"location": "Tokyo"}}}] + +# Through LiteLLM (broken) +curl -s http://localhost:4000/v1/chat/completions \ + -H "Authorization: Bearer sk-xxx" \ + -d '{"model": "qwen3", "messages": [{"role": "user", "content": "Weather in Tokyo?"}], "tools": [...]}' \ + | jq '.choices[0].message' +# Returns: {"content": "{}", "role": "assistant"} # tool_calls missing! +``` + +## Fix Checklist + +- [ ] Find where Ollama response is transformed to OpenAI format +- [ ] Ensure `tool_calls` is extracted from `message.tool_calls` +- [ ] Handle the `thinking` field (either include it or ignore it, but don't let it break tool_calls) +- [ ] Stringify `arguments` dict to JSON string for OpenAI compatibility +- [ ] Add unit test for qwen3-style responses with `thinking` field +- [ ] Test with actual qwen3:14b model From 1b17a43d7695b74b3792b4acd9865be37bb26d1c Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Sun, 11 Jan 2026 01:01:13 -0700 Subject: [PATCH 2/5] fix(ollama): set finish_reason to "tool_calls" when tool_calls present When qwen3 models return tool_calls through Ollama, the finish_reason was incorrectly left as "stop" instead of being set to "tool_calls". This caused clients to miss the tool_calls in the response. Added _get_finish_reason helper method following OpenAI provider's pattern, and fixed both streaming and non-streaming response paths. Fixes: https://github.com/BerriAI/litellm/issues/18922 --- litellm/llms/ollama/chat/transformation.py | 24 +- .../ollama/test_ollama_chat_transformation.py | 329 ++++++++++++++++++ 2 files changed, 352 insertions(+), 1 deletion(-) diff --git a/litellm/llms/ollama/chat/transformation.py b/litellm/llms/ollama/chat/transformation.py index 9c8700daf838..3fbe0e4d63da 100644 --- a/litellm/llms/ollama/chat/transformation.py +++ b/litellm/llms/ollama/chat/transformation.py @@ -353,6 +353,22 @@ def transform_request( return data + def _get_finish_reason( + self, message: litellm.Message, received_finish_reason: str + ) -> str: + """ + Determine the correct finish_reason based on message content. + + If tool_calls are present, return "tool_calls" to ensure clients + properly process the tool call response. + + Follows the same pattern as OpenAI provider's _get_finish_reason. + Fixes: https://github.com/BerriAI/litellm/issues/18922 + """ + if message.tool_calls is not None: + return "tool_calls" + return received_finish_reason + def transform_response( self, model: str, @@ -428,9 +444,11 @@ def transform_response( model_response.choices[0].message = message # type: ignore model_response.choices[0].finish_reason = "tool_calls" else: - _message = litellm.Message(**response_json_message) model_response.choices[0].message = _message # type: ignore + model_response.choices[0].finish_reason = self._get_finish_reason( + _message, "stop" + ) model_response.created = int(time.time()) model_response.model = "ollama_chat/" + model prompt_tokens = response_json.get("prompt_eval_count", litellm.token_counter(messages=messages)) # type: ignore @@ -563,6 +581,10 @@ def chunk_parser(self, chunk: dict) -> ModelResponseStream: if chunk["done"] is True: finish_reason = chunk.get("done_reason", "stop") + # Ensure finish_reason is "tool_calls" when tool_calls are present + # Fixes: https://github.com/BerriAI/litellm/issues/18922 + if tool_calls is not None and finish_reason != "tool_calls": + finish_reason = "tool_calls" choices = [ StreamingChoices( delta=delta, diff --git a/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py b/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py index fc4a3e43573f..431e5277b9fe 100644 --- a/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py +++ b/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py @@ -323,3 +323,332 @@ def test_transform_request_no_images_no_images_key(self): # and the code checks "if images is not None", an empty list will still be set assert "images" in result["messages"][0] assert result["messages"][0]["images"] == [] + + +class TestOllamaChatTransformResponse: + """Tests for transform_response method, especially for qwen3 tool_calls handling. + + Issue: https://github.com/BerriAI/litellm/issues/18922 + Qwen3 includes a 'thinking' field in responses that was causing tool_calls to be dropped. + """ + + def _create_mock_response(self, json_data: dict): + """Create a mock httpx Response object.""" + import json + from unittest.mock import MagicMock, PropertyMock + + mock_response = MagicMock() + mock_response.json.return_value = json_data + mock_response.text = json.dumps(json_data) + return mock_response + + def _create_mock_logging_obj(self): + """Create a mock logging object.""" + from unittest.mock import MagicMock + + mock_logging = MagicMock() + mock_logging.post_call = MagicMock() + return mock_logging + + def _create_model_response(self): + """Create a base ModelResponse object.""" + import litellm + from litellm.types.utils import Choices, Message, ModelResponse + + model_response = ModelResponse() + model_response.choices = [Choices(message=Message(content=""), index=0)] + return model_response + + def test_transform_response_with_qwen3_thinking_and_tool_calls(self): + """Test that qwen3 responses with 'thinking' field preserve tool_calls. + + This is the core bug fix test - qwen3 returns both 'thinking' and 'tool_calls' + and the tool_calls were being dropped. + """ + import json + + config = OllamaChatConfig() + + # Simulated qwen3 response with thinking and tool_calls + ollama_response = { + "model": "qwen3:14b", + "created_at": "2025-01-11T00:00:00.000000Z", + "message": { + "role": "assistant", + "content": "", + "thinking": "Let me analyze this request and call the appropriate function...", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": {"location": "Tokyo", "units": "celsius"}, + } + } + ], + }, + "done": True, + "prompt_eval_count": 100, + "eval_count": 50, + } + + mock_response = self._create_mock_response(ollama_response) + mock_logging = self._create_mock_logging_obj() + model_response = self._create_model_response() + + result = config.transform_response( + model="qwen3:14b", + raw_response=mock_response, + model_response=model_response, + logging_obj=mock_logging, + request_data={}, + messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], + optional_params={}, + litellm_params={"api_base": "http://localhost:11434"}, + encoding="utf-8", + ) + + # Verify tool_calls are preserved + assert result.choices[0].message.tool_calls is not None, ( + "tool_calls should not be None - this is the core qwen3 bug!" + ) + assert len(result.choices[0].message.tool_calls) == 1 + + # Verify tool_call structure matches OpenAI format + tool_call = result.choices[0].message.tool_calls[0] + assert tool_call.function.name == "get_weather" + + # Verify arguments are stringified (OpenAI expects JSON string, not dict) + assert isinstance(tool_call.function.arguments, str) + parsed_args = json.loads(tool_call.function.arguments) + assert parsed_args["location"] == "Tokyo" + assert parsed_args["units"] == "celsius" + + # Verify id and type are set (auto-generated for Ollama responses) + assert tool_call.id is not None + assert tool_call.type == "function" + + # Verify thinking was remapped to reasoning_content + assert result.choices[0].message.reasoning_content is not None + assert "analyze this request" in result.choices[0].message.reasoning_content + + # Verify finish_reason is set to "tool_calls" (critical for clients to process tool calls) + assert result.choices[0].finish_reason == "tool_calls", ( + "finish_reason should be 'tool_calls' when tool_calls are present!" + ) + + def test_transform_response_with_tool_calls_no_thinking(self): + """Test that tool_calls work without thinking field (standard Ollama models).""" + import json + + config = OllamaChatConfig() + + # Standard Ollama response with tool_calls but no thinking + ollama_response = { + "model": "llama3.1:8b", + "created_at": "2025-01-11T00:00:00.000000Z", + "message": { + "role": "assistant", + "content": "", + "tool_calls": [ + { + "function": { + "name": "search_database", + "arguments": {"query": "test"}, + } + } + ], + }, + "done": True, + "prompt_eval_count": 50, + "eval_count": 25, + } + + mock_response = self._create_mock_response(ollama_response) + mock_logging = self._create_mock_logging_obj() + model_response = self._create_model_response() + + result = config.transform_response( + model="llama3.1:8b", + raw_response=mock_response, + model_response=model_response, + logging_obj=mock_logging, + request_data={}, + messages=[{"role": "user", "content": "Search for test"}], + optional_params={}, + litellm_params={"api_base": "http://localhost:11434"}, + encoding="utf-8", + ) + + # Verify tool_calls are preserved + assert result.choices[0].message.tool_calls is not None + assert len(result.choices[0].message.tool_calls) == 1 + assert result.choices[0].message.tool_calls[0].function.name == "search_database" + + # Verify finish_reason is "tool_calls" + assert result.choices[0].finish_reason == "tool_calls" + + def test_transform_response_multiple_tool_calls(self): + """Test handling of multiple tool_calls in a single response.""" + import json + + config = OllamaChatConfig() + + ollama_response = { + "model": "qwen3:14b", + "created_at": "2025-01-11T00:00:00.000000Z", + "message": { + "role": "assistant", + "content": "", + "thinking": "I need to get weather for both cities...", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": {"location": "Tokyo"}, + } + }, + { + "function": { + "name": "get_weather", + "arguments": {"location": "New York"}, + } + }, + ], + }, + "done": True, + "prompt_eval_count": 100, + "eval_count": 75, + } + + mock_response = self._create_mock_response(ollama_response) + mock_logging = self._create_mock_logging_obj() + model_response = self._create_model_response() + + result = config.transform_response( + model="qwen3:14b", + raw_response=mock_response, + model_response=model_response, + logging_obj=mock_logging, + request_data={}, + messages=[{"role": "user", "content": "Weather in Tokyo and New York?"}], + optional_params={}, + litellm_params={"api_base": "http://localhost:11434"}, + encoding="utf-8", + ) + + # Verify both tool_calls are preserved + assert result.choices[0].message.tool_calls is not None + assert len(result.choices[0].message.tool_calls) == 2 + + # Verify each tool_call + tool_call_1 = result.choices[0].message.tool_calls[0] + tool_call_2 = result.choices[0].message.tool_calls[1] + + assert tool_call_1.function.name == "get_weather" + assert json.loads(tool_call_1.function.arguments)["location"] == "Tokyo" + + assert tool_call_2.function.name == "get_weather" + assert json.loads(tool_call_2.function.arguments)["location"] == "New York" + + # Each tool_call should have unique id + assert tool_call_1.id != tool_call_2.id + + # Verify finish_reason is "tool_calls" + assert result.choices[0].finish_reason == "tool_calls" + + def test_transform_response_content_with_tool_calls(self): + """Test that content and tool_calls can coexist.""" + config = OllamaChatConfig() + + ollama_response = { + "model": "qwen3:14b", + "message": { + "role": "assistant", + "content": "I'll check the weather for you.", + "thinking": "User wants weather info...", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": {"location": "Tokyo"}, + } + } + ], + }, + "done": True, + "prompt_eval_count": 50, + "eval_count": 30, + } + + mock_response = self._create_mock_response(ollama_response) + mock_logging = self._create_mock_logging_obj() + model_response = self._create_model_response() + + result = config.transform_response( + model="qwen3:14b", + raw_response=mock_response, + model_response=model_response, + logging_obj=mock_logging, + request_data={}, + messages=[{"role": "user", "content": "Weather?"}], + optional_params={}, + litellm_params={"api_base": "http://localhost:11434"}, + encoding="utf-8", + ) + + # Both content and tool_calls should be present + assert result.choices[0].message.content == "I'll check the weather for you." + assert result.choices[0].message.tool_calls is not None + assert len(result.choices[0].message.tool_calls) == 1 + + # Verify finish_reason is "tool_calls" + assert result.choices[0].finish_reason == "tool_calls" + + def test_transform_response_empty_content_string(self): + """Test that empty content string with tool_calls works correctly.""" + config = OllamaChatConfig() + + # This is what qwen3 often returns - empty content with tool_calls + ollama_response = { + "model": "qwen3:14b", + "message": { + "role": "assistant", + "content": "", # Empty string, not None + "thinking": "Calling the function...", + "tool_calls": [ + { + "function": { + "name": "calculate", + "arguments": {"expression": "2+2"}, + } + } + ], + }, + "done": True, + "prompt_eval_count": 30, + "eval_count": 20, + } + + mock_response = self._create_mock_response(ollama_response) + mock_logging = self._create_mock_logging_obj() + model_response = self._create_model_response() + + result = config.transform_response( + model="qwen3:14b", + raw_response=mock_response, + model_response=model_response, + logging_obj=mock_logging, + request_data={}, + messages=[{"role": "user", "content": "Calculate 2+2"}], + optional_params={}, + litellm_params={"api_base": "http://localhost:11434"}, + encoding="utf-8", + ) + + # Content should be empty string (or None), but tool_calls should be present + assert result.choices[0].message.tool_calls is not None + assert len(result.choices[0].message.tool_calls) == 1 + assert result.choices[0].message.tool_calls[0].function.name == "calculate" + + # Verify finish_reason is "tool_calls" + assert result.choices[0].finish_reason == "tool_calls" From 611c73bde87c07c31266d9d6915b59ac7414500b Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Sun, 11 Jan 2026 02:17:10 -0700 Subject: [PATCH 3/5] fix(ollama): pass tools directly without model capability check The previous code tried to check model capability via get_model_info() which made network calls to localhost:11434. When Ollama is remote, this fails and falls back to JSON format, breaking tool calling. Ollama 0.4+ supports native tool calling - let Ollama handle model capability detection instead of LiteLLM. Fixes #18922 --- litellm/llms/ollama/chat/transformation.py | 43 +++------------------- uv.lock | 3 ++ 2 files changed, 9 insertions(+), 37 deletions(-) create mode 100644 uv.lock diff --git a/litellm/llms/ollama/chat/transformation.py b/litellm/llms/ollama/chat/transformation.py index 3fbe0e4d63da..e62f168520c8 100644 --- a/litellm/llms/ollama/chat/transformation.py +++ b/litellm/llms/ollama/chat/transformation.py @@ -190,46 +190,15 @@ def map_openai_params( else: optional_params["think"] = value in {"low", "medium", "high"} ### FUNCTION CALLING LOGIC ### + # Ollama 0.4+ supports native tool calling - pass tools directly + # and let Ollama handle model capability detection + # See: https://github.com/BerriAI/litellm/issues/18922 if param == "tools": - ## CHECK IF MODEL SUPPORTS TOOL CALLING ## - try: - model_info = litellm.get_model_info( - model=model, custom_llm_provider="ollama" - ) - if model_info.get("supports_function_calling") is True: - optional_params["tools"] = value - else: - raise Exception - except Exception: - optional_params["format"] = "json" - litellm.add_function_to_prompt = ( - True # so that main.py adds the function call to the prompt - ) - optional_params["functions_unsupported_model"] = value - - if len(optional_params["functions_unsupported_model"]) == 1: - optional_params["function_name"] = optional_params[ - "functions_unsupported_model" - ][0]["function"]["name"] + optional_params["tools"] = value if param == "functions": - ## CHECK IF MODEL SUPPORTS TOOL CALLING ## - try: - model_info = litellm.get_model_info( - model=model, custom_llm_provider="ollama" - ) - if model_info.get("supports_function_calling") is True: - optional_params["tools"] = value - else: - raise Exception - except Exception: - optional_params["format"] = "json" - litellm.add_function_to_prompt = ( - True # so that main.py adds the function call to the prompt - ) - optional_params["functions_unsupported_model"] = ( - non_default_params.get("functions") - ) + # Convert functions to tools format for Ollama + optional_params["tools"] = value non_default_params.pop("tool_choice", None) # causes ollama requests to hang non_default_params.pop("functions", None) # causes ollama requests to hang return optional_params diff --git a/uv.lock b/uv.lock new file mode 100644 index 000000000000..bda0207302bb --- /dev/null +++ b/uv.lock @@ -0,0 +1,3 @@ +version = 1 +revision = 3 +requires-python = ">=3.13" From 62a5d627202b5cd59a85836b234f6add4b25085b Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Sun, 11 Jan 2026 02:48:50 -0700 Subject: [PATCH 4/5] fix(ollama): transform tool_calls response to OpenAI format Ollama returns tool_calls with arguments as dict, but OpenAI format requires arguments to be a JSON string. Also ensures 'type': 'function' field is present. Completes the fix for #18922 --- litellm/llms/ollama/chat/transformation.py | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/litellm/llms/ollama/chat/transformation.py b/litellm/llms/ollama/chat/transformation.py index e62f168520c8..516e1d31954e 100644 --- a/litellm/llms/ollama/chat/transformation.py +++ b/litellm/llms/ollama/chat/transformation.py @@ -413,6 +413,27 @@ def transform_response( model_response.choices[0].message = message # type: ignore model_response.choices[0].finish_reason = "tool_calls" else: + # Transform Ollama tool_calls to OpenAI format if present + # Ollama returns arguments as dict, OpenAI expects JSON string + # See: https://github.com/BerriAI/litellm/issues/18922 + if "tool_calls" in response_json_message and response_json_message["tool_calls"]: + transformed_tool_calls = [] + for tc in response_json_message["tool_calls"]: + func = tc.get("function", {}) + args = func.get("arguments", {}) + # Stringify arguments if it's a dict + if isinstance(args, dict): + args = json.dumps(args) + transformed_tool_calls.append({ + "id": tc.get("id", f"call_{str(uuid.uuid4())}"), + "type": "function", + "function": { + "name": func.get("name", ""), + "arguments": args, + } + }) + response_json_message["tool_calls"] = transformed_tool_calls + _message = litellm.Message(**response_json_message) model_response.choices[0].message = _message # type: ignore model_response.choices[0].finish_reason = self._get_finish_reason( From 7685190366e74655e4f6fcf6a796ccc24ab3282d Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Sun, 11 Jan 2026 10:41:30 -0700 Subject: [PATCH 5/5] fix(ollama): set finish_reason to "tool_calls" when tool_calls present Fixes #18922 Two issues addressed: 1. Remove broken model capability check - get_model_info() fails when Ollama runs on remote server - Broken fallback triggered JSON prompt injection - Now passes tools directly - Ollama 0.4+ handles detection 2. Set finish_reason correctly - Was hardcoded to "stop" even with tool_calls present - Clients use this to know how to process the response - Now returns "tool_calls" when tool_calls are in response Both streaming and non-streaming responses are fixed. Tests: - All 14 existing Ollama tests pass - Added 3 focused tests for the fixes --- CLAUDE.md | 245 ++++++------- litellm/llms/ollama/chat/transformation.py | 50 +-- .../ollama/test_ollama_chat_transformation.py | 331 ++++-------------- 3 files changed, 191 insertions(+), 435 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7a80f206015b..23a0e97eaeec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,137 +1,108 @@ -# LiteLLM Bug Fix: Qwen3 Tool Calls Dropped - -## Issue -https://github.com/BerriAI/litellm/issues/18922 - -## Problem Summary -When using qwen3 models through LiteLLM's Ollama provider, `tool_calls` are dropped from the response. The response contains only `content: "{}"` while valid `tool_calls` from Ollama are lost. - -**Root Cause**: Qwen3 includes a `thinking` field in its responses that qwen2.5 does not. The Ollama response handler doesn't properly handle responses that have both `thinking` and `tool_calls`. - -## Files to Investigate - -1. **`litellm/llms/ollama/completion/transformation.py`** - Ollama response transformation -2. **`litellm/llms/ollama_chat.py`** - Ollama chat handler (legacy) -3. **`litellm/llms/ollama/chat/transformation.py`** - Ollama chat transformation - -## Expected Fix Location - -Look for where the Ollama response message is parsed. The code likely does something like: -```python -content = message.get("content", "") -``` - -But doesn't extract: -```python -tool_calls = message.get("tool_calls", []) -thinking = message.get("thinking", "") # qwen3 specific -``` - -## Test Cases - -### Test 1: Qwen3 with tool_calls should work - -```python -def test_ollama_qwen3_tool_calls(): - """Test that qwen3 tool_calls are properly forwarded.""" - import litellm - - response = litellm.completion( - model="ollama/qwen3:14b", - messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], - tools=[{ - "type": "function", - "function": { - "name": "get_weather", - "description": "Get weather for a location", - "parameters": { - "type": "object", - "properties": {"location": {"type": "string"}}, - "required": ["location"] - } - } - }], - api_base="http://localhost:11434" - ) - - # This currently fails - tool_calls is None - assert response.choices[0].message.tool_calls is not None - assert len(response.choices[0].message.tool_calls) > 0 - assert response.choices[0].message.tool_calls[0].function.name == "get_weather" -``` - -### Test 2: Mock Ollama response with thinking field - -```python -def test_ollama_response_with_thinking_field(): - """Test that responses with 'thinking' field preserve tool_calls.""" - from litellm.llms.ollama.chat.transformation import OllamaChatConfig - - # Simulated Ollama response (what qwen3 returns) - mock_ollama_response = { - "message": { - "role": "assistant", - "content": "", - "thinking": "Let me check the weather function...", - "tool_calls": [{ - "id": "call_abc123", - "function": { - "name": "get_weather", - "arguments": {"location": "Tokyo"} - } - }] - }, - "done": True - } - - # Transform to OpenAI format - # The fix should ensure tool_calls are preserved - result = transform_ollama_response(mock_ollama_response) - - assert result["choices"][0]["message"]["tool_calls"] is not None - assert result["choices"][0]["message"]["tool_calls"][0]["function"]["name"] == "get_weather" -``` - -### Test 3: Arguments should be JSON string (OpenAI format) - -```python -def test_ollama_tool_call_arguments_are_stringified(): - """Ollama returns arguments as dict, OpenAI expects JSON string.""" - # Ollama returns: {"arguments": {"location": "Tokyo"}} - # OpenAI expects: {"arguments": "{\"location\": \"Tokyo\"}"} - - # The fix should stringify the arguments - assert isinstance( - result["choices"][0]["message"]["tool_calls"][0]["function"]["arguments"], - str - ) -``` - -## Reproduction Commands - -```bash -# Direct Ollama (works) -curl -s http://localhost:11434/api/chat -d '{ - "model": "qwen3:14b", - "messages": [{"role": "user", "content": "Weather in Tokyo?"}], - "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}], - "stream": false -}' | jq '.message.tool_calls' -# Returns: [{"function": {"name": "get_weather", "arguments": {"location": "Tokyo"}}}] - -# Through LiteLLM (broken) -curl -s http://localhost:4000/v1/chat/completions \ - -H "Authorization: Bearer sk-xxx" \ - -d '{"model": "qwen3", "messages": [{"role": "user", "content": "Weather in Tokyo?"}], "tools": [...]}' \ - | jq '.choices[0].message' -# Returns: {"content": "{}", "role": "assistant"} # tool_calls missing! -``` - -## Fix Checklist - -- [ ] Find where Ollama response is transformed to OpenAI format -- [ ] Ensure `tool_calls` is extracted from `message.tool_calls` -- [ ] Handle the `thinking` field (either include it or ignore it, but don't let it break tool_calls) -- [ ] Stringify `arguments` dict to JSON string for OpenAI compatibility -- [ ] Add unit test for qwen3-style responses with `thinking` field -- [ ] Test with actual qwen3:14b model +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Development Commands + +### Installation +- `make install-dev` - Install core development dependencies +- `make install-proxy-dev` - Install proxy development dependencies with full feature set +- `make install-test-deps` - Install all test dependencies + +### Testing +- `make test` - Run all tests +- `make test-unit` - Run unit tests (tests/test_litellm) with 4 parallel workers +- `make test-integration` - Run integration tests (excludes unit tests) +- `pytest tests/` - Direct pytest execution + +### Code Quality +- `make lint` - Run all linting (Ruff, MyPy, Black, circular imports, import safety) +- `make format` - Apply Black code formatting +- `make lint-ruff` - Run Ruff linting only +- `make lint-mypy` - Run MyPy type checking only + +### Single Test Files +- `poetry run pytest tests/path/to/test_file.py -v` - Run specific test file +- `poetry run pytest tests/path/to/test_file.py::test_function -v` - Run specific test + +### Running Scripts +- `poetry run python script.py` - Run Python scripts (use for non-test files) + +### GitHub Issue & PR Templates +When contributing to the project, use the appropriate templates: + +**Bug Reports** (`.github/ISSUE_TEMPLATE/bug_report.yml`): +- Describe what happened vs. what you expected +- Include relevant log output +- Specify your LiteLLM version + +**Feature Requests** (`.github/ISSUE_TEMPLATE/feature_request.yml`): +- Describe the feature clearly +- Explain the motivation and use case + +**Pull Requests** (`.github/pull_request_template.md`): +- Add at least 1 test in `tests/litellm/` +- Ensure `make test-unit` passes + +## Architecture Overview + +LiteLLM is a unified interface for 100+ LLM providers with two main components: + +### Core Library (`litellm/`) +- **Main entry point**: `litellm/main.py` - Contains core completion() function +- **Provider implementations**: `litellm/llms/` - Each provider has its own subdirectory +- **Router system**: `litellm/router.py` + `litellm/router_utils/` - Load balancing and fallback logic +- **Type definitions**: `litellm/types/` - Pydantic models and type hints +- **Integrations**: `litellm/integrations/` - Third-party observability, caching, logging +- **Caching**: `litellm/caching/` - Multiple cache backends (Redis, in-memory, S3, etc.) + +### Proxy Server (`litellm/proxy/`) +- **Main server**: `proxy_server.py` - FastAPI application +- **Authentication**: `auth/` - API key management, JWT, OAuth2 +- **Database**: `db/` - Prisma ORM with PostgreSQL/SQLite support +- **Management endpoints**: `management_endpoints/` - Admin APIs for keys, teams, models +- **Pass-through endpoints**: `pass_through_endpoints/` - Provider-specific API forwarding +- **Guardrails**: `guardrails/` - Safety and content filtering hooks +- **UI Dashboard**: Served from `_experimental/out/` (Next.js build) + +## Key Patterns + +### Provider Implementation +- Providers inherit from base classes in `litellm/llms/base.py` +- Each provider has transformation functions for input/output formatting +- Support both sync and async operations +- Handle streaming responses and function calling + +### Error Handling +- Provider-specific exceptions mapped to OpenAI-compatible errors +- Fallback logic handled by Router system +- Comprehensive logging through `litellm/_logging.py` + +### Configuration +- YAML config files for proxy server (see `proxy/example_config_yaml/`) +- Environment variables for API keys and settings +- Database schema managed via Prisma (`proxy/schema.prisma`) + +## Development Notes + +### Code Style +- Uses Black formatter, Ruff linter, MyPy type checker +- Pydantic v2 for data validation +- Async/await patterns throughout +- Type hints required for all public APIs + +### Testing Strategy +- Unit tests in `tests/test_litellm/` +- Integration tests for each provider in `tests/llm_translation/` +- Proxy tests in `tests/proxy_unit_tests/` +- Load tests in `tests/load_tests/` + +### Database Migrations +- Prisma handles schema migrations +- Migration files auto-generated with `prisma migrate dev` +- Always test migrations against both PostgreSQL and SQLite + +### Enterprise Features +- Enterprise-specific code in `enterprise/` directory +- Optional features enabled via environment variables +- Separate licensing and authentication for enterprise features \ No newline at end of file diff --git a/litellm/llms/ollama/chat/transformation.py b/litellm/llms/ollama/chat/transformation.py index 516e1d31954e..8c98cc540502 100644 --- a/litellm/llms/ollama/chat/transformation.py +++ b/litellm/llms/ollama/chat/transformation.py @@ -192,12 +192,11 @@ def map_openai_params( ### FUNCTION CALLING LOGIC ### # Ollama 0.4+ supports native tool calling - pass tools directly # and let Ollama handle model capability detection - # See: https://github.com/BerriAI/litellm/issues/18922 + # Fixes: https://github.com/BerriAI/litellm/issues/18922 if param == "tools": optional_params["tools"] = value if param == "functions": - # Convert functions to tools format for Ollama optional_params["tools"] = value non_default_params.pop("tool_choice", None) # causes ollama requests to hang non_default_params.pop("functions", None) # causes ollama requests to hang @@ -322,22 +321,6 @@ def transform_request( return data - def _get_finish_reason( - self, message: litellm.Message, received_finish_reason: str - ) -> str: - """ - Determine the correct finish_reason based on message content. - - If tool_calls are present, return "tool_calls" to ensure clients - properly process the tool call response. - - Follows the same pattern as OpenAI provider's _get_finish_reason. - Fixes: https://github.com/BerriAI/litellm/issues/18922 - """ - if message.tool_calls is not None: - return "tool_calls" - return received_finish_reason - def transform_response( self, model: str, @@ -413,32 +396,13 @@ def transform_response( model_response.choices[0].message = message # type: ignore model_response.choices[0].finish_reason = "tool_calls" else: - # Transform Ollama tool_calls to OpenAI format if present - # Ollama returns arguments as dict, OpenAI expects JSON string - # See: https://github.com/BerriAI/litellm/issues/18922 - if "tool_calls" in response_json_message and response_json_message["tool_calls"]: - transformed_tool_calls = [] - for tc in response_json_message["tool_calls"]: - func = tc.get("function", {}) - args = func.get("arguments", {}) - # Stringify arguments if it's a dict - if isinstance(args, dict): - args = json.dumps(args) - transformed_tool_calls.append({ - "id": tc.get("id", f"call_{str(uuid.uuid4())}"), - "type": "function", - "function": { - "name": func.get("name", ""), - "arguments": args, - } - }) - response_json_message["tool_calls"] = transformed_tool_calls _message = litellm.Message(**response_json_message) model_response.choices[0].message = _message # type: ignore - model_response.choices[0].finish_reason = self._get_finish_reason( - _message, "stop" - ) + # Set finish_reason to "tool_calls" when tool_calls are present + # Fixes: https://github.com/BerriAI/litellm/issues/18922 + if _message.tool_calls: + model_response.choices[0].finish_reason = "tool_calls" model_response.created = int(time.time()) model_response.model = "ollama_chat/" + model prompt_tokens = response_json.get("prompt_eval_count", litellm.token_counter(messages=messages)) # type: ignore @@ -571,9 +535,9 @@ def chunk_parser(self, chunk: dict) -> ModelResponseStream: if chunk["done"] is True: finish_reason = chunk.get("done_reason", "stop") - # Ensure finish_reason is "tool_calls" when tool_calls are present + # Override finish_reason when tool_calls are present # Fixes: https://github.com/BerriAI/litellm/issues/18922 - if tool_calls is not None and finish_reason != "tool_calls": + if tool_calls is not None: finish_reason = "tool_calls" choices = [ StreamingChoices( diff --git a/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py b/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py index 431e5277b9fe..af6481a6cb09 100644 --- a/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py +++ b/tests/test_litellm/llms/ollama/test_ollama_chat_transformation.py @@ -325,63 +325,69 @@ def test_transform_request_no_images_no_images_key(self): assert result["messages"][0]["images"] == [] -class TestOllamaChatTransformResponse: - """Tests for transform_response method, especially for qwen3 tool_calls handling. +class TestOllamaToolCalling: + """Tests for Ollama tool calling fixes. Issue: https://github.com/BerriAI/litellm/issues/18922 - Qwen3 includes a 'thinking' field in responses that was causing tool_calls to be dropped. """ - def _create_mock_response(self, json_data: dict): - """Create a mock httpx Response object.""" - import json - from unittest.mock import MagicMock, PropertyMock - - mock_response = MagicMock() - mock_response.json.return_value = json_data - mock_response.text = json.dumps(json_data) - return mock_response - - def _create_mock_logging_obj(self): - """Create a mock logging object.""" - from unittest.mock import MagicMock + def test_tools_passed_directly_without_capability_check(self): + """Test that tools are passed directly to Ollama without model capability checks. - mock_logging = MagicMock() - mock_logging.post_call = MagicMock() - return mock_logging + Previously, the code called litellm.get_model_info() which could fail + when Ollama runs on a remote server, causing a broken fallback. + Now tools are passed directly - Ollama 0.4+ handles capability detection. + """ + tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather", + "parameters": {"type": "object", "properties": {}}, + }, + } + ] - def _create_model_response(self): - """Create a base ModelResponse object.""" - import litellm - from litellm.types.utils import Choices, Message, ModelResponse + optional_params = get_optional_params( + model="ollama_chat/qwen3:14b", + tools=tools, + custom_llm_provider="ollama_chat", + ) - model_response = ModelResponse() - model_response.choices = [Choices(message=Message(content=""), index=0)] - return model_response + # Tools should be passed through directly + assert "tools" in optional_params + assert optional_params["tools"] == tools + # Should NOT trigger the broken fallback + assert "functions_unsupported_model" not in optional_params + assert "format" not in optional_params or optional_params.get("format") != "json" - def test_transform_response_with_qwen3_thinking_and_tool_calls(self): - """Test that qwen3 responses with 'thinking' field preserve tool_calls. + def test_finish_reason_tool_calls_non_streaming(self): + """Test that finish_reason is set to 'tool_calls' when tool_calls present. - This is the core bug fix test - qwen3 returns both 'thinking' and 'tool_calls' - and the tool_calls were being dropped. + Previously, finish_reason was hardcoded to 'stop' even when tool_calls + were in the response, causing clients to ignore the tool calls. """ import json + from unittest.mock import MagicMock + + import litellm + from litellm.types.utils import Choices, Message, ModelResponse config = OllamaChatConfig() - # Simulated qwen3 response with thinking and tool_calls + # Simulated Ollama response with tool_calls ollama_response = { "model": "qwen3:14b", "created_at": "2025-01-11T00:00:00.000000Z", "message": { "role": "assistant", "content": "", - "thinking": "Let me analyze this request and call the appropriate function...", "tool_calls": [ { "function": { "name": "get_weather", - "arguments": {"location": "Tokyo", "units": "celsius"}, + "arguments": {"location": "Tokyo"}, } } ], @@ -391,247 +397,64 @@ def test_transform_response_with_qwen3_thinking_and_tool_calls(self): "eval_count": 50, } - mock_response = self._create_mock_response(ollama_response) - mock_logging = self._create_mock_logging_obj() - model_response = self._create_model_response() - - result = config.transform_response( - model="qwen3:14b", - raw_response=mock_response, - model_response=model_response, - logging_obj=mock_logging, - request_data={}, - messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], - optional_params={}, - litellm_params={"api_base": "http://localhost:11434"}, - encoding="utf-8", - ) - - # Verify tool_calls are preserved - assert result.choices[0].message.tool_calls is not None, ( - "tool_calls should not be None - this is the core qwen3 bug!" - ) - assert len(result.choices[0].message.tool_calls) == 1 - - # Verify tool_call structure matches OpenAI format - tool_call = result.choices[0].message.tool_calls[0] - assert tool_call.function.name == "get_weather" - - # Verify arguments are stringified (OpenAI expects JSON string, not dict) - assert isinstance(tool_call.function.arguments, str) - parsed_args = json.loads(tool_call.function.arguments) - assert parsed_args["location"] == "Tokyo" - assert parsed_args["units"] == "celsius" - - # Verify id and type are set (auto-generated for Ollama responses) - assert tool_call.id is not None - assert tool_call.type == "function" - - # Verify thinking was remapped to reasoning_content - assert result.choices[0].message.reasoning_content is not None - assert "analyze this request" in result.choices[0].message.reasoning_content - - # Verify finish_reason is set to "tool_calls" (critical for clients to process tool calls) - assert result.choices[0].finish_reason == "tool_calls", ( - "finish_reason should be 'tool_calls' when tool_calls are present!" - ) - - def test_transform_response_with_tool_calls_no_thinking(self): - """Test that tool_calls work without thinking field (standard Ollama models).""" - import json - - config = OllamaChatConfig() + mock_response = MagicMock() + mock_response.json.return_value = ollama_response + mock_response.text = json.dumps(ollama_response) - # Standard Ollama response with tool_calls but no thinking - ollama_response = { - "model": "llama3.1:8b", - "created_at": "2025-01-11T00:00:00.000000Z", - "message": { - "role": "assistant", - "content": "", - "tool_calls": [ - { - "function": { - "name": "search_database", - "arguments": {"query": "test"}, - } - } - ], - }, - "done": True, - "prompt_eval_count": 50, - "eval_count": 25, - } + mock_logging = MagicMock() - mock_response = self._create_mock_response(ollama_response) - mock_logging = self._create_mock_logging_obj() - model_response = self._create_model_response() + model_response = ModelResponse() + model_response.choices = [Choices(message=Message(content=""), index=0)] result = config.transform_response( - model="llama3.1:8b", + model="qwen3:14b", raw_response=mock_response, model_response=model_response, logging_obj=mock_logging, request_data={}, - messages=[{"role": "user", "content": "Search for test"}], + messages=[{"role": "user", "content": "Weather?"}], optional_params={}, - litellm_params={"api_base": "http://localhost:11434"}, - encoding="utf-8", + litellm_params={}, + encoding=None, + api_key=None, + json_mode=False, ) - # Verify tool_calls are preserved - assert result.choices[0].message.tool_calls is not None - assert len(result.choices[0].message.tool_calls) == 1 - assert result.choices[0].message.tool_calls[0].function.name == "search_database" - - # Verify finish_reason is "tool_calls" + # finish_reason should be "tool_calls", not "stop" assert result.choices[0].finish_reason == "tool_calls" + assert result.choices[0].message.tool_calls is not None - def test_transform_response_multiple_tool_calls(self): - """Test handling of multiple tool_calls in a single response.""" + def test_finish_reason_stop_when_no_tool_calls(self): + """Test that finish_reason remains 'stop' when no tool_calls present.""" import json + from unittest.mock import MagicMock + + import litellm + from litellm.types.utils import Choices, Message, ModelResponse config = OllamaChatConfig() + # Simulated Ollama response without tool_calls ollama_response = { "model": "qwen3:14b", "created_at": "2025-01-11T00:00:00.000000Z", "message": { "role": "assistant", - "content": "", - "thinking": "I need to get weather for both cities...", - "tool_calls": [ - { - "function": { - "name": "get_weather", - "arguments": {"location": "Tokyo"}, - } - }, - { - "function": { - "name": "get_weather", - "arguments": {"location": "New York"}, - } - }, - ], + "content": "Hello! How can I help you?", }, "done": True, "prompt_eval_count": 100, - "eval_count": 75, - } - - mock_response = self._create_mock_response(ollama_response) - mock_logging = self._create_mock_logging_obj() - model_response = self._create_model_response() - - result = config.transform_response( - model="qwen3:14b", - raw_response=mock_response, - model_response=model_response, - logging_obj=mock_logging, - request_data={}, - messages=[{"role": "user", "content": "Weather in Tokyo and New York?"}], - optional_params={}, - litellm_params={"api_base": "http://localhost:11434"}, - encoding="utf-8", - ) - - # Verify both tool_calls are preserved - assert result.choices[0].message.tool_calls is not None - assert len(result.choices[0].message.tool_calls) == 2 - - # Verify each tool_call - tool_call_1 = result.choices[0].message.tool_calls[0] - tool_call_2 = result.choices[0].message.tool_calls[1] - - assert tool_call_1.function.name == "get_weather" - assert json.loads(tool_call_1.function.arguments)["location"] == "Tokyo" - - assert tool_call_2.function.name == "get_weather" - assert json.loads(tool_call_2.function.arguments)["location"] == "New York" - - # Each tool_call should have unique id - assert tool_call_1.id != tool_call_2.id - - # Verify finish_reason is "tool_calls" - assert result.choices[0].finish_reason == "tool_calls" - - def test_transform_response_content_with_tool_calls(self): - """Test that content and tool_calls can coexist.""" - config = OllamaChatConfig() - - ollama_response = { - "model": "qwen3:14b", - "message": { - "role": "assistant", - "content": "I'll check the weather for you.", - "thinking": "User wants weather info...", - "tool_calls": [ - { - "function": { - "name": "get_weather", - "arguments": {"location": "Tokyo"}, - } - } - ], - }, - "done": True, - "prompt_eval_count": 50, - "eval_count": 30, + "eval_count": 50, } - mock_response = self._create_mock_response(ollama_response) - mock_logging = self._create_mock_logging_obj() - model_response = self._create_model_response() - - result = config.transform_response( - model="qwen3:14b", - raw_response=mock_response, - model_response=model_response, - logging_obj=mock_logging, - request_data={}, - messages=[{"role": "user", "content": "Weather?"}], - optional_params={}, - litellm_params={"api_base": "http://localhost:11434"}, - encoding="utf-8", - ) - - # Both content and tool_calls should be present - assert result.choices[0].message.content == "I'll check the weather for you." - assert result.choices[0].message.tool_calls is not None - assert len(result.choices[0].message.tool_calls) == 1 - - # Verify finish_reason is "tool_calls" - assert result.choices[0].finish_reason == "tool_calls" - - def test_transform_response_empty_content_string(self): - """Test that empty content string with tool_calls works correctly.""" - config = OllamaChatConfig() + mock_response = MagicMock() + mock_response.json.return_value = ollama_response + mock_response.text = json.dumps(ollama_response) - # This is what qwen3 often returns - empty content with tool_calls - ollama_response = { - "model": "qwen3:14b", - "message": { - "role": "assistant", - "content": "", # Empty string, not None - "thinking": "Calling the function...", - "tool_calls": [ - { - "function": { - "name": "calculate", - "arguments": {"expression": "2+2"}, - } - } - ], - }, - "done": True, - "prompt_eval_count": 30, - "eval_count": 20, - } + mock_logging = MagicMock() - mock_response = self._create_mock_response(ollama_response) - mock_logging = self._create_mock_logging_obj() - model_response = self._create_model_response() + model_response = ModelResponse() + model_response.choices = [Choices(message=Message(content=""), index=0)] result = config.transform_response( model="qwen3:14b", @@ -639,16 +462,14 @@ def test_transform_response_empty_content_string(self): model_response=model_response, logging_obj=mock_logging, request_data={}, - messages=[{"role": "user", "content": "Calculate 2+2"}], + messages=[{"role": "user", "content": "Hello"}], optional_params={}, - litellm_params={"api_base": "http://localhost:11434"}, - encoding="utf-8", + litellm_params={}, + encoding=None, + api_key=None, + json_mode=False, ) - # Content should be empty string (or None), but tool_calls should be present - assert result.choices[0].message.tool_calls is not None - assert len(result.choices[0].message.tool_calls) == 1 - assert result.choices[0].message.tool_calls[0].function.name == "calculate" - - # Verify finish_reason is "tool_calls" - assert result.choices[0].finish_reason == "tool_calls" + # finish_reason should be "stop" (default behavior) + assert result.choices[0].finish_reason == "stop" + assert result.choices[0].message.tool_calls is None