chore: adds deepseek to provider harness api tests

[sammaji]

Summary

Adds DeepSeek provider coverage to the E2E provider harness test collection, including a new dedicated "DeepSeek Anthropic-Compatible Migration" test section that pins the behavior of DeepSeek's routing through the Anthropic-compatible endpoint (/anthropic/v1/messages) instead of the OpenAI-compatible /chat/completions path.

Changes

• Added deepseek/deepseek-v4-flash and deepseek/deepseek-v4-pro entries across all existing harness sections: basic chat completions, streaming, Responses API, OpenAI-passthrough, Anthropic-passthrough, function calling, and reasoning effort.
• Added a new "DeepSeek Features" folder in the provider-specific section covering function calling, SSE streaming, reasoning_effort: max with deepseek-v4-pro, and multi-turn system message conversations.
• Added a new top-level section "25. DeepSeek Anthropic-Compatible Migration" with four pinned test cases:
  • Non-streaming chat completions through the new Anthropic-compatible route, asserting a non-empty content response.
  • Streaming chat completions, asserting finish_reason is reached and content accumulates across deltas.
  • Multiple tool calls in a single turn (get_weather + calculate), asserting both tool calls are returned — a scenario previously broken on the OpenAI-compatible path.
  • Native /v1/responses route (no chat-completion fallback), asserting a message output item with non-empty output_text.
• Reordered the function-calling section to place deepseek/deepseek-v4-flash after gemini/gemini-2.5-flash, and added deepseek/deepseek-v4-flash to the streaming section.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Import tests/e2e/api/collections/provider-harness.json into Postman or run via Newman against a live gateway instance with a valid DeepSeek API key configured.

newman run tests/e2e/api/collections/provider-harness.json \
  --env-var baseUrl=http://localhost:8080 \
  --folder "DeepSeek Features" \
  --folder "25. DeepSeek Anthropic-Compatible Migration"

Expected outcomes:

• All non-streaming requests return HTTP 2xx with non-empty choices[0].message.content.
• Streaming requests produce SSE chunks that accumulate content and include a finish_reason.
• The multiple-tool-call request returns at least two tool_calls entries naming get_weather and calculate.
• The native Responses API request returns an output item of type message containing non-empty output_text.

Breaking changes

• Yes
• No

Related issues

Security considerations

No new secrets or auth mechanisms introduced. DeepSeek API key handling follows the existing provider credential pattern.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[coderabbitai]

📝 Walkthrough

Walkthrough

DeepSeek provider-harness coverage now spans multiple endpoint formats, streaming, reasoning effort, multi-turn conversations, tool calling, provider variants, and Anthropic-compatible migration scenarios.

Changes

DeepSeek provider harness

Layer / File(s)	Summary
DeepSeek endpoint and streaming coverage tests/e2e/api/collections/provider-harness.json	Adds DeepSeek fixtures for chat completions, OpenAI-compatible chat, Anthropic-compatible messages, native Responses, and streaming requests.
DeepSeek features and tool-call verification tests/e2e/api/collections/provider-harness.json	Adds tool calling, reasoning effort, streaming, multi-turn conversations, and weather tool-call validation across provider and model variants.
Anthropic-compatible migration suite tests/e2e/api/collections/provider-harness.json	Adds end-to-end checks for non-streaming and streaming chat completions, multiple tool calls, and native Responses output.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Harness
  participant MigrationEndpoint
  participant DeepSeek
  Harness->>MigrationEndpoint: Send chat-completions or Responses request
  MigrationEndpoint->>DeepSeek: Route request to DeepSeek
  DeepSeek-->>MigrationEndpoint: Return completion, stream, or tool calls
  MigrationEndpoint-->>Harness: Return migrated response
  Harness->>Harness: Assert content, finish_reason, output_text, or tool_calls

Loading

Possibly related PRs

• maximhq/bifrost#5176: Implements the DeepSeek Anthropic-compatible migration endpoints exercised by this harness coverage.

Suggested reviewers: akshaydeo, tejasghatte, danpiths, pratham-mishra04, roroghost17

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly matches the main change: adding DeepSeek coverage to the provider harness API tests.
Description check	✅ Passed	The description largely follows the template and includes summary, changes, type, affected areas, testing, security, and checklist details.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 07-15-chore_adds_deepseek_to_provider_harness_api_tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[sammaji]

• fix: fallback select shows trimmed model names #5257
• fix: drops unsupported reasoning summary values for azure model router #5242
• fix: adds support for anthropic apis in fireworks #5332
• fix: moves vllm to native responses api #5306
• fix: adds count tokens handling for deepseek and sgl #5311
• fix: enable raw request and store for mistral ocr requests #5310
• chore: adds deepseek to provider harness api tests #5208 👈 (View in Graphite)
• chore: update deepseek provider docs #5177 : 1 other dependent PR (#5187 )
• fix: uses anthropic endpoints in chat completion and responses in deepseek #5176
• fix: adds sgl anthropic compatible apis via key level setting #5305
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[greptile-apps]

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
tests/e2e/api/collections/provider-harness.json	Adds DeepSeek requests and assertions across the provider harness, including focused Anthropic-compatible migration coverage.

_{Reviews (10): Last reviewed commit: "chore: adds deepseek to provider harness..." | Re-trigger Greptile}

[coderabbitai]

🧹 Nitpick comments (1)

tests/e2e/api/collections/provider-harness.json (1)

40634-40683: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Multi-tool-call regression test is flaky and can crash instead of failing cleanly.

Two issues in this new "multiple tool calls" assertion, which is the key regression check for the PR's DeepSeek multi-tool-call fix:

• The prompt doesn't force tool_choice, so the model may legitimately answer with 0 or 1 tool calls, making the >= 2 assertion nondeterministic/flaky.
• var msg = pm.response.json().choices[0].message; is unguarded — if choices/choices[0]/message is ever absent, this throws a raw TypeError instead of a clean assertion failure, unlike the defensive if (j.choices && j.choices[0] && j.choices[0].message ...) pattern used by equivalent function-calling checks elsewhere in this file (e.g. the Cross-Cut Round 6/24 function-calling matrices).

🐛 Proposed fix

                   "if ([401, 403, 429, 500, 502, 503, 504].indexOf(pm.response.code) !== -1) { return; }",
                   "pm.test('deepseek multiple tool calls: 2xx', function () {",
                   "  pm.expect(pm.response.code, 'failed: ' + pm.response.text()).to.be.below(400);",
                   "});",
                   "if (pm.response.code >= 400) { return; }",
-                  "var msg = pm.response.json().choices[0].message;",
+                  "var j = pm.response.json();",
+                  "var msg = j.choices && j.choices[0] && j.choices[0].message;",
                   "pm.test('deepseek multiple tool calls: both tools called in a single turn', function () {",
+                  "  pm.expect(msg, 'no message in response: ' + pm.response.text()).to.exist;",
                   "  pm.expect(msg.tool_calls, 'no tool_calls: ' + pm.response.text()).to.be.an('array');",
                   "  pm.expect(msg.tool_calls.length, 'expected at least 2 tool calls: ' + pm.response.text()).to.be.at.least(2);",
                   "  var names = msg.tool_calls.map(function (tc) { return tc.function.name; });",
                   "  pm.expect(names).to.include('get_weather');",
                   "  pm.expect(names).to.include('calculate');",
                   "});"

And to reduce the risk of the model answering with fewer than 2 tool calls, consider forcing tool use in the request body, e.g. "tool_choice": "required" (or "auto" with a stronger prompt if required isn't supported for this endpoint shape).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/api/collections/provider-harness.json` around lines 40634 - 40683,
Make the DeepSeek multi-tool-call test deterministic by adding the supported
required tool-choice setting to the request body, and update the test script to
parse the response defensively before accessing choices[0].message. Ensure
missing response structure produces a clean assertion failure, while preserving
the existing checks that both get_weather and calculate appear in at least two
tool calls.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/e2e/api/collections/provider-harness.json`:
- Around line 40634-40683: Make the DeepSeek multi-tool-call test deterministic
by adding the supported required tool-choice setting to the request body, and
update the test script to parse the response defensively before accessing
choices[0].message. Ensure missing response structure produces a clean assertion
failure, while preserving the existing checks that both get_weather and
calculate appear in at least two tool calls.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 46ba17de-aa57-4f2c-aa2a-dda4789c32b2

📥 Commits

Reviewing files that changed from the base of the PR and between 7455a0a and 1d8bd59.

📒 Files selected for processing (1)

• tests/e2e/api/collections/provider-harness.json

[akshaydeo]

Merge activity

• Jul 18, 4:02 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jul 18, 4:05 PM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.