tests: harness tests for bedrock documents

[TejasGhatte]

Summary

Adds regression coverage for #5472, where Bedrock's document format converter defaulted every uploaded document to format:"pdf" regardless of the actual file type, causing AWS to reject non-PDF documents with ValidationException. This PR adds 14 end-to-end test cases to the provider harness collection covering the fixed behavior across both /v1/chat/completions and /v1/responses.

Changes

• Added folder 42. Bedrock Document Uploads via OpenAI type:"file" ([Bug]: Bedrock rejects office/PDF document uploads via OpenAI type:"file" — "The PDF specified was not valid" #5472) to the provider harness collection with 14 test cases:
  • Cases 1–11 exercise /v1/chat/completions with XLSX, DOCX, CSV, PDF, TXT, and file_url inputs, covering format resolution by data URL media type, filename extension, explicit file_type, charset-parameterized data URLs, non-base64 percent-encoded data URLs, opaque media types, and streaming
  • Cases 12–14 pin the same invariants on /v1/responses input_file blocks (XLSX data URL, CSV data URL, file_url)
  • Every fixture embeds the token BIFROST7788 so assertions confirm the document was actually parsed by Claude, not merely accepted
• Updated HARNESS_COVERAGE_BACKLOG.md to mark the Document input item as partially covered ([~]), noting that the OpenAI type:"file" / Responses input_file path is now covered by folder 42, while a native Converse-shaped document block posted directly at /bedrock/model/{id}/converse remains uncovered

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Import tests/e2e/api/collections/provider-harness.json into Postman and run folder 42. Bedrock Document Uploads via OpenAI type:"file" (#5472) against a running Bifrost instance with Bedrock credentials configured.

Each test asserts:

• The response does not contain "The PDF specified was not valid", "could not be parsed as the specified format", or "The document source bytes" (the AWS rejection messages from the bug)
• The response status is below 400
• For document-content cases, the model's reply includes BIFROST7788, confirming the document was read

Before the fix, cases 1–3, 5–8, and 12–14 all returned a 400 ValidationException.

Breaking changes

• No

Related issues

Closes #5472

Security considerations

None. Test fixtures contain only synthetic document content with no real credentials or PII.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Tests
  • Added regression coverage for Bedrock document uploads across chat completions, responses, native Converse, and streaming APIs.
  • Covered file uploads, inline data, file URLs, MIME and filename detection, explicit file types, character sets, percent-encoded data, and PDF fallbacks.
  • Added validation for XLSX, PDF, CSV, and text documents, extracted content, successful streaming completion, and error-free responses.
• Documentation
  • Updated coverage tracking to reflect partial support for Bedrock Converse document inputs.

Walkthrough

The PR adds Bedrock document-upload regression coverage to the provider harness. Tests cover format detection, data URLs, explicit file types, PDF fallback, remote files, streaming responses, and native Converse document requests.

Changes

Bedrock document coverage

Layer / File(s)	Summary
Chat Completions document coverage tests/e2e/api/collections/provider-harness.json	Added regression requests for XLSX, DOCX, CSV, PDF, data URL variants, explicit file_type, PDF fallback, remote file_url inputs, and streaming XLSX responses.
Responses API document coverage tests/e2e/api/collections/provider-harness.json	Added XLSX, CSV, and remote file_url tests for document conversion and extracted content.
Native Converse document coverage tests/e2e/api/collections/provider-harness.json, tests/e2e/api/HARNESS_COVERAGE_BACKLOG.md	Added native Converse round-trip and streaming tests for XLSX, PDF, CSV, and text documents. Marked the Converse document-input backlog item as partially covered.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• maximhq/bifrost#5503: Adds related Bedrock document-upload regression tests.
• maximhq/bifrost#5662: Changes document materialization and Converse conversion paths covered by these tests.
• maximhq/bifrost#5947: Changes format detection, data URL decoding, and file handling covered by these tests.

Suggested reviewers: pratham-mishra04, akshaydeo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The tests directly cover issue #5472 requirements for MIME-based format inference, explicit file types, PDF fallback, and Bedrock document parsing.
Out of Scope Changes check	✅ Passed	The changes are limited to Bedrock document harness coverage and the related coverage backlog update.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check	✅ Passed	The title clearly summarizes the main change: adding harness tests for Bedrock document handling.
Description check	✅ Passed	The description covers the purpose, changes, testing steps, affected areas, issue link, security impact, and checklist status.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 08-07-tests_harness_tests_for_bedrock_documents

---

_{Comment @coderabbitai help to get the list of available commands.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[TejasGhatte]

• fix: add anthropic error branch on stripping on encrypted content #5960
• fix: cost in usd ticks for xai usage #5950
• tests: harness tests for bedrock documents #5949 👈 (View in Graphite)
• fix: bedrock files handling in inference #5947
• video requests info in logs ui #5946
• fix update passthrough docs #5906
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/e2e/api/collections/provider-harness.json`:
- Around line 49121-49128: Update both remote-file tests at
tests/e2e/api/collections/provider-harness.json:49121-49128 and :49336-49343 to
use the same controlled remote fixture containing a unique marker, then assert
the parsed response content includes that marker. Replace the current generic
nonempty-content-only verification in both test cases while preserving their
respective response parsing and endpoint behavior.
- Around line 49210-49370: Update the Bedrock entry in the provider capabilities
configuration so providers.bedrock.responses is true, preserving the existing
Responses and ResponsesStream capability declarations and matching the
BedrockProvider implementation.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b51fa650-0378-478f-a6ab-0d4a9a060cfd

📥 Commits

Reviewing files that changed from the base of the PR and between 9cf7426 and a5f6ede.

📒 Files selected for processing (2)

• tests/e2e/api/HARNESS_COVERAGE_BACKLOG.md
• tests/e2e/api/collections/provider-harness.json

[coderabbitai]

Actionable comments posted: 2

♻️ Duplicate comments (1)

tests/e2e/api/collections/provider-harness.json (1)

49282-49289: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

This assertion is close to vacuous on the /v1/responses route.

On /v1/responses the response has no choices, so c becomes JSON.stringify(j.output). An empty output array is truthy and stringifies to "[]", which is a nonempty string. The .not.empty check therefore passes even when the model returned no content. The case then only asserts status < 400.

The fixture decodes to code,revenue / BIFROST7788,42. Assert the marker instead, and change the prompt to request the verification code.

🐛 Proposed fix

-                  "  pm.expect(c).to.be.a('string').and.not.empty;",
-                  "  // content-only assertion: this case pins format resolution, not document text",
+                  "  pm.expect(c.toUpperCase(), 'model did not read the document contents').to.include('BIFROST7788');",

Also change the request prompt from Summarize this document in one sentence. to Reply with only the verification code found in the attached document.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/api/collections/provider-harness.json` around lines 49282 - 49289,
Update the `/v1/responses` test around the “document accepted and answered”
assertion to request only the verification code found in the attached document,
replacing the summarization prompt. Replace the vacuous nonempty `c` check with
an assertion that the extracted response content contains the fixture’s expected
marker `BIFROST7788`, while preserving the existing successful-status assertion.

🧹 Nitpick comments (1)

tests/e2e/api/collections/provider-harness.json (1)

48901-48908: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Four cases use the "content-only assertion" pattern although their fixtures embed BIFROST7788. The shared root cause is that these cases assert only a nonempty answer. They therefore prove that Bedrock accepted the document, not that Claude parsed it. On the /v1/responses case the check is weaker still, because JSON.stringify(j.output) returns "[]" for an empty output array and passes the nonempty check.

• tests/e2e/api/collections/provider-harness.json#L48901-L48908: assert BIFROST7788 against the XLSX fixture and change the prompt to request the verification code.
• tests/e2e/api/collections/provider-harness.json#L48956-L48963: assert BIFROST7788 against the CSV fixture and change the prompt to request the verification code.
• tests/e2e/api/collections/provider-harness.json#L49066-L49073: assert BIFROST7788 against the PDF fixture and change the prompt to request the verification code.
• tests/e2e/api/collections/provider-harness.json#L49282-L49289: assert BIFROST7788 against the CSV fixture, change the prompt to request the verification code, and drop the .not.empty check that "[]" satisfies.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/api/collections/provider-harness.json` around lines 48901 - 48908,
Update the four content-only assertions in
tests/e2e/api/collections/provider-harness.json at lines 48901-48908,
48956-48963, 49066-49073, and 49282-49289: change each prompt to request the
fixture’s verification code and assert that the extracted response content
contains BIFROST7788. For the /v1/responses case at lines 49282-49289, remove
the .not.empty check so an empty serialized output cannot pass; retain the
response-success validation and use a content assertion that verifies the code.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/e2e/api/collections/provider-harness.json`:
- Around line 49141-49143: Replace the external Berkshire Hathaway file URL in
both remote-file cases with a repository-hosted or {{baseUrl}} fixture
containing the marker BIFROST7788. Update
tests/e2e/api/collections/provider-harness.json lines 49141-49143 and
49356-49358, covering the chat messages file case and the /v1/responses
input_file case.
- Around line 49176-49180: Update the streaming test in the pm.test block to
parse SSE payloads, extract and concatenate choices[].delta.content values, and
assert the combined streamed document includes BIFROST7788. Keep the existing
response status, data presence, and error-frame assertions unchanged.

---

Duplicate comments:
In `@tests/e2e/api/collections/provider-harness.json`:
- Around line 49282-49289: Update the `/v1/responses` test around the “document
accepted and answered” assertion to request only the verification code found in
the attached document, replacing the summarization prompt. Replace the vacuous
nonempty `c` check with an assertion that the extracted response content
contains the fixture’s expected marker `BIFROST7788`, while preserving the
existing successful-status assertion.

---

Nitpick comments:
In `@tests/e2e/api/collections/provider-harness.json`:
- Around line 48901-48908: Update the four content-only assertions in
tests/e2e/api/collections/provider-harness.json at lines 48901-48908,
48956-48963, 49066-49073, and 49282-49289: change each prompt to request the
fixture’s verification code and assert that the extracted response content
contains BIFROST7788. For the /v1/responses case at lines 49282-49289, remove
the .not.empty check so an empty serialized output cannot pass; retain the
response-success validation and use a content assertion that verifies the code.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1a8cb0e2-e50c-4952-8d1c-030399a0cd27

📥 Commits

Reviewing files that changed from the base of the PR and between a5f6ede and f9b058d.

📒 Files selected for processing (1)

• tests/e2e/api/collections/provider-harness.json

[akshaydeo]

Merge activity

• Aug 10, 3:13 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 10, 3:16 PM UTC: Graphite couldn't merge this pull request because a downstack PR video requests info in logs ui #5946 failed to merge.
• Aug 10, 3:19 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 10, 3:24 PM UTC: Graphite couldn't merge this pull request because a downstack PR fix: bedrock files handling in inference #5947 failed to merge.
• Aug 10, 4:36 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 10, 4:40 PM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.