fix: harness test fixes

[TejasGhatte]

Summary

Updates the provider harness test suite to use correct and available Azure model names, adds support for passing AWS Bedrock Guardrail identifiers and versions into the Newman test runner, and pins the Bedrock performance config test to a model that supports it.

Changes

• Added BEDROCK_GUARDRAIL_IDENTIFIER and BEDROCK_GUARDRAIL_VERSION environment variable passthrough to the Newman invocations in the Makefile, allowing guardrail-aware Bedrock tests to be run from the harness
• Replaced azure/text-embedding-2-ada with the correct model name azure/text-embedding-ada-002 across all embedding test entries (OpenAI, OpenAI-compat, and GenAI endpoints)
• Replaced azure/whisper-1 with azure/gpt-4o-transcribe for audio transcription tests
• Replaced azure/tts-1 with azure/gpt-4o-mini-tts for text-to-speech tests
• Changed the "Bedrock Converse: performance config optimized" test to target us.amazon.nova-pro-v1:0 directly instead of the generic {{bedrockModel}} variable, since performanceConfig is only supported on specific models
• Replaced openai/gpt-5 with openai/gpt-4o-mini in the stop sequences cross-cut test

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Run the provider harness with optional Bedrock guardrail variables:

# Without guardrails
make run-provider-harness-test

# With guardrails
BEDROCK_GUARDRAIL_IDENTIFIER=my-guardrail-id \
BEDROCK_GUARDRAIL_VERSION=1 \
make run-provider-harness-test

Verify that:

• Azure embedding, transcription, and TTS tests pass with the updated model names
• The Bedrock performance config test passes when targeting us.amazon.nova-pro-v1:0
• The stop sequences cross-cut test passes with openai/gpt-4o-mini
• Guardrail env vars are forwarded to Newman when set

Breaking changes

• Yes
• No

Related issues

Security considerations

BEDROCK_GUARDRAIL_IDENTIFIER and BEDROCK_GUARDRAIL_VERSION are passed as Newman environment variables. These values are not secrets but should be treated as infrastructure configuration and kept out of public logs if they reference sensitive guardrail resources.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

Summary by CodeRabbit

• Chores
  • Updated test harnesses to use newer Azure model identifiers for embeddings and audio services, including GPT-4o variants.
  • Enhanced test execution to support Bedrock guardrail configuration parameters.
  • Updated test matrices with latest model versions for improved compatibility testing.

[coderabbitai]

Warning

Review limit reached

@TejasGhatte, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 7 minutes and 13 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3cf037bc-9d57-42b9-b1ec-ed4aced74c57

📥 Commits

Reviewing files that changed from the base of the PR and between fc8c127 and ef74d3f.

📒 Files selected for processing (2)

• Makefile
• tests/e2e/api/collections/provider-harness.json

📝 Walkthrough

Walkthrough

The PR updates test harness configuration and model identifiers across two files. The Makefile now passes Bedrock guardrail environment variables to newman test execution in both parallel and sequential paths. The provider-harness JSON test collection updates Azure embeddings and audio model identifiers to newer variants, adjusts a Bedrock performance test entry, and updates an OpenAI stop-sequence test entry.

Changes

Test Harness and Configuration Updates

Layer / File(s)	Summary
Bedrock guardrail environment variable injection Makefile	run-provider-harness-test target adds conditional --env-var arguments for Bedrock guardrail identifier and version in both parallel per-provider and sequential newman execution paths.
Azure embeddings model identifier updates tests/e2e/api/collections/provider-harness.json	Embeddings test matrices updated: Azure embedding model changed from azure/text-embedding-2-ada to azure/text-embedding-ada-002 across native /v1/embeddings, OpenAI drop-in /openai/v1/embeddings, and GenAI drop-in :embedContent paths.
Azure audio model identifier updates tests/e2e/api/collections/provider-harness.json	Audio test matrices updated: Azure transcription model changed from azure/whisper-1 to azure/gpt-4o-transcribe in native and OpenAI drop-in paths; Azure TTS model changed from azure/tts-1 to azure/gpt-4o-mini-tts in native and OpenAI drop-in paths.
Bedrock and OpenAI test entry adjustments tests/e2e/api/collections/provider-harness.json	Bedrock performance config request explicitly targets us.amazon.nova-pro-v1:0 instead of using {{bedrockModel}} variable; stop-sequences cross-cut matrix OpenAI entry updated from openai/gpt-5 to openai/gpt-4o-mini.

Possibly related PRs

• maximhq/bifrost#3462: Updates run-provider-harness-test Makefile target usage text in the same test orchestration target.
• maximhq/bifrost#3507: Modifies run-provider-harness-test Makefile target execution paths and harness JSON generation at the same harness test orchestration layer.

Suggested reviewers

• akshaydeo
• danpiths

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 Models hop to newer versions bright,
Azure embeddings and audio take flight,
Guardrails guard the Bedrock test runs true,
A harness refresh—fresh and new! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The PR title 'fix: harness test fixes' is vague and generic, using non-descriptive language that doesn't convey the specific changes made to Azure models, Bedrock guardrails, or test configurations.	Consider a more specific title such as 'chore: update provider harness test models and add Bedrock guardrail support' to better reflect the actual changes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The PR description is well-structured, follows the template, provides clear explanations of all changes, includes detailed testing instructions, and addresses security considerations for the new environment variables.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 05-29-fix_harness_test_fixes

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[TejasGhatte]

• fix: harness test fixes #3887 👈 (View in Graphite)
• fix: add http logs back #3885
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

Safe to merge — all changes are confined to test configuration files with no impact on production code paths.

The diff touches only the Newman collection JSON and Makefile test targets. Model name corrections and env-var passthrough are self-contained, and the bash conditional expansion syntax in the Makefile is correct. No logic changes to core, transports, or plugins.

No files require special attention.

Important Files Changed

Filename	Overview
Makefile	Adds bash parameter expansion to pass BEDROCK_GUARDRAIL_IDENTIFIER and BEDROCK_GUARDRAIL_VERSION as Newman env vars in all three Newman invocations; syntax is correct for Make shell recipes.
tests/e2e/api/collections/provider-harness.json	Corrects Azure model names for embeddings, transcription, and TTS; pins Bedrock performance-config test to a model that supports it; replaces gpt-5 with gpt-4o-mini in stop-sequence matrix.

_{Reviews (3): Last reviewed commit: "fix: harness test fixes" | Re-trigger Greptile}

[akshaydeo]

Merge activity

• May 29, 1:50 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• May 29, 1:51 PM UTC: Graphite couldn't merge this PR because it had merge conflicts.
• May 29, 2:16 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• May 29, 2:16 PM UTC: @akshaydeo merged this pull request with Graphite.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Makefile`:
- Around line 1941-1942: The non-interactive sequential branch of the Makefile
(the `PARALLEL=0` / sequential `newman run` command) is missing passthrough of
the BEDROCK_GUARDRAIL_IDENTIFIER and BEDROCK_GUARDRAIL_VERSION env flags; update
the sequential `newman run` invocation to include the same conditional flags
used in the TTY/monitor path (i.e., add
$${BEDROCK_GUARDRAIL_IDENTIFIER:+--env-var
"bedrockGuardrailIdentifier=$$BEDROCK_GUARDRAIL_IDENTIFIER"} and
$${BEDROCK_GUARDRAIL_VERSION:+--env-var
"bedrockGuardrailVersion=$$BEDROCK_GUARDRAIL_VERSION"} to the command) so
CI/non-TTY runs honor guardrail settings.

In `@tests/e2e/api/collections/provider-harness.json`:
- Line 679: The test JSON has a model name "azure/text-embedding-ada-002" but
the URL.path and raw URL still point to "text-embedding-3-small:embedContent";
update the URL.path segment and the url.raw string to use
"text-embedding-ada-002:embedContent" so the request target matches the "name"
field (adjust the URL.path array element and the raw URL string in the same JSON
object).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 354bc099-2485-4b56-b50d-a2d6ca973915

📥 Commits

Reviewing files that changed from the base of the PR and between 96f597d and fc8c127.

📒 Files selected for processing (2)

• Makefile
• tests/e2e/api/collections/provider-harness.json