tests:harness tests with compat and fixes

[TejasGhatte]

Summary

Adds COMPAT mode support to the provider harness test runner, allowing the harness to be executed with the x-bf-compat header injected on every request. This enables side-by-side comparison of provider behavior with compat on vs. off without manually re-running the suite.

Changes

• Added a COMPAT Makefile variable that accepts on, off, or both. When set to both, the harness runs sequentially with compat off then on, saving separate report artifacts (newman-report-compat-{off,on}.{json,html}, harness-failures-compat-{off,on}.md) under tmp/.
• When COMPAT=on (or true/1/yes/y), --env-var "compat=true" is passed to all three Newman invocation paths in the harness target.
• Added a pre-request script in the harness Postman collection that reads the compat environment variable and, when present, upserts the x-bf-compat header on every outgoing request.
• Fixed stop_sequences → stop mapping in the model catalog datasheet parameter extractor so that stop is correctly surfaced as a supported top-level request parameter.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Run the harness in each mode and verify separate report files are produced:

# Run with compat off (baseline)
make run-provider-harness-test COMPAT=off

# Run with compat on
make run-provider-harness-test COMPAT=on

# Run both sequentially and produce split reports
make run-provider-harness-test COMPAT=both

# Verify artifacts
ls tmp/newman-report-compat-{off,on}.{json,html}
ls tmp/harness-failures-compat-{off,on}.md

COMPAT is unset by default, preserving existing harness behavior.

Screenshots/Recordings

N/A

Breaking changes

• Yes
• No

Related issues

N/A

Security considerations

The x-bf-compat header is injected only during test runs via Newman and is controlled by an explicit opt-in variable. No secrets or PII are involved.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a0e34245-f31c-419e-b7ab-a85cd8ff6d0f

📥 Commits

Reviewing files that changed from the base of the PR and between 68d4de2 and bbfc7a3.

📒 Files selected for processing (3)

• Makefile
• framework/modelcatalog/datasheet/types.go
• tests/e2e/api/collections/provider-harness.json

🚧 Files skipped from review as they are similar to previous changes (3)

• tests/e2e/api/collections/provider-harness.json
• Makefile
• framework/modelcatalog/datasheet/types.go

---

📝 Walkthrough

Summary by CodeRabbit

• New Features
  • Introduced compatibility mode support for API testing and execution
  • Enhanced parameter support to recognize stop sequences in model configurations
  • Added dynamic compatibility header management for API requests

Walkthrough

Adds a COMPAT Makefile variable that enables dual harness runs (compat off then on) with renamed report artifacts, injects --env-var "compat=true" into all Newman invocations when compat is enabled, and extends the Postman prerequest script to upsert the x-bf-compat header. Separately, extractSupportedParams gains a stop_sequences → stop parameter mapping.

Changes

Compat mode harness infrastructure

Layer / File(s)	Summary
COMPAT variable, dual-run orchestration, and Newman injection Makefile	Declares COMPAT ?= at the top level, adds an early COMPAT=both branch in run-provider-harness-test that executes two forced CI sub-runs (compat off then on) with artifact renaming and combined exit code, and conditionally appends --env-var "compat=true" to all three newman run paths.
Postman prerequest compat header injection tests/e2e/api/collections/provider-harness.json	Extends the prerequest script to read compat from the Postman environment/variables and upsert the x-bf-compat header on every request when the value is truthy.

stop_sequences model parameter mapping

Layer / File(s)	Summary
stop_sequences → stop parameter mapping framework/modelcatalog/datasheet/types.go	Adds a stop_sequences case to the extractSupportedParams switch, appending stop to the supported-parameter list for models that advertise this parameter.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• maximhq/bifrost#3887: Modifies the same run-provider-harness-test Makefile logic to inject provider-specific environment variables into Newman harness runs, identical pattern to this PR's compat=true injection.
• maximhq/bifrost#3457: Both PRs modify the Makefile's run-provider-harness-test target, with the retrieved PR changing the same target's FEATURE filtering logic.

Suggested reviewers

• danpiths

Poem

🐇 Hop hop, the harness runs twice now,
Off then on, a compat bow-wow!
Headers upsert, the x-bf-compat way,
And stop_sequences joins the fray.
Two runs, one score — the rabbit's done today! 🎉

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately reflects the main changes: adding harness tests with compat mode support and fixes to parameter mapping.
Description check	✅ Passed	The description includes all major sections from the template with comprehensive details about changes, testing approach, and security considerations.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 06-14-tests_harness_tests_with_compat_and_fixes

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[TejasGhatte]

• tests:harness tests with compat and fixes #4379 👈 (View in Graphite)
• fix: set ttl in bedrock cache points #4378
• feat: advisor tool compatibility claude #4357
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Makefile`:
- Line 1912: The COMPAT variable matching filter at Makefile line 1912 (anchor)
and its sibling locations at lines 1995 and 2019 inconsistently handle truthy
values and miss common uppercase variants like TRUE and ON. Update the filter
expressions in all three locations to normalize the truthy value matching by
adding uppercase equivalents (TRUE, ON) to the current set of accepted values
(on, true, 1, yes, YES, y, Y). This ensures that COMPAT=TRUE and COMPAT=ON are
treated the same as their lowercase counterparts and properly trigger compat
header injection.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f681a219-4d61-47bf-94a9-d36a0ecb137d

📥 Commits

Reviewing files that changed from the base of the PR and between d652ac6 and 4c3eb3d.

📒 Files selected for processing (3)

• Makefile
• framework/modelcatalog/datasheet/types.go
• tests/e2e/api/collections/provider-harness.json

👮 Files not reviewed due to content moderation or server errors (1)

• tests/e2e/api/collections/provider-harness.json

[greptile-apps]

Confidence Score: 5/5

Safe to merge — changes are limited to the test harness Makefile, a Postman pre-request script, and a one-line parameter mapping fix with no impact on production code paths.

All three changes are confined to test infrastructure and a small data-extraction helper. The Makefile COMPAT logic correctly gates compat header injection behind an explicit opt-in filter and propagates non-zero exit codes. The stop_sequences to stop mapping is additive and only affects the model catalog parameter surface list. No production request handling, streaming, pooling, or auth paths are touched.

No files require special attention.

Important Files Changed

Filename	Overview
Makefile	Adds COMPAT variable and COMPAT=both logic with recursive make invocations; compat filter pattern and RC propagation are correct, with a minor cosmetic issue in how BOTH_RC is accumulated.
framework/modelcatalog/datasheet/types.go	Adds stop_sequences to stop mapping in extractSupportedParams; straightforward and correct fix.
tests/e2e/api/collections/provider-harness.json	Adds pre-request script to inject x-bf-compat header when compat env var is set; uses upsert correctly, logic is sound.

_{Reviews (3): Last reviewed commit: "tests:harness tests with compat and fixe..." | Re-trigger Greptile}

[akshaydeo]

Merge activity

• Jun 15, 5:19 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 15, 5:21 AM UTC: Graphite couldn't merge this pull request because a downstack PR fix: set ttl in bedrock cache points #4378 failed to merge.
• Jun 15, 8:47 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 15, 8:48 AM UTC: Graphite couldn't merge this PR because it had merge conflicts.
• Jun 15, 9:22 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 15, 9:22 AM UTC: @akshaydeo merged this pull request with Graphite.