adds harness run test - #5933
Merged
Merged
Conversation
akshaydeo
marked this pull request as ready for review
August 7, 2026 05:55
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (11)
🚧 Files skipped from review as they are similar to previous changes (7)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe release pipeline adds provider and CLI harness jobs. Shell helpers build isolated gateways, run integration suites, publish reports, and clean up resources. CI monitoring emits progress and final summaries. Release jobs use CLI harness results for gating. ChangesHarness release CI
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ReleasePipeline
participant ProviderHarness
participant CLIHarness
participant BifrostGateway
participant GitHubStepSummary
ReleasePipeline->>ProviderHarness: run provider validation
ProviderHarness->>BifrostGateway: build, seed, and start gateway
ProviderHarness-->>ReleasePipeline: return provider status and reports
ReleasePipeline->>CLIHarness: run CLI validation
CLIHarness->>BifrostGateway: start isolated gateway
CLIHarness-->>GitHubStepSummary: publish CLI reports
CLIHarness-->>ReleasePipeline: return CLI status
ReleasePipeline->>ReleasePipeline: evaluate release gates
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
akshaydeo
force-pushed
the
08-06-adds_harness_run_test
branch
from
August 7, 2026 05:57
2ceaaf5 to
dca169b
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (3)
Makefile (1)
1834-1836: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThe pin is not enforced when newman is already installed.
which newmansucceeds for any installed version. If a developer has an older global newman, the target skips the install and the pinnedNEWMAN_VERSIONnever applies. The same applies to the reporter check, which usesnpm list -g newman-reporter-htmlextrawithout a version.Compare the installed version against the pin so local runs match CI.
♻️ Proposed refactor
- @$(USE_NODE); which newman > /dev/null 2>&1 || ($(ECHO) "$(YELLOW)Installing newman@$(NEWMAN_VERSION)...$(NC)" && npm install -g newman@$(NEWMAN_VERSION)) - @$(USE_NODE); npm list -g newman-reporter-htmlextra > /dev/null 2>&1 || ($(ECHO) "$(YELLOW)Installing newman-reporter-htmlextra@$(NEWMAN_HTMLEXTRA_VERSION)...$(NC)" && npm install -g newman-reporter-htmlextra@$(NEWMAN_HTMLEXTRA_VERSION)) + @$(USE_NODE); [ "$$(newman --version 2>/dev/null)" = "$(NEWMAN_VERSION)" ] || ($(ECHO) "$(YELLOW)Installing newman@$(NEWMAN_VERSION)...$(NC)" && npm install -g newman@$(NEWMAN_VERSION)) + @$(USE_NODE); npm list -g newman-reporter-htmlextra@$(NEWMAN_HTMLEXTRA_VERSION) > /dev/null 2>&1 || ($(ECHO) "$(YELLOW)Installing newman-reporter-htmlextra@$(NEWMAN_HTMLEXTRA_VERSION)...$(NC)" && npm install -g newman-reporter-htmlextra@$(NEWMAN_HTMLEXTRA_VERSION))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Makefile` around lines 1834 - 1836, Update the install-newman target’s checks for newman and newman-reporter-htmlextra to inspect their installed versions and install the pinned NEWMAN_VERSION or NEWMAN_HTMLEXTRA_VERSION when they are missing or mismatched. Preserve the existing skip behavior only when the installed package version exactly matches its configured pin..github/workflows/scripts/test-provider-harness.sh (1)
71-82: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
umask 077persists for the rest of the script.Line 73 sets
umask 077in the current shell, not in a subshell. Every file the script and its children create afterwards inherits the restrictive mask, including the harness reports undertmp/. This works in CI because the same user reads them. Scope the mask to the key write to keep the intent local.♻️ Proposed refactor
- umask 077 - printf '%s' "$VERTEX_CREDENTIALS" > "$GCLOUD_KEY_FILE" + (umask 077; printf '%s' "$VERTEX_CREDENTIALS" > "$GCLOUD_KEY_FILE")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/scripts/test-provider-harness.sh around lines 71 - 82, Scope the restrictive umask in the Vertex credential setup block so it applies only while writing GCLOUD_KEY_FILE, rather than persisting for the remainder of the script. Update the credential-write operation around VERTEX_CREDENTIALS and preserve the existing gcloud authentication flow and logging..github/workflows/release-pipeline.yml (1)
345-357: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider a manual override for
test-cli-harnessfailures.
test-corehasapprove-flaky-test-corebecause provider API calls are flaky.test-cli-harnessdrives the same real provider APIs through installed CLIs, but it has no equivalent override. Every downstream release job requirestest-cli-harness.result == 'success' || 'skipped'. A single flaky CLI cell then blocks the whole release and forces a full pipeline re-run.Add an approval gate job for
test-cli-harnessthat mirrorsapprove-flaky-test-core, or document why the CLI harness is treated as non-flaky.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/release-pipeline.yml around lines 345 - 357, Add an approval-gate job for test-cli-harness, modeled on approve-flaky-test-core, and wire downstream release conditions to accept the gate’s approved outcome when the harness fails while preserving success/skipped behavior. Use the existing workflow approval symbols and conventions rather than introducing a separate mechanism.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/release-pipeline.yml:
- Around line 363-385: Update the allowed-endpoints list in the release workflow
to include o33249.ingest.sentry.io:443 alongside statsig.anthropic.com:443,
keeping the surrounding Claude Code telemetry configuration and comment
consistent.
In @.github/workflows/scripts/test-cli-harness.sh:
- Around line 82-88: Adjust the timeout budget in run_cases so its two
run-cli-harness-test invocations, together with the workflow setup work, fit
within the test-cli-harness job’s timeout-minutes: 60 limit. Prefer lowering the
per-suite TIMEOUT from 25m while retaining enough headroom for report artifacts
to upload before job cancellation.
- Around line 70-72: Update run_cases in the harness script so the make
run-cli-harness-test invocation passes API_KEY="$HARNESS_API_KEY". Keep the
existing BIFROST_API_KEY export, allowing the target’s explicit environment
assignment to receive the placeholder key.
In @.github/workflows/scripts/test-core.sh:
- Around line 20-33: Add a dedicated core unit-test step in the test-core
workflow after the core build validation, running go test ./... from the core
module before invoking test-provider-harness.sh. Keep the existing build and
provider harness steps unchanged so core unit tests gate the release pipeline.
In `@tests/e2e/api/runners/harness-monitor.mjs`:
- Around line 63-64: Update the ci-interval parsing near CI_INTERVAL_MS to
validate that the parsed value is finite before applying Math.max. Use the
existing default interval value when args["ci-interval"] is non-numeric or
otherwise invalid, while preserving the minimum 5-second interval behavior for
valid values.
---
Nitpick comments:
In @.github/workflows/release-pipeline.yml:
- Around line 345-357: Add an approval-gate job for test-cli-harness, modeled on
approve-flaky-test-core, and wire downstream release conditions to accept the
gate’s approved outcome when the harness fails while preserving success/skipped
behavior. Use the existing workflow approval symbols and conventions rather than
introducing a separate mechanism.
In @.github/workflows/scripts/test-provider-harness.sh:
- Around line 71-82: Scope the restrictive umask in the Vertex credential setup
block so it applies only while writing GCLOUD_KEY_FILE, rather than persisting
for the remainder of the script. Update the credential-write operation around
VERTEX_CREDENTIALS and preserve the existing gcloud authentication flow and
logging.
In `@Makefile`:
- Around line 1834-1836: Update the install-newman target’s checks for newman
and newman-reporter-htmlextra to inspect their installed versions and install
the pinned NEWMAN_VERSION or NEWMAN_HTMLEXTRA_VERSION when they are missing or
mismatched. Preserve the existing skip behavior only when the installed package
version exactly matches its configured pin.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 34b55b54-5ccb-4ca5-9e67-bd009ebe9d29
📒 Files selected for processing (7)
.github/workflows/release-pipeline.yml.github/workflows/scripts/harness-gateway.sh.github/workflows/scripts/test-cli-harness.sh.github/workflows/scripts/test-core.sh.github/workflows/scripts/test-provider-harness.shMakefiletests/e2e/api/runners/harness-monitor.mjs
18 tasks
akshaydeo
force-pushed
the
08-06-adds_harness_run_test
branch
from
August 7, 2026 06:49
dca169b to
9c046db
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/release-pipeline.yml:
- Line 153: Update the test-core job condition at the workflow line guarded by
the core-needs-release output to also run when
needs.detect-changes.outputs.bifrost-http-needs-release is true, matching the
CLI harness condition. Preserve the existing skip-tests and should-skip checks
so bifrost-http-only releases execute provider-harness validation.
In @.github/workflows/scripts/test-cli-harness.sh:
- Around line 44-45: Update the CLI harness around CLAUDE_CASES and CODEX_CASES
to isolate each filter run by clearing reports or using a unique run-specific
reports directory. After every go test -run invocation, validate that the
expected cells were produced rather than relying only on the exit status and
reports/ existence, and fail the release gate when a provider is skipped or a
filter matches no TestCLIs subtests.
In `@tests/e2e/api/runners/lib/ci-interval.mjs`:
- Around line 17-19: Update the interval calculation around parsed and seconds
to cap the resolved integer-second value at 2,147,483 before multiplying by 1000
and passing it to setInterval. Preserve the existing default and minimum
handling, and add a regression test covering an oversized value such as
2_147_484.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9d7df02a-7deb-4f8c-b1c6-657bb170af6e
📒 Files selected for processing (9)
.github/workflows/release-pipeline.yml.github/workflows/scripts/harness-gateway.sh.github/workflows/scripts/test-cli-harness.sh.github/workflows/scripts/test-core.sh.github/workflows/scripts/test-provider-harness.shMakefiletests/e2e/api/runners/harness-monitor.mjstests/e2e/api/runners/lib/ci-interval.mjstests/e2e/api/runners/lib/ci-interval.test.mjs
🚧 Files skipped from review as they are similar to previous changes (3)
- .github/workflows/scripts/harness-gateway.sh
- Makefile
- tests/e2e/api/runners/harness-monitor.mjs
akshaydeo
force-pushed
the
08-06-adds_harness_run_test
branch
from
August 7, 2026 07:28
9c046db to
ddbb11d
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/release-pipeline.yml:
- Around line 153-163: Update the approval-job condition in
approve-flaky-test-core to allow execution when either core-needs-release or
bifrost-http-needs-release is true, so bifrost-http-only releases can use the
flaky-test override. Preserve the existing skip-tests and other gating
conditions.
In @.github/workflows/scripts/test-provider-harness.sh:
- Around line 51-57: Update cleanup() so harness_stop_gateway is best-effort and
cannot abort the function under set -e; ensure rm -f "$GCLOUD_KEY_FILE" always
executes before exiting with the original exit_code.
- Around line 99-105: Redact sensitive diagnostics before publishing or
uploading harness-failures.md, including authorization values, API keys, tokens,
query secrets, and sensitive payloads from raw response bodies, URLs, assertion
messages, and Bifrost log lines. Update the analyze-failures.mjs generation path
and ensure the summary block guarded by GITHUB_STEP_SUMMARY only consumes the
sanitized file.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e1347be7-6689-490c-8486-1c33fec7f4b7
📒 Files selected for processing (11)
.github/workflows/release-pipeline.yml.github/workflows/scripts/harness-gateway.sh.github/workflows/scripts/test-cli-harness.sh.github/workflows/scripts/test-core.sh.github/workflows/scripts/test-provider-harness.shMakefiletests/e2e/api/runners/harness-monitor.mjstests/e2e/api/runners/lib/ci-interval.mjstests/e2e/api/runners/lib/ci-interval.test.mjstests/e2e/clis/clis_test.gotests/e2e/clis/reportsdir_test.go
🚧 Files skipped from review as they are similar to previous changes (4)
- .github/workflows/scripts/harness-gateway.sh
- tests/e2e/api/runners/harness-monitor.mjs
- tests/e2e/api/runners/lib/ci-interval.test.mjs
- Makefile
akshaydeo
force-pushed
the
08-06-adds_harness_run_test
branch
from
August 7, 2026 07:40
ddbb11d to
6e6112d
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Contributor
Author
Merge activity
|
akshaydeo
added a commit
that referenced
this pull request
Aug 7, 2026
## Summary Replaces the `test-core` CI job's in-process `go test` suite with an end-to-end provider harness run. Instead of running unit tests with coverage, the job now builds a `bifrost-http` binary and drives the full provider harness collection (`tests/e2e/api/collections/provider-harness.json`) against it via Newman, exercising real provider APIs from a live gateway. ## Changes - **`test-core` job** no longer runs `go test` or uploads coverage to Codecov. It now validates the core build, then delegates to `test-provider-harness.sh` which boots a real `bifrost-http` binary and runs the Newman harness against it. - **`test-provider-harness.sh`** (new script) handles the full CI lifecycle: builds the UI and binary, seeds a throwaway SQLite app dir, authenticates gcloud from `VERTEX_CREDENTIALS` for Vertex token-parity cells, waits for `/health`, then invokes `make run-provider-harness-test CI=1 USE_INFISICAL=0`. - **Egress allowlist** expanded to cover Vertex (`aiplatform.googleapis.com`, `us-central1-aiplatform.googleapis.com`, `accounts.google.com`, `oauth2.googleapis.com`), gcloud CLI install (`dl.google.com`, `packages.cloud.google.com`, `objects.githubusercontent.com`), Replicate (`api.replicate.com`), and the getbifrost.ai pricing datasheet used by the dbverify reporter. - **New CI dependencies** installed in the job: `jq`, `newman@6.2.1`, `newman-reporter-htmlextra@1.23.1`, and the gcloud CLI (`google-github-actions/setup-gcloud@v3`). - **New secrets forwarded** to the run step: `AWS_REGION`, `GOOGLE_LOCATION`, `XAI_API_KEY`, `REPLICATE_API_KEY`, `BEDROCK_GUARDRAIL_IDENTIFIER`, `BEDROCK_GUARDRAIL_VERSION`. `CODECOV_TOKEN` removed. - **Harness artifacts** (`newman-report*.json`, `newman-cli*.log`, `harness-failures.md`, `harness-token-parity.md`, `bifrost-dev.log`, etc.) are uploaded on every run (including failures) with a 30-day retention. - **`harness-monitor.mjs`** gains a `--ci` mode that disables the alternate screen buffer and cursor-home redraws, emits append-only progress snapshots every `--ci-interval` seconds (default 30), logs each assertion failure inline as it is parsed, and writes a plain-text final table to both stdout and `$GITHUB_STEP_SUMMARY`. - **Makefile** pins Newman versions via `NEWMAN_VERSION` and `NEWMAN_HTMLEXTRA_VERSION` variables (defaulting to `6.2.1` / `1.23.1`) and fixes the `CI=1` branch of `run-provider-harness-test` to start the monitor in append-only mode and still print per-provider pass/fail lines and log tails. - Job timeout set to 90 minutes to accommodate the full provider sweep. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test The harness runs automatically in CI on any change that triggers `core-needs-release`. To run locally: ```sh make run-provider-harness-test ``` For CI-equivalent mode (append-only output, no terminal UI): ```sh make run-provider-harness-test CI=1 USE_INFISICAL=0 PARALLEL=1 ``` Required environment variables for the full sweep: all existing provider API keys plus `AWS_REGION`, `GOOGLE_LOCATION`, `VERTEX_CREDENTIALS` (service-account JSON), `XAI_API_KEY`, `REPLICATE_API_KEY`, `BEDROCK_GUARDRAIL_IDENTIFIER`, `BEDROCK_GUARDRAIL_VERSION`. ## Breaking changes - [x] Yes - [ ] No The `test-core` job no longer runs `go test` or reports coverage to Codecov. Any downstream tooling that depends on Codecov coverage reports from this job will stop receiving them. ## Security considerations - The Vertex service-account key (`VERTEX_CREDENTIALS`) is written to disk only for the duration of the gcloud auth call and is deleted in the `cleanup` trap before the artifact-upload step runs, ensuring it is never captured in uploaded artifacts. - The egress allowlist is explicitly extended for each new external endpoint; no wildcard rules are added. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
This was referenced Aug 8, 2026
atharvamhaske
pushed a commit
to atharvamhaske/bifrost
that referenced
this pull request
Aug 13, 2026
## Summary Replaces the `test-core` CI job's in-process `go test` suite with an end-to-end provider harness run. Instead of running unit tests with coverage, the job now builds a `bifrost-http` binary and drives the full provider harness collection (`tests/e2e/api/collections/provider-harness.json`) against it via Newman, exercising real provider APIs from a live gateway. ## Changes - **`test-core` job** no longer runs `go test` or uploads coverage to Codecov. It now validates the core build, then delegates to `test-provider-harness.sh` which boots a real `bifrost-http` binary and runs the Newman harness against it. - **`test-provider-harness.sh`** (new script) handles the full CI lifecycle: builds the UI and binary, seeds a throwaway SQLite app dir, authenticates gcloud from `VERTEX_CREDENTIALS` for Vertex token-parity cells, waits for `/health`, then invokes `make run-provider-harness-test CI=1 USE_INFISICAL=0`. - **Egress allowlist** expanded to cover Vertex (`aiplatform.googleapis.com`, `us-central1-aiplatform.googleapis.com`, `accounts.google.com`, `oauth2.googleapis.com`), gcloud CLI install (`dl.google.com`, `packages.cloud.google.com`, `objects.githubusercontent.com`), Replicate (`api.replicate.com`), and the getbifrost.ai pricing datasheet used by the dbverify reporter. - **New CI dependencies** installed in the job: `jq`, `newman@6.2.1`, `newman-reporter-htmlextra@1.23.1`, and the gcloud CLI (`google-github-actions/setup-gcloud@v3`). - **New secrets forwarded** to the run step: `AWS_REGION`, `GOOGLE_LOCATION`, `XAI_API_KEY`, `REPLICATE_API_KEY`, `BEDROCK_GUARDRAIL_IDENTIFIER`, `BEDROCK_GUARDRAIL_VERSION`. `CODECOV_TOKEN` removed. - **Harness artifacts** (`newman-report*.json`, `newman-cli*.log`, `harness-failures.md`, `harness-token-parity.md`, `bifrost-dev.log`, etc.) are uploaded on every run (including failures) with a 30-day retention. - **`harness-monitor.mjs`** gains a `--ci` mode that disables the alternate screen buffer and cursor-home redraws, emits append-only progress snapshots every `--ci-interval` seconds (default 30), logs each assertion failure inline as it is parsed, and writes a plain-text final table to both stdout and `$GITHUB_STEP_SUMMARY`. - **Makefile** pins Newman versions via `NEWMAN_VERSION` and `NEWMAN_HTMLEXTRA_VERSION` variables (defaulting to `6.2.1` / `1.23.1`) and fixes the `CI=1` branch of `run-provider-harness-test` to start the monitor in append-only mode and still print per-provider pass/fail lines and log tails. - Job timeout set to 90 minutes to accommodate the full provider sweep. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test The harness runs automatically in CI on any change that triggers `core-needs-release`. To run locally: ```sh make run-provider-harness-test ``` For CI-equivalent mode (append-only output, no terminal UI): ```sh make run-provider-harness-test CI=1 USE_INFISICAL=0 PARALLEL=1 ``` Required environment variables for the full sweep: all existing provider API keys plus `AWS_REGION`, `GOOGLE_LOCATION`, `VERTEX_CREDENTIALS` (service-account JSON), `XAI_API_KEY`, `REPLICATE_API_KEY`, `BEDROCK_GUARDRAIL_IDENTIFIER`, `BEDROCK_GUARDRAIL_VERSION`. ## Breaking changes - [x] Yes - [ ] No The `test-core` job no longer runs `go test` or reports coverage to Codecov. Any downstream tooling that depends on Codecov coverage reports from this job will stop receiving them. ## Security considerations - The Vertex service-account key (`VERTEX_CREDENTIALS`) is written to disk only for the duration of the gcloud auth call and is deleted in the `cleanup` trap before the artifact-upload step runs, ensuring it is never captured in uploaded artifacts. - The egress allowlist is explicitly extended for each new external endpoint; no wildcard rules are added. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
akhsaul
pushed a commit
to akhsaul/bifrost
that referenced
this pull request
Aug 27, 2026
## Summary Replaces the `test-core` CI job's in-process `go test` suite with an end-to-end provider harness run. Instead of running unit tests with coverage, the job now builds a `bifrost-http` binary and drives the full provider harness collection (`tests/e2e/api/collections/provider-harness.json`) against it via Newman, exercising real provider APIs from a live gateway. ## Changes - **`test-core` job** no longer runs `go test` or uploads coverage to Codecov. It now validates the core build, then delegates to `test-provider-harness.sh` which boots a real `bifrost-http` binary and runs the Newman harness against it. - **`test-provider-harness.sh`** (new script) handles the full CI lifecycle: builds the UI and binary, seeds a throwaway SQLite app dir, authenticates gcloud from `VERTEX_CREDENTIALS` for Vertex token-parity cells, waits for `/health`, then invokes `make run-provider-harness-test CI=1 USE_INFISICAL=0`. - **Egress allowlist** expanded to cover Vertex (`aiplatform.googleapis.com`, `us-central1-aiplatform.googleapis.com`, `accounts.google.com`, `oauth2.googleapis.com`), gcloud CLI install (`dl.google.com`, `packages.cloud.google.com`, `objects.githubusercontent.com`), Replicate (`api.replicate.com`), and the getbifrost.ai pricing datasheet used by the dbverify reporter. - **New CI dependencies** installed in the job: `jq`, `newman@6.2.1`, `newman-reporter-htmlextra@1.23.1`, and the gcloud CLI (`google-github-actions/setup-gcloud@v3`). - **New secrets forwarded** to the run step: `AWS_REGION`, `GOOGLE_LOCATION`, `XAI_API_KEY`, `REPLICATE_API_KEY`, `BEDROCK_GUARDRAIL_IDENTIFIER`, `BEDROCK_GUARDRAIL_VERSION`. `CODECOV_TOKEN` removed. - **Harness artifacts** (`newman-report*.json`, `newman-cli*.log`, `harness-failures.md`, `harness-token-parity.md`, `bifrost-dev.log`, etc.) are uploaded on every run (including failures) with a 30-day retention. - **`harness-monitor.mjs`** gains a `--ci` mode that disables the alternate screen buffer and cursor-home redraws, emits append-only progress snapshots every `--ci-interval` seconds (default 30), logs each assertion failure inline as it is parsed, and writes a plain-text final table to both stdout and `$GITHUB_STEP_SUMMARY`. - **Makefile** pins Newman versions via `NEWMAN_VERSION` and `NEWMAN_HTMLEXTRA_VERSION` variables (defaulting to `6.2.1` / `1.23.1`) and fixes the `CI=1` branch of `run-provider-harness-test` to start the monitor in append-only mode and still print per-provider pass/fail lines and log tails. - Job timeout set to 90 minutes to accommodate the full provider sweep. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test The harness runs automatically in CI on any change that triggers `core-needs-release`. To run locally: ```sh make run-provider-harness-test ``` For CI-equivalent mode (append-only output, no terminal UI): ```sh make run-provider-harness-test CI=1 USE_INFISICAL=0 PARALLEL=1 ``` Required environment variables for the full sweep: all existing provider API keys plus `AWS_REGION`, `GOOGLE_LOCATION`, `VERTEX_CREDENTIALS` (service-account JSON), `XAI_API_KEY`, `REPLICATE_API_KEY`, `BEDROCK_GUARDRAIL_IDENTIFIER`, `BEDROCK_GUARDRAIL_VERSION`. ## Breaking changes - [x] Yes - [ ] No The `test-core` job no longer runs `go test` or reports coverage to Codecov. Any downstream tooling that depends on Codecov coverage reports from this job will stop receiving them. ## Security considerations - The Vertex service-account key (`VERTEX_CREDENTIALS`) is written to disk only for the duration of the gcloud auth call and is deleted in the `cleanup` trap before the artifact-upload step runs, ensuring it is never captured in uploaded artifacts. - The egress allowlist is explicitly extended for each new external endpoint; no wildcard rules are added. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the
test-coreCI job's in-processgo testsuite with an end-to-end provider harness run. Instead of running unit tests with coverage, the job now builds abifrost-httpbinary and drives the full provider harness collection (tests/e2e/api/collections/provider-harness.json) against it via Newman, exercising real provider APIs from a live gateway.Changes
test-corejob no longer runsgo testor uploads coverage to Codecov. It now validates the core build, then delegates totest-provider-harness.shwhich boots a realbifrost-httpbinary and runs the Newman harness against it.test-provider-harness.sh(new script) handles the full CI lifecycle: builds the UI and binary, seeds a throwaway SQLite app dir, authenticates gcloud fromVERTEX_CREDENTIALSfor Vertex token-parity cells, waits for/health, then invokesmake run-provider-harness-test CI=1 USE_INFISICAL=0.aiplatform.googleapis.com,us-central1-aiplatform.googleapis.com,accounts.google.com,oauth2.googleapis.com), gcloud CLI install (dl.google.com,packages.cloud.google.com,objects.githubusercontent.com), Replicate (api.replicate.com), and the getbifrost.ai pricing datasheet used by the dbverify reporter.jq,newman@6.2.1,newman-reporter-htmlextra@1.23.1, and the gcloud CLI (google-github-actions/setup-gcloud@v3).AWS_REGION,GOOGLE_LOCATION,XAI_API_KEY,REPLICATE_API_KEY,BEDROCK_GUARDRAIL_IDENTIFIER,BEDROCK_GUARDRAIL_VERSION.CODECOV_TOKENremoved.newman-report*.json,newman-cli*.log,harness-failures.md,harness-token-parity.md,bifrost-dev.log, etc.) are uploaded on every run (including failures) with a 30-day retention.harness-monitor.mjsgains a--cimode that disables the alternate screen buffer and cursor-home redraws, emits append-only progress snapshots every--ci-intervalseconds (default 30), logs each assertion failure inline as it is parsed, and writes a plain-text final table to both stdout and$GITHUB_STEP_SUMMARY.NEWMAN_VERSIONandNEWMAN_HTMLEXTRA_VERSIONvariables (defaulting to6.2.1/1.23.1) and fixes theCI=1branch ofrun-provider-harness-testto start the monitor in append-only mode and still print per-provider pass/fail lines and log tails.Type of change
Affected areas
How to test
The harness runs automatically in CI on any change that triggers
core-needs-release. To run locally:For CI-equivalent mode (append-only output, no terminal UI):
Required environment variables for the full sweep: all existing provider API keys plus
AWS_REGION,GOOGLE_LOCATION,VERTEX_CREDENTIALS(service-account JSON),XAI_API_KEY,REPLICATE_API_KEY,BEDROCK_GUARDRAIL_IDENTIFIER,BEDROCK_GUARDRAIL_VERSION.Breaking changes
The
test-corejob no longer runsgo testor reports coverage to Codecov. Any downstream tooling that depends on Codecov coverage reports from this job will stop receiving them.Security considerations
VERTEX_CREDENTIALS) is written to disk only for the duration of the gcloud auth call and is deleted in thecleanuptrap before the artifact-upload step runs, ensuring it is never captured in uploaded artifacts.Checklist
docs/contributing/README.mdand followed the guidelines