adds make recipie to run integration tests locally

[akshaydeo]

Summary

Adds a make test-integrations target that runs both the Python and TypeScript SDK integration suites in parallel (one process per test file) against a single shared gateway, and improves the underlying test-integrations.sh script with live output streaming, a structured markdown failure report, INTEGRATION_TEST_FILTER support for narrowing to a single provider, and SKIP_GATEWAY_START support for attaching to an already-running gateway.

Changes

• make test-integrations: New Makefile target that delegates to test-integrations.sh --parallel-files. Detects whether a gateway is already serving HOST:PORT and sets SKIP_GATEWAY_START=1 if so, reuses tmp/bifrost-http by default (skipping the full UI build), accepts JOBS=N and INTEGRATION=<name> overrides, and validates bash 5.1+ before spending time on secrets or builds.

• SKIP_GATEWAY_START: When set, the script skips building the binary, skips starting the MCP fixture, and re-probes /health on the existing server rather than launching its own. cleanup() leaves the reused server running since it did not start it.

• INTEGRATION_TEST_FILTER / --parallel-files narrowing: The filter is validated to [A-Za-z0-9_-] only (interpolated into a glob, so metacharacters would escape the test directories). In the parallel path it filters both test_<name>.py and test-<name>.test.ts lists after globbing, errors if nothing matches, and lists available integrations to help correct typos. In the sequential path it warns that the filter is ignored rather than silently running the full suite.

• Live output streaming: launch_test_file now pipes each job through tee so lines appear as they happen. When more than one file runs concurrently, an awk prefix ([label] line) keeps concurrent output attributable. PYTHONUNBUFFERED=1 is exported so Python does not block-buffer into the pipe. Only failed jobs are replayed at the end; passing jobs are not echoed a second time.

• Markdown failure report: write_failure_report writes test-reports/integration-failures.md on every run (overwriting a stale red report on a green run). It extracts failed case names from pytest (::.*FAILED) and vitest (× suite > case) output after stripping ANSI escapes, copies each failed job's full log to test-reports/integration-<slug>.log, and includes a collapsible failure-output excerpt per file. TEST_FINAL_STATUS[] is introduced alongside TEST_STATUSES[] because the throttle's bookkeeping array is empty for jobs the final wait reaps.

• mktemp portability: Removed .log suffixes from mktemp templates. BSD/macOS mktemp only substitutes trailing Xs, so name.XXXXXX.log is created literally and the second run fails with "File exists". GNU mktemp accepts a suffix flag, which is why this only manifests outside CI.

• bash version hint: The "bash 5.1 required" error now also prints the macOS remedy (brew install bash).

• Trailing whitespace: Removed a stray trailing space in cleanup().

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

# Run all integration tests (starts its own gateway if none is running)
make test-integrations

# Run only the OpenAI integration
make test-integrations INTEGRATION=openai

# Attach to an already-running gateway on a non-default port
make test-integrations PORT=8081

# Force a fresh binary build before running
make test-integrations BUILD=1

# Limit concurrency (e.g. to avoid 429s under quota pressure)
make test-integrations JOBS=4

After a run, inspect test-reports/integration-failures.md for a structured digest and test-reports/integration-<slug>.log for each failed file's full output.

Set INTEGRATION_TEST_FILTER=openai (or INTEGRATION=openai via the Makefile) to confirm the filter rejects invalid characters, errors on an unmatched name with a list of available integrations, and runs only the matching Python and TypeScript files.

Breaking changes

• Yes
• No

Related issues

Security considerations

INTEGRATION_TEST_FILTER is validated to [A-Za-z0-9_-] before being interpolated into a glob. A path separator, glob metacharacter, or .. sequence would be rejected, preventing the filter from reaching files outside the two tests/ directories. mktemp paths use random suffixes rather than predictable names to avoid symlink-based truncation attacks in world-writable /tmp.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: 1731a4ab-281a-4eff-a00d-62ac4be6726b

📥 Commits

Reviewing files that changed from the base of the PR and between a29f749 and c5cc564.

📒 Files selected for processing (2)

• .github/workflows/scripts/test-integrations.sh
• Makefile

---

📝 Summary

Summary by CodeRabbit

• New Features

  • Added a test-integrations command for running Python and TypeScript integration tests in parallel.
  • Added configurable host, port, job count, test filters, build behavior, and report output settings.
  • Added support for reusing a healthy gateway and skipping unnecessary startup steps.
  • Added streamed test output, per-file status reporting, failure logs, and Markdown test reports.
  • Added validation and actionable guidance for invalid filters, ports, job counts, and unsupported environments.

• Improvements

  • Improved compatibility for temporary files across macOS and GNU/Linux.

Walkthrough

Changes

Integration test execution

Layer / File(s)	Summary
Gateway orchestration and Make target .github/workflows/scripts/test-integrations.sh, Makefile	The new test-integrations target validates settings, reuses healthy gateways, manages gateway startup, and delegates parallel test execution.
Parallel output and failure reporting .github/workflows/scripts/test-integrations.sh	Parallel runs stream labeled output, track per-file results, extract failures, copy failed logs, and write Markdown reports.
Test filtering and execution modes .github/workflows/scripts/test-integrations.sh	Parallel runs validate and apply filename filters. Sequential runs warn that filters are ignored and run both suites.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Developer
  participant MakeTarget
  participant TestIntegrationsScript
  participant Gateway
  participant TestFiles
  Developer->>MakeTarget: run test-integrations
  MakeTarget->>TestIntegrationsScript: pass gateway and parallelism settings
  TestIntegrationsScript->>Gateway: health-check or start gateway
  TestIntegrationsScript->>TestFiles: run selected files in parallel
  TestFiles-->>TestIntegrationsScript: streamed output and exit status
  TestIntegrationsScript-->>Developer: statuses and failure report

Loading

Suggested reviewers: tejasghatte

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 09-15-adds_make_recipie_to_run_integration_tests_locally

---

_{Comment @coderabbitai help to get the list of available commands.}

[akshaydeo]

• adds make recipie to run integration tests locally #7196 👈 (View in Graphite)
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[akshaydeo]

Merge activity

• Sep 15, 4:49 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Sep 15, 4:49 PM UTC: @akshaydeo merged this pull request with Graphite.